geo/task_plan.deploy-39.md

Task Plan: Sync Current Workspace to 39

Goal

Synchronize the complete current workspace state to 39.105.229.239, deploy it to the existing k3s installation, apply required migrations, and verify service health with a rollback point preserved.

Current Phase

Complete

Phases

Phase 1: Audit

• Read repository deployment instructions and scripts
• Identify remote repository path, runtime configuration, workloads, and current image state
• Confirm the exact local file set and required migrations

Phase 2: Backup and Sync

• Create a timestamped remote rollback snapshot
• Synchronize the current workspace without overwriting secrets or runtime-owned data
• Verify remote source checksums for representative changed files

Phase 3: Build and Rollout

• Build and import affected images into k3s
• Apply database migration and manifests/configuration
• Wait for all affected workloads to finish rolling out

Phase 4: Verification

• Check pods, events, rollout revisions, and recent logs
• Exercise health/API endpoints
• Record deployed image identifiers and rollback instructions

Phase 5: July 12 Delta Audit

• Compare current clean HEAD with the July 10 deployed source snapshot
• Identify affected images, configuration, and migrations
• Run tests/builds appropriate to the delta

Phase 6: July 12 Backup and Sync

• Build and checksum target-native release archives
• Capture a new remote rollback point
• Upload and independently verify release artifacts

Phase 7: July 12 Rollout

• Import release images; migration Job correctly skipped because no migration changed
• Confirm no configuration changes are required
• Roll affected workloads and wait for readiness

Phase 8: July 12 Verification and Cleanup

• Verify controllers, endpoints, logs, schema, and release contents
• Remove stale application ReplicaSets and unreferenced old application images
• Clear only geo:cache:* and preserve Redis operational state

Decisions

Decision	Rationale
Deploy the whole current workspace	The user explicitly requested syncing the current changes, and the worktree contains several integrated features.
Preserve remote secrets and runtime data	Source synchronization must not replace environment-specific credentials, volumes, or cluster state.
Create a rollback snapshot before mutation	The source is a dirty worktree and cannot be reconstructed from a single commit.
Clear old runtime state only after successful rollout	Preserves rollback and avoids taking down the healthy release before its replacement is ready.
Use the July 10 remote source snapshot as the delta baseline	That release included a workspace snapshot, so commit history alone may overstate what is new.

Errors

Error	Attempt	Resolution
zsh expanded brackets in an unquoted custom-columns expression during two read-only inventory commands	2	Stop using bracket-bearing custom-columns in outer shell commands; use get deploy -o wide, go-template, or per-deployment JSONPath inside a quoted bash script.
OrbStack retained only the final amd64 image after sequential cross-architecture buildx --load operations, so release inventory failed	1	Keep the warm BuildKit cache but rebuild each target directly to its own Docker archive and import archives independently.
First planning-file patch used stale table context and did not apply	1	Re-read the current files and applied smaller exact-context updates.
Redis audit addressed deployment/redis, but Redis is a StatefulSet/pod on this cluster	1	Retry against pod/redis-0; do not issue any deletion until key classes are understood.
Broad local config context read exposed adjacent tracked secret-bearing values internally	1	Do not repeat or record values; use exact structural queries without config context for the remainder of deployment.
Source-snapshot verification changed into the release directory before comparing repository prompt files, so the final two cmp checks used the wrong relative path	1	The archive and checksum completed successfully; rerun prompt comparisons from the repository root with absolute paths.
Workspace checksum manifest was verified from the repository root even though it stores a release-directory-relative filename	1	Verify workspace.sha256 from inside the release artifact directory.
macOS bundled rsync does not support --info=progress2	1	No file transfer started; retry with portable --progress --stats options.
A guessed migration path plus an unmatched zsh glob prevented a read-only table-name lookup	1	Locate the migration with rg --files and then read the exact path without shell globbing.
The target host does not have jq, so ConfigMap merge preparation exited at its explicit prerequisite check	1	No patch was applied; generate the one-field JSON merge patch locally with jq, upload it, and use kubectl patch --patch-file.
Immediate Service curl after the repaired official-placeholder rollout returned connection refused even though rollout status succeeded	1	Inspect the new Pod, EndpointSlice, and nginx logs; retry only after confirming whether this was endpoint propagation or a runtime/config issue.
Rollback-prompt generation assumed Git HEAD still contained the pre-release prompt, but HEAD already contains the strict prompt rules	1	Upload did not start; parse the backed-up pre-release ConfigMap with local kubectl and extract only data["prompts.yml"] into the merge patch.
Local kubectl still attempted API discovery for a client dry-run of the backed-up ConfigMap and could not reach its configured local cluster	1	No patch was produced; use Ruby's standard YAML and JSON parsers on the trusted backup, then verify the extracted prompt against the known old hash before upload.
Final verification stopped after HTTP checks because one log-archive command returned nonzero under set -e	1	Deployment state remains healthy; run each selector log read independently to identify the exact failure, then make the final verifier tolerate empty/no-log sources while still scanning available logs.
Two geo:cache:* keys were naturally repopulated by live traffic after the verified cleanup, so a later zero-cache assertion stopped final verification	1	Complete all remaining read checks first, then repeat the same prefix-scoped deletion as the final action and immediately verify protected Redis key classes are unchanged.
July 12 source comparison used path as a zsh loop variable, which overwrote the shell's special PATH array and made later commands unavailable	1	Archive verification/extraction succeeded and no remote state changed; rerun the comparison with a neutral loop variable such as item.
A July 12 archive diagnostic assigned to zsh's read-only special variable status and exited after printing the correct source commit	1	Image tag/platform/content and checksums had already passed; finish source/build-input verification with a neutral variable such as rc, without rebuilding.
`gzip -dc	git get-tar-commit-idreturned 141 underpipefail` because Git intentionally stops reading after the PAX commit header and gzip receives SIGPIPE	2
The first restored pause-image upload placed the tar at incoming root while its checksum manifest referenced images/, so target verification failed before import	1	Move the already uploaded tar into incoming/images/, reverify SHA-256, then import it; no unverified image was imported.
The original frontend rollout SSH client remained open after the remote process disappeared, despite the deployment succeeding	1	Confirmed the rollout and new Pod separately, then terminated only the orphaned local SSH client and will repeat the remaining checks read-only.
First rollback-guide patch omitted an Add File + marker on one line and was rejected before writing	1	Reissued the same guide with a syntactically valid patch; no existing file was changed by the failed attempt.

Outcome

• Release workspace-5463319-20260710120256 is fully deployed on 39.
• Migration 20260710183000 is clean and all three media-favorite tables exist.
• All application and infrastructure controllers are Ready; all release Pods have zero restarts.
• Old zero-replica ReplicaSets, unreferenced old geo-rankly images, and geo:cache:* were removed while protected Redis/runtime data and rollback artifacts were retained.
• July 12 frontend-only release workspace-e68db14-20260712150344 is deployed and verified; desktop-client publishing was explicitly left to the user.