geo/task_plan.deploy-39.md

Task Plan: Sync Current Workspace to 39

Goal

Synchronize the complete current workspace state to 39.105.229.239, deploy it to the existing k3s installation, apply required migrations, and verify service health with a rollback point preserved.

Current Phase

Complete

Phases

Phase 1: Audit

• Read repository deployment instructions and scripts
• Identify remote repository path, runtime configuration, workloads, and current image state
• Confirm the exact local file set and required migrations

Phase 2: Backup and Sync

• Create a timestamped remote rollback snapshot
• Synchronize the current workspace without overwriting secrets or runtime-owned data
• Verify remote source checksums for representative changed files

Phase 3: Build and Rollout

• Build and import affected images into k3s
• Apply database migration and manifests/configuration
• Wait for all affected workloads to finish rolling out

Phase 4: Verification

• Check pods, events, rollout revisions, and recent logs
• Exercise health/API endpoints
• Record deployed image identifiers and rollback instructions

Decisions

Decision	Rationale
Deploy the whole current workspace	The user explicitly requested syncing the current changes, and the worktree contains several integrated features.
Preserve remote secrets and runtime data	Source synchronization must not replace environment-specific credentials, volumes, or cluster state.
Create a rollback snapshot before mutation	The source is a dirty worktree and cannot be reconstructed from a single commit.
Clear old runtime state only after successful rollout	Preserves rollback and avoids taking down the healthy release before its replacement is ready.

Errors

Error	Attempt	Resolution
zsh expanded brackets in an unquoted custom-columns expression during two read-only inventory commands	2	Stop using bracket-bearing custom-columns in outer shell commands; use get deploy -o wide, go-template, or per-deployment JSONPath inside a quoted bash script.
OrbStack retained only the final amd64 image after sequential cross-architecture buildx --load operations, so release inventory failed	1	Keep the warm BuildKit cache but rebuild each target directly to its own Docker archive and import archives independently.
First planning-file patch used stale table context and did not apply	1	Re-read the current files and applied smaller exact-context updates.
Redis audit addressed deployment/redis, but Redis is a StatefulSet/pod on this cluster	1	Retry against pod/redis-0; do not issue any deletion until key classes are understood.
Broad local config context read exposed adjacent tracked secret-bearing values internally	1	Do not repeat or record values; use exact structural queries without config context for the remainder of deployment.
Source-snapshot verification changed into the release directory before comparing repository prompt files, so the final two cmp checks used the wrong relative path	1	The archive and checksum completed successfully; rerun prompt comparisons from the repository root with absolute paths.
Workspace checksum manifest was verified from the repository root even though it stores a release-directory-relative filename	1	Verify workspace.sha256 from inside the release artifact directory.
macOS bundled rsync does not support --info=progress2	1	No file transfer started; retry with portable --progress --stats options.
A guessed migration path plus an unmatched zsh glob prevented a read-only table-name lookup	1	Locate the migration with rg --files and then read the exact path without shell globbing.
The target host does not have jq, so ConfigMap merge preparation exited at its explicit prerequisite check	1	No patch was applied; generate the one-field JSON merge patch locally with jq, upload it, and use kubectl patch --patch-file.
Immediate Service curl after the repaired official-placeholder rollout returned connection refused even though rollout status succeeded	1	Inspect the new Pod, EndpointSlice, and nginx logs; retry only after confirming whether this was endpoint propagation or a runtime/config issue.
Rollback-prompt generation assumed Git HEAD still contained the pre-release prompt, but HEAD already contains the strict prompt rules	1	Upload did not start; parse the backed-up pre-release ConfigMap with local kubectl and extract only data["prompts.yml"] into the merge patch.
Local kubectl still attempted API discovery for a client dry-run of the backed-up ConfigMap and could not reach its configured local cluster	1	No patch was produced; use Ruby's standard YAML and JSON parsers on the trusted backup, then verify the extracted prompt against the known old hash before upload.
Final verification stopped after HTTP checks because one log-archive command returned nonzero under set -e	1	Deployment state remains healthy; run each selector log read independently to identify the exact failure, then make the final verifier tolerate empty/no-log sources while still scanning available logs.
Two geo:cache:* keys were naturally repopulated by live traffic after the verified cleanup, so a later zero-cache assertion stopped final verification	1	Complete all remaining read checks first, then repeat the same prefix-scoped deletion as the final action and immediately verify protected Redis key classes are unchanged.

Outcome

• Release workspace-5463319-20260710120256 is fully deployed on 39.
• Migration 20260710183000 is clean and all three media-favorite tables exist.
• All application and infrastructure controllers are Ready; all release Pods have zero restarts.
• Old zero-replica ReplicaSets, unreferenced old geo-rankly images, and geo:cache:* were removed while protected Redis/runtime data and rollback artifacts were retained.