geo/task_plan.deploy-39.md

# Task Plan: Sync Current Workspace to 39

## Goal
Synchronize the complete current workspace state to `39.105.229.239`, deploy it to the existing k3s installation, apply required migrations, and verify service health with a rollback point preserved.

## Current Phase
Complete

## Phases

### Phase 1: Audit
- [x] Read repository deployment instructions and scripts
- [x] Identify remote repository path, runtime configuration, workloads, and current image state
- [x] Confirm the exact local file set and required migrations

### Phase 2: Backup and Sync
- [x] Create a timestamped remote rollback snapshot
- [x] Synchronize the current workspace without overwriting secrets or runtime-owned data
- [x] Verify remote source checksums for representative changed files

### Phase 3: Build and Rollout
- [x] Build and import affected images into k3s
- [x] Apply database migration and manifests/configuration
- [x] Wait for all affected workloads to finish rolling out

### Phase 4: Verification
- [x] Check pods, events, rollout revisions, and recent logs
- [x] Exercise health/API endpoints
- [x] Record deployed image identifiers and rollback instructions

### Phase 5: July 12 Delta Audit
- [x] Compare current clean HEAD with the July 10 deployed source snapshot
- [x] Identify affected images, configuration, and migrations
- [x] Run tests/builds appropriate to the delta

### Phase 6: July 12 Backup and Sync
- [x] Build and checksum target-native release archives
- [x] Capture a new remote rollback point
- [x] Upload and independently verify release artifacts

### Phase 7: July 12 Rollout
- [x] Import release images; migration Job correctly skipped because no migration changed
- [x] Confirm no configuration changes are required
- [x] Roll affected workloads and wait for readiness

### Phase 8: July 12 Verification and Cleanup
- [x] Verify controllers, endpoints, logs, schema, and release contents
- [x] Remove stale application ReplicaSets and unreferenced old application images
- [x] Clear only `geo:cache:*` and preserve Redis operational state

## Decisions
| Decision | Rationale |
|----------|-----------|
| Deploy the whole current workspace | The user explicitly requested syncing the current changes, and the worktree contains several integrated features. |
| Preserve remote secrets and runtime data | Source synchronization must not replace environment-specific credentials, volumes, or cluster state. |
| Create a rollback snapshot before mutation | The source is a dirty worktree and cannot be reconstructed from a single commit. |
| Clear old runtime state only after successful rollout | Preserves rollback and avoids taking down the healthy release before its replacement is ready. |
| Use the July 10 remote source snapshot as the delta baseline | That release included a workspace snapshot, so commit history alone may overstate what is new. |

## Errors
| Error | Attempt | Resolution |
|-------|---------|------------|
| zsh expanded brackets in an unquoted `custom-columns` expression during two read-only inventory commands | 2 | Stop using bracket-bearing custom-columns in outer shell commands; use `get deploy -o wide`, go-template, or per-deployment JSONPath inside a quoted bash script. |
| OrbStack retained only the final amd64 image after sequential cross-architecture `buildx --load` operations, so release inventory failed | 1 | Keep the warm BuildKit cache but rebuild each target directly to its own Docker archive and import archives independently. |
| First planning-file patch used stale table context and did not apply | 1 | Re-read the current files and applied smaller exact-context updates. |
| Redis audit addressed `deployment/redis`, but Redis is a StatefulSet/pod on this cluster | 1 | Retry against `pod/redis-0`; do not issue any deletion until key classes are understood. |
| Broad local config context read exposed adjacent tracked secret-bearing values internally | 1 | Do not repeat or record values; use exact structural queries without config context for the remainder of deployment. |
| Source-snapshot verification changed into the release directory before comparing repository prompt files, so the final two `cmp` checks used the wrong relative path | 1 | The archive and checksum completed successfully; rerun prompt comparisons from the repository root with absolute paths. |
| Workspace checksum manifest was verified from the repository root even though it stores a release-directory-relative filename | 1 | Verify `workspace.sha256` from inside the release artifact directory. |
| macOS bundled rsync does not support `--info=progress2` | 1 | No file transfer started; retry with portable `--progress --stats` options. |
| A guessed migration path plus an unmatched zsh glob prevented a read-only table-name lookup | 1 | Locate the migration with `rg --files` and then read the exact path without shell globbing. |
| The target host does not have `jq`, so ConfigMap merge preparation exited at its explicit prerequisite check | 1 | No patch was applied; generate the one-field JSON merge patch locally with jq, upload it, and use `kubectl patch --patch-file`. |
| Immediate Service curl after the repaired official-placeholder rollout returned connection refused even though rollout status succeeded | 1 | Inspect the new Pod, EndpointSlice, and nginx logs; retry only after confirming whether this was endpoint propagation or a runtime/config issue. |
| Rollback-prompt generation assumed Git HEAD still contained the pre-release prompt, but HEAD already contains the strict prompt rules | 1 | Upload did not start; parse the backed-up pre-release ConfigMap with local kubectl and extract only `data["prompts.yml"]` into the merge patch. |
| Local kubectl still attempted API discovery for a client dry-run of the backed-up ConfigMap and could not reach its configured local cluster | 1 | No patch was produced; use Ruby's standard YAML and JSON parsers on the trusted backup, then verify the extracted prompt against the known old hash before upload. |
| Final verification stopped after HTTP checks because one log-archive command returned nonzero under `set -e` | 1 | Deployment state remains healthy; run each selector log read independently to identify the exact failure, then make the final verifier tolerate empty/no-log sources while still scanning available logs. |
| Two `geo:cache:*` keys were naturally repopulated by live traffic after the verified cleanup, so a later zero-cache assertion stopped final verification | 1 | Complete all remaining read checks first, then repeat the same prefix-scoped deletion as the final action and immediately verify protected Redis key classes are unchanged. |
| July 12 source comparison used `path` as a zsh loop variable, which overwrote the shell's special PATH array and made later commands unavailable | 1 | Archive verification/extraction succeeded and no remote state changed; rerun the comparison with a neutral loop variable such as `item`. |
| A July 12 archive diagnostic assigned to zsh's read-only special variable `status` and exited after printing the correct source commit | 1 | Image tag/platform/content and checksums had already passed; finish source/build-input verification with a neutral variable such as `rc`, without rebuilding. |
| `gzip -dc | git get-tar-commit-id` returned 141 under `pipefail` because Git intentionally stops reading after the PAX commit header and gzip receives SIGPIPE | 2 | Do not repeat the pipeline; decompress once to a temporary tar, read the commit ID via file redirection, then remove the temporary file. |
| The first restored pause-image upload placed the tar at incoming root while its checksum manifest referenced `images/`, so target verification failed before import | 1 | Move the already uploaded tar into `incoming/images/`, reverify SHA-256, then import it; no unverified image was imported. |
| The original frontend rollout SSH client remained open after the remote process disappeared, despite the deployment succeeding | 1 | Confirmed the rollout and new Pod separately, then terminated only the orphaned local SSH client and will repeat the remaining checks read-only. |
| First rollback-guide patch omitted an Add File `+` marker on one line and was rejected before writing | 1 | Reissued the same guide with a syntactically valid patch; no existing file was changed by the failed attempt. |

## Outcome
- Release `workspace-5463319-20260710120256` is fully deployed on 39.
- Migration `20260710183000` is clean and all three media-favorite tables exist.
- All application and infrastructure controllers are Ready; all release Pods have zero restarts.
- Old zero-replica ReplicaSets, unreferenced old geo-rankly images, and `geo:cache:*` were removed while protected Redis/runtime data and rollback artifacts were retained.
- July 12 frontend-only release `workspace-e68db14-20260712150344` is deployed and verified; desktop-client publishing was explicitly left to the user.