geo/progress.deploy-39.md

Progress: Sync Current Workspace to 39

2026-07-10

• Initialized a deployment-specific plan.
• Captured the local base commit and full dirty-worktree scope.
• Deployment documentation and remote topology audit are next.
• Read the k3s deployment contract and confirmed migration execution is image-based through the migrate Job.
• Confirmed local Docker is ARM64 and recorded the need to check the target architecture before building.
• Connected read-only to the target and confirmed it is an x86_64 single-node k3s host with sufficient free disk.
• The first audit command stopped at a zsh glob expansion in custom-columns; the safe retry will avoid bracket syntax.
• Completed the safe remote audit: all pods healthy, main migration clean at 20260708110000, and current image tags captured.
• Selected local linux/amd64 cross-build plus the repository image-import rollout script because the target does not host a source checkout/build engine.
• Reviewed the migration SQL, measured the 313 MB main database, found the actual readiness endpoint, and confirmed the local cross-builder supports amd64.
• Passed backend go test ./..., admin-web production build, and ops-web production build against the release worktree.
• Assigned unique release tag workspace-5463319-20260710120256 and captured pre-build worktree fingerprints.
• All nine build targets completed, but release inventory caught that OrbStack retained only the final cross-loaded image; no remote mutation occurred.
• Switched the transfer plan to direct per-service Docker archives using the now-warm BuildKit cache.
• Rebuilt and checksummed all nine linux/amd64 Docker archives; verified expected tags/platforms and inspected the migration file inside the migration image.
• Created and checksummed a source snapshot for remote release traceability.
• Initial Redis audit used the wrong workload kind and made no changes; selective key-prefix discovery is next.
• Completed Redis classification and limited post-rollout deletion to geo:cache:*; authentication and operational runtime keys will be preserved.
• Reverified all nine rebuilt image archives and the source snapshot against their SHA-256 manifests after the user's local image cleanup.
• Synchronized the stale k3s prompt-config copy with the canonical deployment/server prompt so the new brand and numeric-fidelity constraints reach mounted application config.
• Regenerated the source snapshot after synchronizing the prompt copy; archive checksum passed. A follow-on prompt comparison used release-directory-relative paths and will be rerun from the repository root.
• Reconfirmed the cluster is healthy and the release directory is absent immediately before backup. The remote mounted prompt hash matches the stale k3s copy, confirming that a ConfigMap patch is required.
• Created the timestamped remote release directory and completed the pre-mutation rollback set: eight application Deployments, migrate Job, ReplicaSets, mounted ConfigMap, pod/image inventory, exact active image archives, and a verified compressed main-database dump (895 MB total).
• First artifact upload invocation exited before transfer because the local rsync is too old for --info=progress2; switching to portable progress/statistics flags.
• Uploaded the complete 348 MB incoming release set. All nine image archives, the current workspace snapshot, and the canonical prompt file passed SHA-256 verification independently on 39.
• Imported all nine workspace-5463319-20260710120256 linux/amd64 images into k3s containerd and confirmed all nine tagged refs exist.
• Replaced the completed old migration Job with the release-tagged Job. It completed successfully; main schema is 20260710183000, dirty=false, all three media-favorite tables exist, and platform-template seeding completed.
• The first ConfigMap merge attempt stopped before mutation because jq is absent on 39; the mounted prompt remains on its old checksum. Local jq is available, and the ConfigMap's non-prompt keys are config.yaml and ops-config.yaml.
• Applied a locally generated one-field merge patch to geo-rankly-app-config; both non-prompt values and the key set remained unchanged, while prompts.yml now matches SHA-256 61b768b....
• Rolled all six backend/worker Deployments and both web Deployments to the release tag. Every desired replica is Ready.
• Initial post-rollout checks passed: tenant/ops/browser readiness and both web roots return 200, mounted prompts match in four consumers, all release Pods have zero restarts, and the targeted recent-log scan found no panic/fatal/error candidates.
• Cleanup deleted 80 zero-replica application ReplicaSets, 10 unreferenced old geo-rankly image refs, and exactly three geo:cache:* keys. Redis total keys changed 308 to 305 while 268 refresh and 25 desktop keys were preserved exactly.
• Cleanup inventory exposed a pre-existing missing image record for the still-running official-placeholder Pod; it was not among deleted refs. The deployment will be repointed to the current frontend nginx image and rolled once to prove restartability.
• Repaired official-placeholder with the current frontend nginx image. After one transient Endpoint propagation miss, Service and Pod both return 200 with zero restarts; seven historical zero-replica ReplicaSets were removed.
• Updated the declarative placeholder manifest away from its unavailable hard-coded image and refreshed/reverified the remote source snapshot.
• Investigated the final scheduler log candidates: 22 lines represented 11 records repeated across two five-minute checks. Aggregate SQL showed all candidates were soft-deleted articles, with no missing rows or tenant mismatch. Final log verification will exclude only this exact known anomaly and require every other error class to be zero.
• Adjusted log verification accordingly: all 34 current scheduler candidates are the exact known soft-delete anomaly and unexpected log candidates are zero. A later final check found two shared cache keys had been naturally repopulated after the earlier verified deletion; they will be cleared again only after all endpoint checks finish.
• Completed final verification: 11 Deployments and 6 StatefulSets Ready, migration clean, six HTTP checks at 200, release Pod restarts zero, unexpected log candidates zero, stale zero-replica ReplicaSets zero, nine release image refs present, and no old geo-rankly refs remain.
• Cleared the two naturally repopulated geo:cache:* keys as the last runtime action and immediately verified cache count zero while 270 refresh and 25 desktop keys remained intact.
• Repacked the source snapshot without macOS extended attributes and verified both incoming and rollback checksums. Stored rollback instructions and the old prompt merge patch under the timestamped release directory.