geo/progress.deploy-39.md

Progress: Sync Current Workspace to 39

2026-07-10

• Initialized a deployment-specific plan.
• Captured the local base commit and full dirty-worktree scope.
• Deployment documentation and remote topology audit are next.
• Read the k3s deployment contract and confirmed migration execution is image-based through the migrate Job.
• Confirmed local Docker is ARM64 and recorded the need to check the target architecture before building.
• Connected read-only to the target and confirmed it is an x86_64 single-node k3s host with sufficient free disk.
• The first audit command stopped at a zsh glob expansion in custom-columns; the safe retry will avoid bracket syntax.
• Completed the safe remote audit: all pods healthy, main migration clean at 20260708110000, and current image tags captured.
• Selected local linux/amd64 cross-build plus the repository image-import rollout script because the target does not host a source checkout/build engine.
• Reviewed the migration SQL, measured the 313 MB main database, found the actual readiness endpoint, and confirmed the local cross-builder supports amd64.
• Passed backend go test ./..., admin-web production build, and ops-web production build against the release worktree.
• Assigned unique release tag workspace-5463319-20260710120256 and captured pre-build worktree fingerprints.
• All nine build targets completed, but release inventory caught that OrbStack retained only the final cross-loaded image; no remote mutation occurred.
• Switched the transfer plan to direct per-service Docker archives using the now-warm BuildKit cache.
• Rebuilt and checksummed all nine linux/amd64 Docker archives; verified expected tags/platforms and inspected the migration file inside the migration image.
• Created and checksummed a source snapshot for remote release traceability.
• Initial Redis audit used the wrong workload kind and made no changes; selective key-prefix discovery is next.
• Completed Redis classification and limited post-rollout deletion to geo:cache:*; authentication and operational runtime keys will be preserved.
• Reverified all nine rebuilt image archives and the source snapshot against their SHA-256 manifests after the user's local image cleanup.
• Synchronized the stale k3s prompt-config copy with the canonical deployment/server prompt so the new brand and numeric-fidelity constraints reach mounted application config.
• Regenerated the source snapshot after synchronizing the prompt copy; archive checksum passed. A follow-on prompt comparison used release-directory-relative paths and will be rerun from the repository root.
• Reconfirmed the cluster is healthy and the release directory is absent immediately before backup. The remote mounted prompt hash matches the stale k3s copy, confirming that a ConfigMap patch is required.
• Created the timestamped remote release directory and completed the pre-mutation rollback set: eight application Deployments, migrate Job, ReplicaSets, mounted ConfigMap, pod/image inventory, exact active image archives, and a verified compressed main-database dump (895 MB total).
• First artifact upload invocation exited before transfer because the local rsync is too old for --info=progress2; switching to portable progress/statistics flags.
• Uploaded the complete 348 MB incoming release set. All nine image archives, the current workspace snapshot, and the canonical prompt file passed SHA-256 verification independently on 39.
• Imported all nine workspace-5463319-20260710120256 linux/amd64 images into k3s containerd and confirmed all nine tagged refs exist.
• Replaced the completed old migration Job with the release-tagged Job. It completed successfully; main schema is 20260710183000, dirty=false, all three media-favorite tables exist, and platform-template seeding completed.
• The first ConfigMap merge attempt stopped before mutation because jq is absent on 39; the mounted prompt remains on its old checksum. Local jq is available, and the ConfigMap's non-prompt keys are config.yaml and ops-config.yaml.
• Applied a locally generated one-field merge patch to geo-rankly-app-config; both non-prompt values and the key set remained unchanged, while prompts.yml now matches SHA-256 61b768b....
• Rolled all six backend/worker Deployments and both web Deployments to the release tag. Every desired replica is Ready.
• Initial post-rollout checks passed: tenant/ops/browser readiness and both web roots return 200, mounted prompts match in four consumers, all release Pods have zero restarts, and the targeted recent-log scan found no panic/fatal/error candidates.
• Cleanup deleted 80 zero-replica application ReplicaSets, 10 unreferenced old geo-rankly image refs, and exactly three geo:cache:* keys. Redis total keys changed 308 to 305 while 268 refresh and 25 desktop keys were preserved exactly.
• Cleanup inventory exposed a pre-existing missing image record for the still-running official-placeholder Pod; it was not among deleted refs. The deployment will be repointed to the current frontend nginx image and rolled once to prove restartability.
• Repaired official-placeholder with the current frontend nginx image. After one transient Endpoint propagation miss, Service and Pod both return 200 with zero restarts; seven historical zero-replica ReplicaSets were removed.
• Updated the declarative placeholder manifest away from its unavailable hard-coded image and refreshed/reverified the remote source snapshot.
• Investigated the final scheduler log candidates: 22 lines represented 11 records repeated across two five-minute checks. Aggregate SQL showed all candidates were soft-deleted articles, with no missing rows or tenant mismatch. Final log verification will exclude only this exact known anomaly and require every other error class to be zero.
• Adjusted log verification accordingly: all 34 current scheduler candidates are the exact known soft-delete anomaly and unexpected log candidates are zero. A later final check found two shared cache keys had been naturally repopulated after the earlier verified deletion; they will be cleared again only after all endpoint checks finish.
• Completed final verification: 11 Deployments and 6 StatefulSets Ready, migration clean, six HTTP checks at 200, release Pod restarts zero, unexpected log candidates zero, stale zero-replica ReplicaSets zero, nine release image refs present, and no old geo-rankly refs remain.
• Cleared the two naturally repopulated geo:cache:* keys as the last runtime action and immediately verified cache count zero while 270 refresh and 25 desktop keys remained intact.
• Repacked the source snapshot without macOS extended attributes and verified both incoming and rollback checksums. Stored rollback instructions and the old prompt merge patch under the timestamped release directory.

2026-07-12

• Started an incremental deployment audit from clean HEAD e68db14 against the stored July 10 production snapshot.
• Verified and extracted the prior source snapshot. The first comparison command did not run because a zsh special variable shadowed PATH; the retry will use a safe loop variable.
• Confirmed the prior deployed application tree equals 04c89c6. The July 12 deployable server delta is frontend-only; no SQL migration or backend runtime rollout is required.
• Release tests passed: full Go suite, admin-web Vitest (3/3), and desktop-client Vitest (207/207 across 30 files).
• Admin-web production build, desktop-client typecheck, and desktop-client build all passed. A concurrently running signed desktop packaging process is being left untouched and is outside the k3s rollout.
• Built the frontend directly to a linux/amd64 Docker archive tagged workspace-e68db14-20260712150344; tag, architecture, AIAccounts chunk, and SHA checks passed. A zsh diagnostic-variable error interrupted only the final source-commit printout.
• Confirmed the source archive emits the expected commit, but pipefail treated gzip's expected early-reader SIGPIPE as an error; final verification will use a temporary uncompressed tar instead of that pipeline.
• Finished release artifact verification: frontend Docker archive and Git source archive both pass SHA-256, platform is linux/amd64, the expected AIAccounts chunk is present, and all frontend build inputs stayed unchanged during the build.
• Captured an 18 MB frontend-only rollback set on 39, uploaded the 100 MB incoming release, and independently verified both archive checksums on the target.
• Imported the release frontend image. The new Pod initially could not create its sandbox because the node's pause image had been deleted and Docker Hub timed out; restored a checksummed linux/amd64 rancher/mirrored-pause:3.6, after which the Pod became Ready with zero restarts and the old Pod terminated.
• Verified the new frontend: root, /ai-accounts, and the 14.9 KB AIAccounts chunk all return 200; frontend log candidates and restarts are zero, tenant readiness is 200, and schema remains 20260710183000|false.
• Completed July 12 cleanup and final verification: 11 Deployments and 6 StatefulSets Ready, one stale frontend ReplicaSet removed, no unreferenced geo-rankly image refs found, pause image retained, and release/rollback checksums pass.
• Cleared exactly two geo:cache:* keys as the last application-affecting step; cache remains zero while 265 refresh and 42 desktop runtime keys were preserved. Desktop-client publishing was not touched after the user's instruction.