# Progress: Sync Current Workspace to 39

## 2026-07-10
- Initialized a deployment-specific plan.
- Captured the local base commit and full dirty-worktree scope.
- Deployment documentation and remote topology audit are next.
- Read the k3s deployment contract and confirmed migration execution is image-based through the `migrate` Job.
- Confirmed local Docker is ARM64 and recorded the need to check the target architecture before building.
- Connected read-only to the target and confirmed it is an x86_64 single-node k3s host with sufficient free disk.
- The first audit command stopped at a zsh glob expansion in `custom-columns`; the safe retry will avoid bracket syntax.
- Completed the safe remote audit: all pods healthy, main migration clean at `20260708110000`, and current image tags captured.
- Selected local `linux/amd64` cross-build plus the repository image-import rollout script because the target does not host a source checkout/build engine.
- Reviewed the migration SQL, measured the 313 MB main database, found the actual readiness endpoint, and confirmed the local cross-builder supports amd64.
- Passed backend `go test ./...`, admin-web production build, and ops-web production build against the release worktree.
- Assigned unique release tag `workspace-5463319-20260710120256` and captured pre-build worktree fingerprints.
- All nine build targets completed, but release inventory caught that OrbStack retained only the final cross-loaded image; no remote mutation occurred.
- Switched the transfer plan to direct per-service Docker archives using the now-warm BuildKit cache.
- Rebuilt and checksummed all nine linux/amd64 Docker archives; verified expected tags/platforms and inspected the migration file inside the migration image.
- Created and checksummed a source snapshot for remote release traceability.
- Initial Redis audit used the wrong workload kind and made no changes; selective key-prefix discovery is next.
- Completed Redis classification and limited post-rollout deletion to `geo:cache:*`; authentication and operational runtime keys will be preserved.
- Reverified all nine rebuilt image archives and the source snapshot against their SHA-256 manifests after the user's local image cleanup.
- Synchronized the stale k3s prompt-config copy with the canonical deployment/server prompt so the new brand and numeric-fidelity constraints reach mounted application config.
- Regenerated the source snapshot after synchronizing the prompt copy; archive checksum passed. A follow-on prompt comparison used release-directory-relative paths and will be rerun from the repository root.
- Reconfirmed the cluster is healthy and the release directory is absent immediately before backup. The remote mounted prompt hash matches the stale k3s copy, confirming that a ConfigMap patch is required.
- Created the timestamped remote release directory and completed the pre-mutation rollback set: eight application Deployments, migrate Job, ReplicaSets, mounted ConfigMap, pod/image inventory, exact active image archives, and a verified compressed main-database dump (895 MB total).
- First artifact upload invocation exited before transfer because the local rsync is too old for `--info=progress2`; switching to portable progress/statistics flags.
- Uploaded the complete 348 MB incoming release set. All nine image archives, the current workspace snapshot, and the canonical prompt file passed SHA-256 verification independently on 39.
- Imported all nine `workspace-5463319-20260710120256` linux/amd64 images into k3s containerd and confirmed all nine tagged refs exist.
- Replaced the completed old migration Job with the release-tagged Job. It completed successfully; main schema is `20260710183000`, `dirty=false`, all three media-favorite tables exist, and platform-template seeding completed.
- The first ConfigMap merge attempt stopped before mutation because jq is absent on 39; the mounted prompt remains on its old checksum. Local jq is available, and the ConfigMap's non-prompt keys are `config.yaml` and `ops-config.yaml`.
- Applied a locally generated one-field merge patch to `geo-rankly-app-config`; both non-prompt values and the key set remained unchanged, while `prompts.yml` now matches SHA-256 `61b768b...`.
- Rolled all six backend/worker Deployments and both web Deployments to the release tag. Every desired replica is Ready.
- Initial post-rollout checks passed: tenant/ops/browser readiness and both web roots return 200, mounted prompts match in four consumers, all release Pods have zero restarts, and the targeted recent-log scan found no panic/fatal/error candidates.
- Cleanup deleted 80 zero-replica application ReplicaSets, 10 unreferenced old geo-rankly image refs, and exactly three `geo:cache:*` keys. Redis total keys changed 308 to 305 while 268 refresh and 25 desktop keys were preserved exactly.
- Cleanup inventory exposed a pre-existing missing image record for the still-running `official-placeholder` Pod; it was not among deleted refs. The deployment will be repointed to the current frontend nginx image and rolled once to prove restartability.
- Repaired `official-placeholder` with the current frontend nginx image. After one transient Endpoint propagation miss, Service and Pod both return 200 with zero restarts; seven historical zero-replica ReplicaSets were removed.
- Updated the declarative placeholder manifest away from its unavailable hard-coded image and refreshed/reverified the remote source snapshot.
- Investigated the final scheduler log candidates: 22 lines represented 11 records repeated across two five-minute checks. Aggregate SQL showed all candidates were soft-deleted articles, with no missing rows or tenant mismatch. Final log verification will exclude only this exact known anomaly and require every other error class to be zero.
- Adjusted log verification accordingly: all 34 current scheduler candidates are the exact known soft-delete anomaly and unexpected log candidates are zero. A later final check found two shared cache keys had been naturally repopulated after the earlier verified deletion; they will be cleared again only after all endpoint checks finish.
- Completed final verification: 11 Deployments and 6 StatefulSets Ready, migration clean, six HTTP checks at 200, release Pod restarts zero, unexpected log candidates zero, stale zero-replica ReplicaSets zero, nine release image refs present, and no old geo-rankly refs remain.
- Cleared the two naturally repopulated `geo:cache:*` keys as the last runtime action and immediately verified cache count zero while 270 refresh and 25 desktop keys remained intact.
- Repacked the source snapshot without macOS extended attributes and verified both incoming and rollback checksums. Stored rollback instructions and the old prompt merge patch under the timestamped release directory.

## 2026-07-12
- Started an incremental deployment audit from clean HEAD `e68db14` against the stored July 10 production snapshot.
- Verified and extracted the prior source snapshot. The first comparison command did not run because a zsh special variable shadowed PATH; the retry will use a safe loop variable.
- Confirmed the prior deployed application tree equals `04c89c6`. The July 12 deployable server delta is frontend-only; no SQL migration or backend runtime rollout is required.
- Release tests passed: full Go suite, admin-web Vitest (3/3), and desktop-client Vitest (207/207 across 30 files).
- Admin-web production build, desktop-client typecheck, and desktop-client build all passed. A concurrently running signed desktop packaging process is being left untouched and is outside the k3s rollout.
- Built the frontend directly to a linux/amd64 Docker archive tagged `workspace-e68db14-20260712150344`; tag, architecture, AIAccounts chunk, and SHA checks passed. A zsh diagnostic-variable error interrupted only the final source-commit printout.
- Confirmed the source archive emits the expected commit, but pipefail treated gzip's expected early-reader SIGPIPE as an error; final verification will use a temporary uncompressed tar instead of that pipeline.
- Finished release artifact verification: frontend Docker archive and Git source archive both pass SHA-256, platform is linux/amd64, the expected AIAccounts chunk is present, and all frontend build inputs stayed unchanged during the build.
- Captured an 18 MB frontend-only rollback set on 39, uploaded the 100 MB incoming release, and independently verified both archive checksums on the target.
- Imported the release frontend image. The new Pod initially could not create its sandbox because the node's pause image had been deleted and Docker Hub timed out; restored a checksummed linux/amd64 `rancher/mirrored-pause:3.6`, after which the Pod became Ready with zero restarts and the old Pod terminated.
- Verified the new frontend: root, `/ai-accounts`, and the 14.9 KB AIAccounts chunk all return 200; frontend log candidates and restarts are zero, tenant readiness is 200, and schema remains `20260710183000|false`.
- Completed July 12 cleanup and final verification: 11 Deployments and 6 StatefulSets Ready, one stale frontend ReplicaSet removed, no unreferenced geo-rankly image refs found, pause image retained, and release/rollback checksums pass.
- Cleared exactly two `geo:cache:*` keys as the last application-affecting step; cache remains zero while 265 refresh and 42 desktop runtime keys were preserved. Desktop-client publishing was not touched after the user's instruction.