harden article generation reliability

task_plan.md

@@ -4,7 +4,7 @@
 Continue the desktop AI monitoring implementation by first adding a client-side local scheduler with durable queueing and adaptive execution policy, then adding a reusable hidden Playwright CDP execution layer that attaches to Electron Chromium and reuses existing account partitions.
 
 ## Current Phase
-Phase 41
+Phase 45
 
 ## Phases
 ### Phase 1: Progress Verification
@@ -303,6 +303,18 @@ Phase 41
 - [x] Update findings/progress and summarize completed functions plus any deferred risk
 - **Status:** complete
 
+### Phase 44: Article Generation Reliability Hardening
+- [x] Add structured generation task logs with stable IDs, stages, transitions, retry/recovery context, and error causes
+- [x] Add generation state consistency checks plus alert-ready metrics for task/article/version mismatches
+- [x] Stop swallowing generation status/cleanup errors in worker-facing paths
+- **Status:** complete
+
+### Phase 45: Article Generation Reliability Verification
+- [x] Add targeted unit tests for log fields, metrics, and anomaly classification
+- [x] Run targeted Go tests for tenant generation and worker recovery packages
+- [x] Commit only the reliability hardening files after verification
+- **Status:** complete
+
 ## Key Questions
 1. How should the desktop client defer and locally optimize monitor-task execution without violating one-task-per-platform serialism?
 2. What is the smallest durable local cache that allows same-day resume while safely dropping stale next-day monitor tasks?
@@ -356,6 +368,8 @@ Phase 41
 | Start k3s with a self-contained stack that mirrors the existing compose deployment | The repo already has working Docker images and compose topology; k3s should first preserve service names and runtime assumptions before introducing managed cloud replacements |
 | Publish deployable images only from frontend/backend CI | Deployment config has no runtime image ownership; `deploy-config-ci` stays a validation workflow while frontend/backend CI publish Gitea Registry images tagged by commit8 |
 | Use commit hash first 8 chars as the canonical deploy image tag | CD/offline packaging can compare the pushed commit to Registry tags and fail fast when CI has not produced the matching image |
+| Treat article generation DB state scanning as a low-frequency safety net | Main reliability comes from write-path validation, structured errors, metrics, and alerts; the scheduler check is bounded by interval, timeout, lookback, batch size, and partial indexes so it does not become a DB polling dependency |
+| Expose article generation alert signals through Prometheus metrics and rules | Cleanup failures, state anomalies, state-check failures, queue failures, failure spikes, and recovery finalization should page operators without relying on humans tailing logs |
 
 ## Errors Encountered
 | Error | Attempt | Resolution |
@@ -371,6 +385,8 @@ Phase 41
 | `pnpm --filter @geo/desktop-client typecheck` fails in `electron.vite.config.ts` due to a pre-existing Vite 5/8 plugin type mismatch | 1 | Verified this change with `pnpm --filter @geo/desktop-client build` and `pnpm --filter @geo/desktop-client test`; leave config dependency alignment as a separate fix |
 | `kubectl apply --dry-run=client` still attempted to contact the local kube-apiserver, which is not running at `127.0.0.1:26443` | 1 | Verified with `kubectl kustomize`, parsed the rendered YAML locally, and recorded that live API validation should run on the target k3s cluster |
 | Production compose PgBouncer smoke test could not start bundled Postgres containers because existing `server/docker-compose.yaml` containers already own `geo-postgres` / `geo-monitoring-postgres` names | 1 | Did not stop user/local DB containers; verified PgBouncer with a temporary test container attached to the existing `server_default` network and removed it after `SELECT 1` passed |
+| Article generation state-check worker initially failed to compile because the local DB interface returned a custom rows type instead of `pgx.Rows` | 1 | Changed the interface to match `pgxpool.Pool.Query` directly and reran targeted/full Go tests |
+| Prometheus last-event test depended on label output order | 1 | Changed the assertion to validate required labels independent of exporter ordering |
 
 ## Notes
 - Re-check task_plan.md before major implementation decisions.