docs: record Phase 22-35 monitoring scheduler review

Captures findings, task plan, and per-phase progress for the daily
monitoring task generation repair, heartbeat primary lease and
snapshot scalability, scheduler metrics HTTP hardening, worker
graceful shutdown, workspace scoping, and daily plan batching.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

findings.md

@@ -7,6 +7,11 @@
 - For desktop AI monitoring, put scheduling authority on the client, keep scraping silent/background-only, allow stale cross-day tasks to be dropped, and avoid fanning one question to all models at once.
 
 ## Research Findings
+- Current review scope: determine whether the monitoring data detail path and the daily scheduled generation of question collection tasks are working.
+- `cmd/scheduler` starts `ScheduleDispatchWorker`, monitoring result recovery, lease recovery, received inspection, and knowledge cleanup; it does not start a worker that generates daily `monitoring_collect_tasks`.
+- The only production code path found so far that inserts `monitoring_collect_tasks` is `MonitoringService.CollectNow` via `ensureCollectNowTasks`; this writes `trigger_source = 'manual'`.
+- Current local Monitoring PG data confirms recent task creation is manual: on 2026-04-24 there are 3 desktop/manual/plugin_standard tasks; on 2026-04-23 there are 6 desktop/manual/plugin_standard tasks. The latest `automatic` monitoring collect tasks in this local DB are from 2026-04-17, not today.
+- Current local business PG has `tenant_monitoring_quotas.collect_frequency = 'daily'` and `task_daily_hard_cap = 36`, but those fields are configuration only in the code paths inspected so far; no scheduler reads them to generate today's monitoring tasks.
 - The repository already contains core backend scaffolding under `server/`, including `cmd/tenant-api`, `internal/bootstrap`, `shared`, `tenant`, `migrations`, and SQL query files.
 - `server/Makefile` already defines `dev-init`, `dev-api`, `migrate-up`, `sqlc-generate`, `seed`, `lint`, and `test`.
 - The bootstrap layer already exposes `/api/health/live` and `/api/health/ready`.
@@ -149,3 +154,54 @@
 - Later browser verification succeeded with Playwright CLI, and the captured login/workspace screenshots matched the intended layout and data density.
 - Position-aware OCR of the reference images confirmed the sidebar navigation occupies roughly the left 10-12% of the frame, with content titles aligned at the upper-left of the main canvas and most action buttons aligned to the right edge of page headers.
 - The workspace screenshot places stats on the upper-right and the recent-articles table directly beneath the template card block, so the existing all-grid treatment should be tightened into a more asymmetric two-column composition.
+
+## Monitoring Daily Collection Review
+- Data-detail/dashboard reads are working for existing monitoring data: the detail API returned 2026-04-24 question samples, and the dashboard composite showed manual collection progress for brand 3.
+- The daily automatic path was not working before this repair: `cmd/scheduler` started article scheduled generation plus monitoring recovery/inspection workers, but no worker generated `monitoring_collect_tasks` from `tenant_monitoring_quotas.collect_frequency = daily`.
+- Local monitoring DB evidence matched the code gap: 2026-04-24 and 2026-04-23 tasks were manual/plugin-standard; the latest automatic tasks were older 2026-04-17 and 2026-04-10 rows.
+- `MonitoringService.CollectNow` was the only production path found that inserted monitoring collect tasks before this repair, and it writes `trigger_source = manual`.
+- The repair adds `MonitoringDailyTaskWorker` under `internal/tenant/app` and starts it from `cmd/scheduler`.
+- Daily task generation now materializes durable `monitoring_collect_tasks` after midnight using stable jitter, per-run/per-plan/per-brand caps, daily hard-cap enforcement, tenant/brand advisory locks, and the existing idempotent unique key.
+- Desktop offline is now a normal deferred state: automatic daily tasks stay `pending` with `execution_owner = desktop_tasks`; missing live desktop account/client no longer falls back to legacy backend execution.
+- Due desktop dispatch is claimed with `FOR UPDATE SKIP LOCKED`, `last_dispatched_at`, and `dispatch_attempts`, so multiple scheduler instances do not repeatedly fan out the same pending tasks.
+- P0 follow-up: when `desktop_tasks` rollout is disabled or the tenant has no `primary_client_id`, the daily worker now skips the plan instead of inserting `automatic/legacy` rows. This prevents orphan daily tasks that no intended worker will dispatch.
+- Follow-up simplification: Go runtime and dev seed code no longer depend on `tenant_monitoring_quotas`. Daily plans are derived from registered `desktop_clients`; platform coverage uses the canonical default monitoring platform set; dashboard/projection no longer read quota rows; and heartbeat/collect-now no longer write a primary client pointer into that table.
+- P0 follow-up: daily worker RabbitMQ publish no longer uses `context.Background()`. It now derives a 5-second child context from the worker tick context, so RabbitMQ publish is bounded by both the per-publish timeout and the worker's 45-second deadline.
+- P0 follow-up: one tenant/brand failure no longer aborts the whole tick. Plan/brand load and generation failures increment summary failure counts, log scoped fields, and continue unless the worker context is canceled or reaches its deadline.
+- Observability follow-up: the daily worker now records real counters/gauges for completed/failed/partial runs, plan/brand failures, skipped plans, planned/due/created/desktop/deferred task counts, last timestamps, duration, scope counts, business date, and last fatal error code.
+- `cmd/scheduler` now starts a lightweight HTTP server on `scheduler.http_port` default `8081`; `/metrics` returns Prometheus text for scraping and `/api/internal/metrics/monitoring/daily-tasks` returns the JSON snapshot from the scheduler process.
+- Fairness follow-up: daily hard-cap accounting now starts from the durable count of already materialized automatic desktop tasks for that tenant/date, keeps the full hard-cap candidate horizon for existing-key filtering, and only decrements in-memory remaining capacity by rows actually inserted in the current tick. Existing candidates no longer consume later brands' new-task capacity.
+- Product-policy follow-up: the daily worker no longer treats any registered desktop client as eligible. Daily plans now require an active tenant subscription, use subscription/config brand limits, use only live desktop `platform_accounts` as authorized AI platforms, and compute per-brand task capacity from `min(active configured questions, brand_library.max_keywords * max_questions_per_keyword) * authorized_platform_count`.
+- Dashboard/projection follow-up: platform lists no longer backfill the full six-platform catalog. Tenant UI quota resolution is based on the current user's registered desktop client and bound platform accounts; projection platform rows are derived from actual task platforms for that brand/date.
+- Verification passed with targeted Go tests and a live Postgres `EXPLAIN` of the new SQL claim statement.
+- Heartbeat primary follow-up: selecting the primary desktop client by `ORDER BY last_seen_at DESC` is unsafe because the current heartbeat has not touched `last_seen_at` yet and multiple desktop clients can alternate primary. The fix uses a per-tenant/per-workspace `desktop_client_primary_leases` row with a TTL: the incumbent renews its lease, non-primary clients cannot steal a live lease, and failover only happens after lease expiry or primary client revocation.
+- Daily-plan follow-up: daily automatic task generation now prefers the sticky primary lease when choosing `primary_client_id`; if no lease exists or the leased client has no bound platform accounts, it falls back to a deterministic eligible desktop client.
+- UI quota follow-up: current-user registered client lookup no longer sorts by `last_seen_at`; it now uses registration order so heartbeat cadence cannot make platform authorization display flip between the same user's desktop clients.
+- Index follow-up: `desktop_clients` did not have a tenant/workspace/last_seen covering index. A migration now adds tenant/workspace and tenant/workspace/user partial indexes for active desktop-client lookup paths.
+- Scheduler metrics hardening: `/metrics` and `/api/internal/metrics/monitoring/daily-tasks` now require a configured internal metrics token and accept either `Authorization: Bearer <token>` or `X-Internal-Metrics-Token`. If the token is missing, the HTTP metrics server does not start.
+- Scheduler bind hardening: `scheduler.http_host` defaults to `127.0.0.1` instead of `0.0.0.0`; deployments that need pod/network scraping must explicitly set `SCHEDULER_HTTP_HOST=0.0.0.0` or config equivalent and provide `SCHEDULER_INTERNAL_METRICS_TOKEN`.
+- Scheduler shutdown hardening: the metrics server now uses `http.Server` with `Shutdown`, and `ListenAndServe` errors are logged without `Fatalf`, so a port bind failure does not kill all scheduler workers or bypass deferred cleanup.
+- Scheduler disablement: `scheduler.http_port: -1` is preserved as an explicit disable sentinel, while omitted/zero ports still normalize to `8081`.
+- Scheduler worker shutdown follow-up: the scheduler process now starts every long-running worker through a `sync.WaitGroup` and waits on shutdown before returning from `main`, so deferred DB/MQ cleanup does not race active worker goroutines.
+- Worker lifecycle follow-up: scheduler workers expose blocking `Run(ctx)` entrypoints. Their loops still stop when the process context is canceled, but active `runOnce` work uses its own timeout context and is not interrupted immediately by the signal.
+- Authorization-contract follow-up: dashboard runtime and question-detail responses now expose `platform_authorization_status` with `authorized`, `no_desktop_client`, and `no_authorized_platforms`. Dashboard runtime also exposes `authorized_monitoring_platform_count`.
+- Frontend no-auth follow-up: `TrackingView` does not render a dashboard no-authorized/no-desktop alert; it keeps the platform matrix stable and disables collect-now for no authorized platforms and for a selected platform outside the authorized set. `TrackingQuestionDetailView` still renders an explicit unavailable state when detail data has no executable platforms.
+- Browser regression: admin-web was exercised with a no-authorized-platform response. Dashboard rendered all six platform rows, blocked collection, and did not show the removed no-authorized alert; question detail showed the unavailable state and did not render the answer/detail grid.
+- Product display follow-up: the dashboard/projection display contract is now fixed at the canonical six AI platforms (`yuanbao`, `kimi`, `wenxin`, `deepseek`, `doubao`, `qwen`). This is separate from executable platform authorization, so a revoked/offline/unbound platform remains visible with zero samples/pending status instead of disappearing.
+- Authorization execution follow-up: dashboard runtime now returns `authorized_monitoring_platform_ids`; frontend collect-now uses this explicit list to decide whether a selected platform is executable, not the six display rows.
+- Workspace-scope follow-up: monitoring quota/runtime/access-state and collect-now client resolution now use `auth.CurrentWorkspaceID(ctx)` instead of `actor.PrimaryWorkspaceID`, so a future multi-workspace switch can scope desktop clients and platform accounts correctly.
+- Workspace request scope is set by `WorkspaceScope`: absent `X-Workspace-ID` falls back to the actor's primary workspace; a non-primary workspace id must be present in `workspace_memberships` for the current tenant/user.
+- Daily-plan config follow-up: malformed `enabled_platforms` JSON now returns a decode error, logs a scoped Warn with tenant/workspace/business date, increments the plan failure counter through `FailedPlanCount`, and skips only that plan.
+- Heartbeat primary lease follow-up: the primary decision path no longer runs a write transaction or `FOR UPDATE` on every heartbeat. Normal primary/non-primary heartbeats are read-only; writes happen only for missing lease insert, throttled primary renew, expired/unavailable primary claim, or CAS race handling.
+- Heartbeat lease observability now records resolution outcomes, DB write operations, CAS contention, errors, and duration; tenant-api exposes JSON and Prometheus-format internal endpoints under `/api/internal/metrics/monitoring/heartbeat-primary-lease` when `scheduler.internal_metrics_token` is configured.
+- Monitoring heartbeat snapshot follow-up: access snapshots are now handled as semantic state instead of a per-heartbeat liveness write. The heartbeat path batch-loads current platform states, skips fresh unchanged rows, refreshes unchanged rows at most every 10 minutes, and only opens a monitoring DB transaction if a snapshot insert/update/refresh or pending-task reconciliation is actually needed.
+- Inaccessible-platform reconciliation now does a read-only pending-task platform check first. Stable `not_logged_in` / `unavailable` reports with no pending tasks do not open a transaction; if pending rows exist, the primary client updates only those rows to `skipped`.
+- Monitoring heartbeat snapshot observability now tracks reported platforms, opened transactions, snapshot inserts/updates/refreshes/skips, reconciled skipped tasks, projection rebuilds, errors, and duration. Tenant-api exposes token-protected JSON and Prometheus endpoints under `/api/internal/metrics/monitoring/heartbeat-snapshots`.
+- Added a monitoring DB partial index for the reconciliation check: `idx_collect_tasks_pending_platform_reconcile` on `(tenant_id, business_date, ai_platform_id)` where `collector_type = 'desktop'` and `status = 'pending'`.
+- Prometheus exposition follow-up: daily-task, heartbeat-primary-lease, and heartbeat-snapshot Prometheus text output now uses `prometheus/client_golang` collectors plus `promhttp` instead of hand-written `Fprintf` formatting.
+- Scheduler `/metrics` now uses a dedicated Prometheus registry with the custom daily-task collector plus `collectors.NewGoCollector()` and `collectors.NewProcessCollector()`. Process metrics are emitted on supported production OSes such as Linux; local Darwin tests assert runtime metrics and keep process assertion OS-gated.
+- `client_golang` v1.20.5 still does not escape raw carriage returns in label values before exposition on this local toolchain, so the custom collector path normalizes `\r` to a literal `\\r` sequence before passing label values into `ConstMetric`.
+- Daily task materialization follow-up: `ensureDailyMonitoringCollectTasks` no longer loops with one `tx.Exec` per candidate. It now builds arrays of question IDs, question hashes, platform IDs, and dispatch times, then performs one `INSERT ... SELECT FROM unnest(...) ON CONFLICT DO NOTHING` per batch. This preserves idempotency and created-row counting while keeping transaction round trips fixed.
+- Daily plan loading follow-up: `loadDailyMonitoringPlans` no longer uses one SQL statement for lease-aware client election and platform JSON aggregation. It first selects the primary client candidate per `(tenant_id, workspace_id)` with subscription policy and lease preference, then batch-loads authorized platform JSON for the selected `(tenant_id, workspace_id, client_id)` tuples. If authorization disappears between reads, the candidate is skipped rather than producing an empty executable plan.
+- Authorization bug fix: monitoring authorization now means a bound, non-deleted platform account for a supported AI platform. `platform_accounts.health` is treated as execution state, not authorization state, so non-live accounts no longer suppress daily scheduled task generation, dashboard authorization, desktop task targeting, or question-detail history display.
+- Immediate collection remains stricter than daily scheduled collection: it still requires the target desktop client to be online, and the target client is now the same client that owns the authorized platform accounts. Daily scheduled collection still materializes backend tasks while the desktop client is offline and dispatches when the client becomes available.