docs: record Phase 22-35 monitoring scheduler review

Captures findings, task plan, and per-phase progress for the daily
monitoring task generation repair, heartbeat primary lease and
snapshot scalability, scheduler metrics HTTP hardening, worker
graceful shutdown, workspace scoping, and daily plan batching.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

progress.md

@@ -387,6 +387,232 @@
 | 2026-03-31 | Playwright MCP browser open failed on `/.playwright-mcp` | 1 | Used preview HTTP verification instead of blocking on browser environment setup |
 | 2026-03-31 | Browser click on the login CTA did not submit the form, so Playwright stayed on `/login` | 1 | Added an explicit `@click=\"handleSubmit\"` binding to the CTA and re-ran the E2E flow successfully |
 
+### Phase 22: Monitoring Daily Collection Review
+- **Status:** complete
+- Actions taken:
+  - Read the existing planning files and restored context for the monitoring scheduler/runtime work.
+  - Inspected `cmd/scheduler`, the shared cron helper, `ScheduleDispatchWorker`, monitoring handlers, monitoring service, phase2 desktop task dispatch, migrations, and deployment config.
+  - Queried local Postgres/Monitoring Postgres containers for current task evidence.
+  - Verified the data-detail/dashboard read path with existing 2026-04-24 monitoring data.
+- Findings so far:
+  - Current DB evidence shows 2026-04-24 monitoring collection tasks are `desktop/manual/plugin_standard`, not daily automatic tasks.
+  - `tenant_monitoring_quotas` is configured as `collect_frequency = daily`, but the inspected runtime does not contain a daily monitoring task generation worker.
+
+### Phase 23: Monitoring Daily Task Generation Repair
+- **Status:** complete
+- Actions taken:
+  - Added `server/internal/tenant/app/monitoring_daily_task_worker.go`.
+  - Wired `MonitoringDailyTaskWorker` into `server/cmd/scheduler/main.go`.
+  - Added `server/internal/tenant/app/monitoring_daily_task_worker_test.go` for midnight-after materialization, existing-task idempotency, and deterministic plan ordering.
+  - Implemented stable jitter after midnight, global/plan/brand batch caps, daily hard-cap selection, and brand-level advisory locking.
+  - Implemented due desktop dispatch claim with `FOR UPDATE SKIP LOCKED`, `last_dispatched_at`, `dispatch_attempts`, and a retry cooldown.
+  - Kept automatic daily desktop tasks pending when no eligible desktop account/client is available; no legacy fallback is applied for this path.
+  - Fixed the P0 rollout-disabled case: daily generation now requires `desktop_tasks` rollout plus `primary_client_id`; otherwise it skips the plan and records `SkippedPlanCount` rather than creating orphan `automatic/legacy` rows.
+  - Removed Go runtime and dev-seed dependency on `tenant_monitoring_quotas`: daily plans now come from `desktop_clients`, dashboard/projection use default monitoring platforms, and primary client selection is derived from recent/registered desktop clients.
+  - Fixed the P0 RabbitMQ publish context issue in daily worker: `publishMonitorDesktopTaskAvailable` now receives a 5-second child context derived from the worker tick context instead of `context.Background()`.
+- Verification:
+  - `go test ./internal/tenant/app -run 'Test.*MonitoringDaily|Test.*Monitoring|TestEncodeMonitoringProjectionRebuildEnvelope|TestBuildMonitoringRawPayload' -count=1` passed.
+  - `go test ./cmd/scheduler ./cmd/dev-seed ./internal/tenant/app -count=1` passed.
+  - `docker exec geo-monitoring-postgres psql ... EXPLAIN ...` passed for the new claim SQL.
+
+### Phase 24: Monitoring Daily Observability Hardening
+- **Status:** complete
+- Actions taken:
+  - Completed the plan/brand scoped error-isolation path so transient failures increment summary failure counts and do not stop other tenants/brands in the same tick.
+  - Added `monitoring_daily_task_metrics.go` with atomic counters/gauges for worker runs, fatal and partial failures, plan/brand failures, skipped plans, task stage counts, last timestamps, duration, business date, and last fatal error code/message.
+  - Recorded metrics on every `runOnce` result, including zero-work ticks and fatal errors.
+  - Exposed `/metrics` in Prometheus text format and `/api/internal/metrics/monitoring/daily-tasks` as JSON.
+  - Started the scheduler's health/metrics HTTP server on `scheduler.http_port` default `8081`.
+  - Reworked daily hard-cap accounting so existing materialized tasks are counted from DB, full candidate horizons are preserved for de-duplication, and remaining capacity is decremented only by rows inserted in the current tick.
+  - Split daily materialization limit from desktop dispatch limit, so a full daily cap does not stop already-pending tasks from being dispatched.
+  - Replaced the hard-coded daily plan defaults with subscription/config-driven rules: active subscription required, max brands from plan/config, platforms from live bound desktop accounts, and per-brand task cap from configured active questions capped by `brand_library`.
+  - Changed dashboard and projection platform selection so unbound platforms are not silently shown or scheduled.
+- Verification:
+  - `go test ./internal/tenant/app -run 'Test.*MonitoringDaily' -count=1` passed.
+  - `go test ./internal/tenant/app -run 'Test.*MonitoringDaily|TestReconcile|TestResolve|TestIntersect|TestFilter' -count=1` passed.
+  - `go test ./cmd/scheduler ./internal/bootstrap ./internal/shared/config -count=1` passed.
+  - `go test ./cmd/scheduler ./cmd/dev-seed ./internal/tenant/app -count=1` passed.
+  - `go test ./...` passed from `server/`.
+  - `git diff --check` passed.
+
+### Phase 25: Heartbeat Primary Stability Hardening
+- **Status:** complete
+- Actions taken:
+  - Replaced heartbeat primary resolution with a per-tenant/per-workspace `desktop_client_primary_leases` lease row.
+  - The current primary renews the lease on heartbeat; other desktop clients are non-primary while the incumbent lease is live.
+  - Failover now happens only when the lease has expired or the recorded primary desktop client is unavailable/revoked.
+  - Updated daily monitoring plan selection to prefer the sticky lease primary before deterministic fallback.
+  - Removed `last_seen_at` ordering from the current-user registered-client lookup so UI platform selection is not heartbeat-driven.
+  - Added migration `20260424130000_add_desktop_primary_leases` with the lease table, backfill, and active desktop-client lookup indexes.
+  - Added a unit test for the lease-claim decision matrix.
+- Verification:
+  - `go test ./internal/tenant/app -run 'TestShouldClaimHeartbeatPrimaryLease|Test.*MonitoringDaily|TestReconcile|TestResolve|TestIntersect|TestFilter' -count=1` passed.
+  - `go test ./...` passed from `server/`.
+  - `git diff --check` passed.
+
+### Phase 26: Scheduler Metrics HTTP Hardening
+- **Status:** complete
+- Actions taken:
+  - Added `scheduler.http_host` with default `127.0.0.1`.
+  - Added `scheduler.internal_metrics_token`, with env overrides `SCHEDULER_INTERNAL_METRICS_TOKEN` / `SCHEDULER_METRICS_TOKEN`.
+  - Protected `/metrics` and `/api/internal/metrics/monitoring/daily-tasks` with token auth.
+  - Changed `scheduler.http_port < 0` to mean explicit HTTP disablement; `0` still defaults to `8081`.
+  - Replaced `app.Engine.Run(... Fatalf ...)` with `http.Server.ListenAndServe`, non-fatal error logging, and graceful `Shutdown`.
+  - Added scheduler HTTP auth/address tests and config normalization/env tests.
+- Verification:
+  - `go test ./cmd/scheduler ./internal/shared/config -count=1` passed.
+  - `go test ./...` passed from `server/`.
+  - `git diff --check` passed.
+
+### Phase 27: Scheduler Worker Graceful Shutdown
+- **Status:** complete
+- Actions taken:
+  - Added blocking `Run(ctx)` methods to scheduler workers while keeping existing `Start(ctx)` wrappers.
+  - Updated `cmd/scheduler` to create workers explicitly and run them under a `sync.WaitGroup`.
+  - On signal, scheduler now stops the metrics HTTP server, cancels worker loops, and waits up to 60 seconds for workers to finish.
+  - Changed scheduler worker loops so shutdown cancellation prevents the next tick but does not immediately cancel an active `runOnce`; active work still has the existing per-worker timeout.
+  - Added tests for worker wait completion and timeout behavior.
+- Verification:
+  - `go test ./cmd/scheduler ./internal/scheduler ./internal/tenant/app -run 'TestScheduler|TestShouldClaimHeartbeatPrimaryLease|Test.*MonitoringDaily' -count=1` passed.
+  - `go test ./...` passed from `server/`.
+  - `git diff --check` passed.
+
+### Phase 28: Monitoring Platform Authorization UI Contract
+- **Status:** complete
+- Actions taken:
+  - Added explicit monitoring platform authorization status to dashboard runtime and question-detail API responses.
+  - Added shared frontend type `MonitoringPlatformAuthorizationStatus`.
+  - Updated `TrackingView` to keep dashboard platform rows stable and disable collect-now for no-authorized or selected-unauthorized platform states without rendering a no-authorized dashboard alert.
+  - Updated `TrackingQuestionDetailView` to show an explicit unavailable state and suppress the platform tabs/detail grid when no platforms are authorized.
+  - Added i18n strings for the new no-desktop, no-authorized, and selected-unauthorized states.
+  - Added a Go unit test for the authorization-status decision matrix.
+- Verification:
+  - `go test ./internal/tenant/app -run 'TestMonitoringPlatformAuthorizationStatus|TestReconcileEnabledMonitoringPlatforms|TestFilterMonitoringPlatforms|TestResolveDashboardDateWindowAt|TestParseDetailDateRangeAt' -count=1` passed.
+  - `pnpm typecheck:admin` passed.
+  - Browser regression against admin-web with `platform_authorization_status = no_authorized_platforms` passed for dashboard and question detail; dashboard does not show a no-authorized alert.
+  - `go test ./...` passed from `server/`.
+  - `pnpm build:admin` passed; it still reports existing bundle/CSS warnings unrelated to this change.
+  - `git diff --check` passed.
+
+### Phase 29: Monitoring Six-Platform Display Contract
+- **Status:** complete
+- Actions taken:
+  - Changed `loadMonitoringProjectionPlatforms` to return the canonical six-platform catalog instead of deriving display rows from same-day task rows.
+  - Changed dashboard platform breakdown to use the same six-platform display catalog, so platforms with zero tasks/samples still appear.
+  - Added `authorized_monitoring_platform_ids` to dashboard runtime and shared frontend types.
+  - Updated `TrackingView` collect-now gating to use the explicit authorized platform ID list instead of inferring authorization from visible platform rows.
+  - Added unit coverage for the six-platform catalog copy and projection platform selection.
+- Verification:
+  - `go test ./internal/tenant/app -run 'TestDefaultMonitoringPlatformMetadata|TestLoadMonitoringProjectionPlatforms|TestMonitoringPlatformAuthorizationStatus|TestReconcileEnabledMonitoringPlatforms|TestFilterMonitoringPlatforms' -count=1` passed.
+  - `pnpm typecheck:admin` passed.
+  - Browser regression confirmed a no-authorized-platform dashboard still renders all 6 platform names and keeps collect-now disabled.
+  - `go test ./...` passed from `server/`.
+  - `pnpm build:admin` passed; existing CSS/chunk warnings remain unrelated.
+  - `git diff --check` passed.
+
+### Phase 30: Workspace Scope And Daily Config Visibility
+- **Status:** complete
+- Actions taken:
+  - Added current-workspace context helpers and a `WorkspaceScope` middleware with `X-Workspace-ID` support plus membership validation for non-primary workspace requests.
+  - Wired the tenant route group through `WorkspaceScope`.
+  - Updated monitoring dashboard, question detail, and collect-now paths to use the current request workspace for quota/client/platform-account/access-state resolution.
+  - Documented `findLatestRegisteredClientForUser` as created-at ordered and intentionally independent of online health.
+  - Changed daily worker platform JSON decoding to return errors instead of swallowing malformed JSON.
+  - Counted malformed daily platform config as a plan failure and logged it with tenant/workspace/business-date context.
+  - Added unit coverage for current workspace fallback/override, workspace-id parsing, and daily platform JSON decode failures.
+- Verification:
+  - `go test ./internal/shared/auth ./internal/shared/middleware ./internal/tenant/app -run 'TestCurrentWorkspaceID|TestWorkspaceScope|TestRequestedWorkspaceID|TestDecodeDailyMonitoringEnabledPlatforms|Test.*MonitoringDaily|TestMonitoringPlatformAuthorizationStatus|TestDefaultMonitoringPlatformMetadata|TestLoadMonitoringProjectionPlatforms' -count=1` passed.
+  - `go test ./internal/tenant/transport -run TestProtectedRoutes -count=1` passed.
+  - `go test ./...` passed from `server/`.
+  - `git diff --check` passed.
+
+### Phase 31: Heartbeat Primary Lease Scalability
+- **Status:** complete
+- Actions taken:
+  - Replaced per-heartbeat `BEGIN` / `SELECT ... FOR UPDATE` / `UPDATE` primary lease handling with read-mostly lease resolution.
+  - Normal primary heartbeats now return from a plain read unless the renew interval has elapsed.
+  - Non-primary heartbeats return from a plain read while the incumbent lease is live and available.
+  - Missing, expired, unavailable, and race cases use single-statement `INSERT` or CAS-style `UPDATE` writes instead of an explicit lock transaction.
+  - Added heartbeat primary lease metrics for hit/miss outcomes, DB writes, contention, errors, and duration.
+  - Added token-protected tenant-api internal metrics endpoints for heartbeat primary lease JSON and Prometheus output.
+  - Documented the renew throttle relative to the 25s desktop heartbeat and 90s TTL.
+- Verification:
+  - `go test ./internal/tenant/app -run 'TestDecideHeartbeatPrimaryLease|TestMonitoringHeartbeatPrimaryLeaseMetrics|TestBuildMonitoringRawPayloadKimi' -count=1` passed.
+  - `go test ./internal/tenant/transport -run 'TestProtectedRoutes|TestPublicRoutes' -count=1` passed.
+  - `go test ./...` passed from `server/`.
+  - `git diff --check` passed.
+
+### Phase 32: Monitoring Heartbeat Snapshot Scalability
+- **Status:** complete
+- Actions taken:
+  - Changed `MonitoringCallbackService.Heartbeat` to batch-load current access snapshot state before deciding whether a monitoring DB transaction is required.
+  - Stable unchanged platform access reports now skip snapshot upserts; unchanged rows refresh at most every 10 minutes.
+  - Primary-client inaccessible-platform reconciliation now first checks whether matching pending tasks exist, so stable inaccessible reports with no pending work do not open a write transaction.
+  - Added heartbeat snapshot metrics for platform reports, transaction opens, snapshot insert/update/refresh/skip counts, reconciled tasks, projection rebuilds, errors, and duration.
+  - Exposed heartbeat snapshot metrics through token-protected tenant-api internal JSON and Prometheus endpoints.
+  - Added a monitoring DB partial index for pending-platform reconciliation lookup.
+- Verification:
+  - `go test ./internal/tenant/app -run 'TestDecideMonitoringHeartbeatSnapshot|TestMonitoringHeartbeatSnapshotMetrics|TestDecideHeartbeatPrimaryLease|TestMonitoringHeartbeatPrimaryLeaseMetrics' -count=1` passed.
+  - `go test ./internal/tenant/app -count=1` passed.
+  - `go test ./internal/tenant/transport -count=1` passed.
+  - `go test ./...` passed from `server/`.
+  - `git diff --check` passed.
+
+### Phase 33: Prometheus Metrics Exposition Hardening
+- **Status:** complete
+- Actions taken:
+  - Added `prometheus/client_golang` and replaced custom hand-written Prometheus text builders with `prometheus.Collector` implementations plus `promhttp`.
+  - Kept existing JSON snapshot endpoints unchanged.
+  - Changed scheduler `/metrics` to serve a dedicated registry containing daily-task metrics plus Go runtime and process collectors.
+  - Changed tenant-api internal heartbeat Prometheus endpoints to serve through `promhttp` as well.
+  - Removed raw error-message labels from heartbeat snapshot Prometheus output; detailed messages remain available in JSON snapshots.
+  - Added explicit carriage-return normalization for label values before creating const metrics, because the local client_golang text path does not escape raw CR.
+  - Added tests covering scheduler runtime metrics and CR label normalization.
+- Verification:
+  - `go test ./internal/tenant/app -run 'TestMonitoringDailyTaskMetrics|TestMonitoringDailyTaskPrometheus|TestMonitoringSchedulerMetricsPrometheusHandler|TestMonitoringHeartbeatPrimaryLeaseMetrics|TestMonitoringHeartbeatSnapshotMetrics' -count=1` passed.
+  - `go test ./cmd/scheduler ./internal/tenant/transport -run 'TestScheduler|TestProtectedRoutes|TestPublicRoutes|TestDesktopClientRoutes' -count=1` passed.
+  - `go test ./...` passed from `server/`.
+  - `git diff --check` passed.
+
+### Phase 34: Daily Task Batch Materialization
+- **Status:** complete
+- Actions taken:
+  - Changed `ensureDailyMonitoringCollectTasks` from per-candidate `tx.Exec` to a single batched `INSERT ... SELECT FROM unnest(...)`.
+  - Batched arrays carry `question_id`, `question_hash`, `ai_platform_id`, and `dispatch_after`; tenant/brand/business date/collector/run mode/target client/shard key/execution owner remain constant per call.
+  - Kept `ON CONFLICT DO NOTHING`, so duplicate materialization still returns only the number of newly inserted rows via `RowsAffected`.
+- Verification:
+  - `go test ./internal/tenant/app -run 'Test.*MonitoringDaily|TestMonitoringDailyTaskMetrics|TestMonitoringDailyTaskPrometheus|TestMonitoringSchedulerMetricsPrometheusHandler' -count=1` passed.
+  - `go test ./...` passed from `server/`.
+  - `git diff --check` passed.
+
+### Phase 35: Daily Plan SQL Decomposition
+- **Status:** complete
+- Actions taken:
+  - Split `loadDailyMonitoringPlans` into `loadDailyMonitoringPrimaryClientPlans`, `loadDailyMonitoringPlanPlatformJSON`, and a pure assembly helper.
+  - Kept the first query focused on active subscription admission, lease-preferred primary client selection, workspace scoping, and subscription/config-derived brand/question limits.
+  - Moved authorized platform aggregation to a second batched query keyed by `(tenant_id, workspace_id, client_id)`.
+  - Added a helper test covering the race where a selected client loses its platform snapshot between the two reads; that candidate is skipped instead of becoming an empty executable plan.
+- Verification:
+  - `go test ./internal/tenant/app -run 'TestBuildDailyMonitoringPlansFromSelectedClients|Test.*MonitoringDaily' -count=1` passed.
+  - `go test ./internal/tenant/app -run 'Test.*MonitoringDaily|TestMonitoringDailyTaskMetrics|TestMonitoringDailyTaskPrometheus|TestMonitoringSchedulerMetricsPrometheusHandler' -count=1` passed.
+  - `go test ./...` passed from `server/`.
+  - `git diff --check` passed.
+
+### Phase 36: Monitoring Authorization And Detail Display Bug Fix
+- **Status:** complete
+- Actions taken:
+  - Changed monitoring platform authorization to mean a bound, non-deleted platform account on a supported AI platform; `platform_accounts.health` no longer has to be `live`.
+  - Removed `health = 'live'` from daily scheduled plan eligibility and platform aggregation, so daily automatic collection can materialize tasks for offline clients and non-live account states.
+  - Kept immediate collection gated by desktop-client online state, and made the online check target the same desktop client that owns the authorized platform accounts.
+  - Changed question detail to display the canonical platform history instead of hiding existing data behind current `platform_authorization_status`.
+  - Updated frontend date validation to use the monitoring business date, so today's date such as 2026-04-24 is not rejected by local/browser timezone drift.
+- Verification:
+  - `go test ./internal/tenant/app -run 'Test.*MonitoringDaily|TestMonitoringPlatformAuthorizationStatus|TestReconcileEnabledMonitoringPlatforms|TestFilterMonitoringPlatforms|TestDefaultMonitoringPlatformMetadata|TestLoadMonitoringProjectionPlatforms' -count=1` passed.
+  - `pnpm --filter admin-web typecheck` passed.
+  - `go test ./...` passed from `server/`.
+  - `pnpm --filter admin-web build` passed.
+  - `git diff --check` passed.
+
 ### Phase 21: Foreground Publish Admission
 - **Status:** complete
 - Actions taken: