root/geo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: record Phase 22-35 monitoring scheduler review

Browse Source 
Captures findings, task plan, and per-phase progress for the daily
monitoring task generation repair, heartbeat primary lease and
snapshot scalability, scheduler metrics HTTP hardening, worker
graceful shutdown, workspace scoping, and daily plan batching.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
liangxu
2026-04-24 22:21:09 +08:00
parent 1b1ab7ee0c
commit bc2e23cfcb
3 changed files with 421 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
task_plan.md
+139 -1
View File
@@ -4,7 +4,7 @@
Continue the desktop AI monitoring implementation by first adding a client-side local scheduler with durable queueing and adaptive execution policy, then adding a reusable hidden Playwright CDP execution layer that attaches to Electron Chromium and reuses existing account partitions.
## Current Phase
Phase 21
Phase 36
## Phases
### Phase 1: Progress Verification
@@ -141,6 +141,122 @@ Phase 21
- [x] Run targeted desktop verification
- **Status:** complete
### Phase 22: Monitoring Daily Collection Review
- [x] Trace the scheduler entrypoints that could generate monitoring collect tasks
- [x] Inspect current database evidence for automatic vs manual monitoring tasks
- [x] Verify desktop dispatch/result/detail query behavior for existing monitoring data
- [x] Run targeted compile/tests where practical
- [x] Deliver findings on whether the daily scheduled collection path is working
- **Status:** complete
### Phase 23: Monitoring Daily Task Generation Repair
- [x] Add a scheduler-owned daily monitoring task worker
- [x] Materialize daily tasks after midnight with stable jitter and per-run/per-plan/per-brand caps
- [x] Keep desktop-offline tasks pending instead of falling back to backend legacy execution
- [x] Claim due desktop dispatch batches with `FOR UPDATE SKIP LOCKED` and retry cooldown
- [x] Run targeted Go tests and SQL syntax verification
- **Status:** complete
### Phase 24: Monitoring Daily Observability Hardening
- [x] Keep transient tenant/brand errors from aborting the entire worker tick
- [x] Add daily worker counters/gauges for runs, failures, skips, task stages, and last-run state
- [x] Expose scheduler-local metrics over HTTP for Prometheus scraping and JSON inspection
- [x] Align daily hard-cap accounting with durable materialized task counts and actual inserts
- [x] Restore product-policy enforcement for subscription limits, brand-library question limits, and user-authorized AI platforms
- [x] Run targeted Go verification
- **Status:** complete
### Phase 25: Heartbeat Primary Stability Hardening
- [x] Replace heartbeat primary selection by latest `last_seen_at` with a sticky per-workspace lease
- [x] Keep the incumbent desktop client primary while its lease is live, and switch only after expiry/revocation
- [x] Make daily monitoring plans prefer the sticky primary client before fallback selection
- [x] Add desktop-client indexes for tenant/workspace heartbeat and online-client lookup paths
- [x] Run targeted and full Go verification
- **Status:** complete
### Phase 26: Scheduler Metrics HTTP Hardening
- [x] Stop binding scheduler metrics to all interfaces by default
- [x] Require an internal metrics token for `/metrics` and internal JSON metrics
- [x] Preserve explicit HTTP disablement with negative `http_port`
- [x] Replace `Engine.Run`/`Fatalf` with `http.Server` and graceful shutdown
- [x] Add targeted config and scheduler HTTP tests
- **Status:** complete
### Phase 27: Scheduler Worker Graceful Shutdown
- [x] Add blocking `Run(ctx)` entrypoints for scheduler workers while preserving `Start(ctx)`
- [x] Start scheduler workers under a process-level `sync.WaitGroup`
- [x] Stop new ticks on shutdown but allow the current tick to finish under its own timeout
- [x] Wait up to a bounded grace period before process exit and DB/MQ cleanup
- [x] Add targeted scheduler lifecycle tests
- **Status:** complete
### Phase 28: Monitoring Platform Authorization UI Contract
- [x] Expose explicit monitoring platform authorization status in dashboard and question-detail responses
- [x] Update admin-web dashboard and question detail pages to render no-desktop/no-authorized-platform states instead of blank platform surfaces
- [x] Disable collect-now for no-authorized and selected-unauthorized platform states
- [x] Run backend, frontend build, diff, and browser no-authorized-platform regression checks
- **Status:** complete
### Phase 29: Monitoring Six-Platform Display Contract
- [x] Restore projection platform rows to the canonical 6-platform display catalog
- [x] Make dashboard platform breakdown use the 6-platform display catalog instead of authorized execution platforms
- [x] Keep collect-now authorization checks based on explicit authorized platform IDs
- [x] Verify dashboard still displays all 6 platforms when no platform is authorized
- **Status:** complete
### Phase 30: Workspace Scope And Daily Config Visibility
- [x] Add request-scoped current workspace context with validated `X-Workspace-ID` support
- [x] Update monitoring quota/runtime/access-state/collect-now paths to use current workspace instead of actor primary workspace
- [x] Document latest-registered client ordering semantics
- [x] Make daily monitoring platform JSON decode errors visible through logs and plan-failure metrics
- [x] Run targeted backend tests, full `go test ./...`, and `git diff --check`
- **Status:** complete
### Phase 31: Heartbeat Primary Lease Scalability
- [x] Remove per-heartbeat primary lease write transaction and row-lock read
- [x] Convert primary lease handling to read-mostly plus conditional insert/update paths
- [x] Add primary-hit, non-primary-miss, renew/claim, contention, error, and duration metrics
- [x] Expose heartbeat primary lease metrics through token-protected tenant-api internal endpoints
- [x] Run targeted backend tests, full `go test ./...`, and `git diff --check`
- **Status:** complete
### Phase 32: Monitoring Heartbeat Snapshot Scalability
- [x] Avoid opening a monitoring DB transaction for stable heartbeat snapshot reports
- [x] Refresh unchanged platform access snapshots at a bounded low frequency
- [x] Reconcile inaccessible-platform pending tasks only when pending rows exist
- [x] Add heartbeat snapshot write/skip/reconcile/projection metrics
- [x] Run targeted backend tests, full `go test ./...`, and `git diff --check`
- **Status:** complete
### Phase 33: Prometheus Metrics Exposition Hardening
- [x] Replace hand-written Prometheus text generation with `prometheus/client_golang`
- [x] Register Go runtime and process collectors on scheduler `/metrics`
- [x] Keep internal JSON metrics endpoints unchanged while serving Prometheus through `promhttp`
- [x] Normalize carriage returns in label values before collector emission
- [x] Run targeted backend tests, full `go test ./...`, and `git diff --check`
- **Status:** complete
### Phase 34: Daily Task Batch Materialization
- [x] Replace per-candidate daily monitoring task INSERT calls with one batch INSERT
- [x] Preserve conflict de-duplication, dispatch timing, target client, shard key, and execution-owner semantics
- [x] Run targeted backend tests, full `go test ./...`, and `git diff --check`
- **Status:** complete
### Phase 35: Daily Plan SQL Decomposition
- [x] Split `loadDailyMonitoringPlans` into primary-client selection and platform-list loading
- [x] Keep lease preference, subscription-derived brand limits, workspace scoping, and platform authorization semantics
- [x] Add a focused helper test for plan assembly when platform snapshots disappear between queries
- [x] Run targeted backend tests, full `go test ./...`, and `git diff --check`
- **Status:** complete
### Phase 36: Monitoring Authorization And Detail Display Bug Fix
- [x] Treat bound platform accounts as authorization without requiring `health = 'live'`
- [x] Keep daily scheduled task generation independent of desktop-client online state
- [x] Keep immediate collection gated by the authorized target desktop client being online
- [x] Stop question detail from hiding historical data behind current authorization status
- [x] Run targeted backend tests, full `go test ./...`, admin-web typecheck/build, and `git diff --check`
- **Status:** complete
## Key Questions
1. How should the desktop client defer and locally optimize monitor-task execution without violating one-task-per-platform serialism?
2. What is the smallest durable local cache that allows same-day resume while safely dropping stale next-day monitor tasks?
@@ -169,6 +285,28 @@ Phase 21
| Treat media publish as foreground and AI monitoring as background | The user explicitly wants publish to have the highest priority and never be starved by monitor execution |
| Reserve runtime capacity for publish instead of relying on queue priority alone | Running monitor tasks cannot be preempted by queue ordering, so production-grade priority needs admission control and reserved capacity |
| Keep publish scheduling in memory with per-platform locks/cooldowns | Publish needs foreground platform admission but not monitor-style durable recovery, cross-day pruning, or question cooldown semantics |
| Generate daily monitoring tasks after midnight with stable jitter and hard caps | The backend needs a durable daily plan without creating a single scheduler spike |
| Keep daily automatic monitoring pending when desktop is offline | Offline clients should defer desktop work, not force backend legacy execution |
| Claim due desktop dispatch batches in the database | `FOR UPDATE SKIP LOCKED` plus dispatch cooldown prevents multi-instance duplicate dispatch storms |
| Skip daily generation when `desktop_tasks` rollout is disabled or no primary client is configured | Creating `automatic/legacy` tasks would be an orphan path and hide a disabled execution channel as successful scheduling |
| Do not use `tenant_monitoring_quotas` in current monitoring runtime | Current product scope can derive execution plans from registered desktop clients and use canonical default platform coverage |
| Bound daily worker RabbitMQ publish with the worker context plus a per-publish timeout | Scheduler ticks must not leave unbounded goroutines when the broker stalls |
| Expose daily-worker metrics from the scheduler process itself | The worker runs in `cmd/scheduler`, so observing only `tenant-api` would still miss cron stalls and generation drops |
| Count daily cap against durable materialized tasks, not candidate history | Existing candidates should not consume later brands' new-task budget, while already-pending tasks must still be dispatchable after the cap is full |
| Source daily monitoring policy from subscriptions, brand_library config, and bound platform_accounts | Hard-coded `1 brand / 6 platforms / 36 tasks` bypassed SaaS entitlement and user authorization; account `health` is execution state, not authorization state |
| Use `desktop_client_primary_leases` for monitoring heartbeat leadership | A TTL lease gives sticky primary semantics without reusing `tenant_monitoring_quotas`, and prevents multiple desktop clients from alternating primary on every heartbeat |
| Use negative `scheduler.http_port` as the explicit metrics-server disable sentinel | `0` remains the default-port shorthand while `-1` can intentionally disable scheduler HTTP without being normalized back to `8081` |
| Require a scheduler internal metrics token before starting HTTP metrics | Failing closed avoids exposing tenant/brand counts and worker error state through unauthenticated endpoints |
| Separate worker lifecycle cancellation from per-tick operation contexts | Signal handling should stop the next tick, but an in-flight DB transaction or RabbitMQ publish should finish or hit its own timeout before the process exits |
| Treat no authorized monitoring platforms as an explicit API/UI state | Empty platform arrays are a weak contract; dashboard and detail pages should render a stable state and block collection actions |
| Separate monitoring display platforms from executable platforms | Product expects today's 6 AI platform columns to remain visible; task generation and collect-now use bound platform accounts, while collect-now still requires the target desktop client to be online |
| Resolve monitoring desktop state from the request-scoped workspace | Multi-workspace SaaS views must not borrow desktop clients or platform account state from the actor's primary workspace |
| Keep heartbeat primary lease resolution read-mostly | Stable heartbeat traffic should not create a row-lock/write-transaction stream; writes are reserved for lease lifecycle transitions and throttled renewals |
| Treat monitoring access snapshots as semantic state, not heartbeat liveness | Stable platform reports skip snapshot writes; unchanged rows refresh at most every 10 minutes, and pending-task reconciliation only opens a transaction when pending rows exist |
| Use `prometheus/client_golang` for Prometheus text exposition | Avoid hand-written text-format escaping/typing drift; scheduler `/metrics` now also registers Go runtime and process collectors while JSON metric snapshots remain unchanged |
| Materialize daily monitoring tasks with batched array INSERT | The per-brand cap is currently small, but one `INSERT ... SELECT FROM unnest(...) ON CONFLICT DO NOTHING` keeps DB round trips fixed if caps or retry batches grow |
| Split daily monitoring plan loading into selection and aggregation steps | Primary-client election and platform authorization aggregation have different explainability and testability concerns; two smaller queries are easier to inspect and later move to sqlc |
| Treat platform-account health as execution state, not authorization state | `health != live` should not suppress daily task planning, dashboard authorization, or detail history; only desktop-client online state gates immediate collection |
## Errors Encountered
| Error | Attempt | Resolution |