docs: record Phase 22-35 monitoring scheduler review
Captures findings, task plan, and per-phase progress for the daily monitoring task generation repair, heartbeat primary lease and snapshot scalability, scheduler metrics HTTP hardening, worker graceful shutdown, workspace scoping, and daily plan batching. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+56
@@ -7,6 +7,11 @@
|
||||
- For desktop AI monitoring, put scheduling authority on the client, keep scraping silent/background-only, allow stale cross-day tasks to be dropped, and avoid fanning one question to all models at once.
|
||||
|
||||
## Research Findings
|
||||
- Current review scope: determine whether the monitoring data detail path and the daily scheduled generation of question collection tasks are working.
|
||||
- `cmd/scheduler` starts `ScheduleDispatchWorker`, monitoring result recovery, lease recovery, received inspection, and knowledge cleanup; it does not start a worker that generates daily `monitoring_collect_tasks`.
|
||||
- The only production code path found so far that inserts `monitoring_collect_tasks` is `MonitoringService.CollectNow` via `ensureCollectNowTasks`; this writes `trigger_source = 'manual'`.
|
||||
- Current local Monitoring PG data confirms recent task creation is manual: on 2026-04-24 there are 3 desktop/manual/plugin_standard tasks; on 2026-04-23 there are 6 desktop/manual/plugin_standard tasks. The latest `automatic` monitoring collect tasks in this local DB are from 2026-04-17, not today.
|
||||
- Current local business PG has `tenant_monitoring_quotas.collect_frequency = 'daily'` and `task_daily_hard_cap = 36`, but those fields are configuration only in the code paths inspected so far; no scheduler reads them to generate today's monitoring tasks.
|
||||
- The repository already contains core backend scaffolding under `server/`, including `cmd/tenant-api`, `internal/bootstrap`, `shared`, `tenant`, `migrations`, and SQL query files.
|
||||
- `server/Makefile` already defines `dev-init`, `dev-api`, `migrate-up`, `sqlc-generate`, `seed`, `lint`, and `test`.
|
||||
- The bootstrap layer already exposes `/api/health/live` and `/api/health/ready`.
|
||||
@@ -149,3 +154,54 @@
|
||||
- Later browser verification succeeded with Playwright CLI, and the captured login/workspace screenshots matched the intended layout and data density.
|
||||
- Position-aware OCR of the reference images confirmed the sidebar navigation occupies roughly the left 10-12% of the frame, with content titles aligned at the upper-left of the main canvas and most action buttons aligned to the right edge of page headers.
|
||||
- The workspace screenshot places stats on the upper-right and the recent-articles table directly beneath the template card block, so the existing all-grid treatment should be tightened into a more asymmetric two-column composition.
|
||||
|
||||
## Monitoring Daily Collection Review
|
||||
- Data-detail/dashboard reads are working for existing monitoring data: the detail API returned 2026-04-24 question samples, and the dashboard composite showed manual collection progress for brand 3.
|
||||
- The daily automatic path was not working before this repair: `cmd/scheduler` started article scheduled generation plus monitoring recovery/inspection workers, but no worker generated `monitoring_collect_tasks` from `tenant_monitoring_quotas.collect_frequency = daily`.
|
||||
- Local monitoring DB evidence matched the code gap: 2026-04-24 and 2026-04-23 tasks were manual/plugin-standard; the latest automatic tasks were older 2026-04-17 and 2026-04-10 rows.
|
||||
- `MonitoringService.CollectNow` was the only production path found that inserted monitoring collect tasks before this repair, and it writes `trigger_source = manual`.
|
||||
- The repair adds `MonitoringDailyTaskWorker` under `internal/tenant/app` and starts it from `cmd/scheduler`.
|
||||
- Daily task generation now materializes durable `monitoring_collect_tasks` after midnight using stable jitter, per-run/per-plan/per-brand caps, daily hard-cap enforcement, tenant/brand advisory locks, and the existing idempotent unique key.
|
||||
- Desktop offline is now a normal deferred state: automatic daily tasks stay `pending` with `execution_owner = desktop_tasks`; missing live desktop account/client no longer falls back to legacy backend execution.
|
||||
- Due desktop dispatch is claimed with `FOR UPDATE SKIP LOCKED`, `last_dispatched_at`, and `dispatch_attempts`, so multiple scheduler instances do not repeatedly fan out the same pending tasks.
|
||||
- P0 follow-up: when `desktop_tasks` rollout is disabled or the tenant has no `primary_client_id`, the daily worker now skips the plan instead of inserting `automatic/legacy` rows. This prevents orphan daily tasks that no intended worker will dispatch.
|
||||
- Follow-up simplification: Go runtime and dev seed code no longer depend on `tenant_monitoring_quotas`. Daily plans are derived from registered `desktop_clients`; platform coverage uses the canonical default monitoring platform set; dashboard/projection no longer read quota rows; and heartbeat/collect-now no longer write a primary client pointer into that table.
|
||||
- P0 follow-up: daily worker RabbitMQ publish no longer uses `context.Background()`. It now derives a 5-second child context from the worker tick context, so RabbitMQ publish is bounded by both the per-publish timeout and the worker's 45-second deadline.
|
||||
- P0 follow-up: one tenant/brand failure no longer aborts the whole tick. Plan/brand load and generation failures increment summary failure counts, log scoped fields, and continue unless the worker context is canceled or reaches its deadline.
|
||||
- Observability follow-up: the daily worker now records real counters/gauges for completed/failed/partial runs, plan/brand failures, skipped plans, planned/due/created/desktop/deferred task counts, last timestamps, duration, scope counts, business date, and last fatal error code.
|
||||
- `cmd/scheduler` now starts a lightweight HTTP server on `scheduler.http_port` default `8081`; `/metrics` returns Prometheus text for scraping and `/api/internal/metrics/monitoring/daily-tasks` returns the JSON snapshot from the scheduler process.
|
||||
- Fairness follow-up: daily hard-cap accounting now starts from the durable count of already materialized automatic desktop tasks for that tenant/date, keeps the full hard-cap candidate horizon for existing-key filtering, and only decrements in-memory remaining capacity by rows actually inserted in the current tick. Existing candidates no longer consume later brands' new-task capacity.
|
||||
- Product-policy follow-up: the daily worker no longer treats any registered desktop client as eligible. Daily plans now require an active tenant subscription, use subscription/config brand limits, use only live desktop `platform_accounts` as authorized AI platforms, and compute per-brand task capacity from `min(active configured questions, brand_library.max_keywords * max_questions_per_keyword) * authorized_platform_count`.
|
||||
- Dashboard/projection follow-up: platform lists no longer backfill the full six-platform catalog. Tenant UI quota resolution is based on the current user's registered desktop client and bound platform accounts; projection platform rows are derived from actual task platforms for that brand/date.
|
||||
- Verification passed with targeted Go tests and a live Postgres `EXPLAIN` of the new SQL claim statement.
|
||||
- Heartbeat primary follow-up: selecting the primary desktop client by `ORDER BY last_seen_at DESC` is unsafe because the current heartbeat has not touched `last_seen_at` yet and multiple desktop clients can alternate primary. The fix uses a per-tenant/per-workspace `desktop_client_primary_leases` row with a TTL: the incumbent renews its lease, non-primary clients cannot steal a live lease, and failover only happens after lease expiry or primary client revocation.
|
||||
- Daily-plan follow-up: daily automatic task generation now prefers the sticky primary lease when choosing `primary_client_id`; if no lease exists or the leased client has no bound platform accounts, it falls back to a deterministic eligible desktop client.
|
||||
- UI quota follow-up: current-user registered client lookup no longer sorts by `last_seen_at`; it now uses registration order so heartbeat cadence cannot make platform authorization display flip between the same user's desktop clients.
|
||||
- Index follow-up: `desktop_clients` did not have a tenant/workspace/last_seen covering index. A migration now adds tenant/workspace and tenant/workspace/user partial indexes for active desktop-client lookup paths.
|
||||
- Scheduler metrics hardening: `/metrics` and `/api/internal/metrics/monitoring/daily-tasks` now require a configured internal metrics token and accept either `Authorization: Bearer <token>` or `X-Internal-Metrics-Token`. If the token is missing, the HTTP metrics server does not start.
|
||||
- Scheduler bind hardening: `scheduler.http_host` defaults to `127.0.0.1` instead of `0.0.0.0`; deployments that need pod/network scraping must explicitly set `SCHEDULER_HTTP_HOST=0.0.0.0` or config equivalent and provide `SCHEDULER_INTERNAL_METRICS_TOKEN`.
|
||||
- Scheduler shutdown hardening: the metrics server now uses `http.Server` with `Shutdown`, and `ListenAndServe` errors are logged without `Fatalf`, so a port bind failure does not kill all scheduler workers or bypass deferred cleanup.
|
||||
- Scheduler disablement: `scheduler.http_port: -1` is preserved as an explicit disable sentinel, while omitted/zero ports still normalize to `8081`.
|
||||
- Scheduler worker shutdown follow-up: the scheduler process now starts every long-running worker through a `sync.WaitGroup` and waits on shutdown before returning from `main`, so deferred DB/MQ cleanup does not race active worker goroutines.
|
||||
- Worker lifecycle follow-up: scheduler workers expose blocking `Run(ctx)` entrypoints. Their loops still stop when the process context is canceled, but active `runOnce` work uses its own timeout context and is not interrupted immediately by the signal.
|
||||
- Authorization-contract follow-up: dashboard runtime and question-detail responses now expose `platform_authorization_status` with `authorized`, `no_desktop_client`, and `no_authorized_platforms`. Dashboard runtime also exposes `authorized_monitoring_platform_count`.
|
||||
- Frontend no-auth follow-up: `TrackingView` does not render a dashboard no-authorized/no-desktop alert; it keeps the platform matrix stable and disables collect-now for no authorized platforms and for a selected platform outside the authorized set. `TrackingQuestionDetailView` still renders an explicit unavailable state when detail data has no executable platforms.
|
||||
- Browser regression: admin-web was exercised with a no-authorized-platform response. Dashboard rendered all six platform rows, blocked collection, and did not show the removed no-authorized alert; question detail showed the unavailable state and did not render the answer/detail grid.
|
||||
- Product display follow-up: the dashboard/projection display contract is now fixed at the canonical six AI platforms (`yuanbao`, `kimi`, `wenxin`, `deepseek`, `doubao`, `qwen`). This is separate from executable platform authorization, so a revoked/offline/unbound platform remains visible with zero samples/pending status instead of disappearing.
|
||||
- Authorization execution follow-up: dashboard runtime now returns `authorized_monitoring_platform_ids`; frontend collect-now uses this explicit list to decide whether a selected platform is executable, not the six display rows.
|
||||
- Workspace-scope follow-up: monitoring quota/runtime/access-state and collect-now client resolution now use `auth.CurrentWorkspaceID(ctx)` instead of `actor.PrimaryWorkspaceID`, so a future multi-workspace switch can scope desktop clients and platform accounts correctly.
|
||||
- Workspace request scope is set by `WorkspaceScope`: absent `X-Workspace-ID` falls back to the actor's primary workspace; a non-primary workspace id must be present in `workspace_memberships` for the current tenant/user.
|
||||
- Daily-plan config follow-up: malformed `enabled_platforms` JSON now returns a decode error, logs a scoped Warn with tenant/workspace/business date, increments the plan failure counter through `FailedPlanCount`, and skips only that plan.
|
||||
- Heartbeat primary lease follow-up: the primary decision path no longer runs a write transaction or `FOR UPDATE` on every heartbeat. Normal primary/non-primary heartbeats are read-only; writes happen only for missing lease insert, throttled primary renew, expired/unavailable primary claim, or CAS race handling.
|
||||
- Heartbeat lease observability now records resolution outcomes, DB write operations, CAS contention, errors, and duration; tenant-api exposes JSON and Prometheus-format internal endpoints under `/api/internal/metrics/monitoring/heartbeat-primary-lease` when `scheduler.internal_metrics_token` is configured.
|
||||
- Monitoring heartbeat snapshot follow-up: access snapshots are now handled as semantic state instead of a per-heartbeat liveness write. The heartbeat path batch-loads current platform states, skips fresh unchanged rows, refreshes unchanged rows at most every 10 minutes, and only opens a monitoring DB transaction if a snapshot insert/update/refresh or pending-task reconciliation is actually needed.
|
||||
- Inaccessible-platform reconciliation now does a read-only pending-task platform check first. Stable `not_logged_in` / `unavailable` reports with no pending tasks do not open a transaction; if pending rows exist, the primary client updates only those rows to `skipped`.
|
||||
- Monitoring heartbeat snapshot observability now tracks reported platforms, opened transactions, snapshot inserts/updates/refreshes/skips, reconciled skipped tasks, projection rebuilds, errors, and duration. Tenant-api exposes token-protected JSON and Prometheus endpoints under `/api/internal/metrics/monitoring/heartbeat-snapshots`.
|
||||
- Added a monitoring DB partial index for the reconciliation check: `idx_collect_tasks_pending_platform_reconcile` on `(tenant_id, business_date, ai_platform_id)` where `collector_type = 'desktop'` and `status = 'pending'`.
|
||||
- Prometheus exposition follow-up: daily-task, heartbeat-primary-lease, and heartbeat-snapshot Prometheus text output now uses `prometheus/client_golang` collectors plus `promhttp` instead of hand-written `Fprintf` formatting.
|
||||
- Scheduler `/metrics` now uses a dedicated Prometheus registry with the custom daily-task collector plus `collectors.NewGoCollector()` and `collectors.NewProcessCollector()`. Process metrics are emitted on supported production OSes such as Linux; local Darwin tests assert runtime metrics and keep process assertion OS-gated.
|
||||
- `client_golang` v1.20.5 still does not escape raw carriage returns in label values before exposition on this local toolchain, so the custom collector path normalizes `\r` to a literal `\\r` sequence before passing label values into `ConstMetric`.
|
||||
- Daily task materialization follow-up: `ensureDailyMonitoringCollectTasks` no longer loops with one `tx.Exec` per candidate. It now builds arrays of question IDs, question hashes, platform IDs, and dispatch times, then performs one `INSERT ... SELECT FROM unnest(...) ON CONFLICT DO NOTHING` per batch. This preserves idempotency and created-row counting while keeping transaction round trips fixed.
|
||||
- Daily plan loading follow-up: `loadDailyMonitoringPlans` no longer uses one SQL statement for lease-aware client election and platform JSON aggregation. It first selects the primary client candidate per `(tenant_id, workspace_id)` with subscription policy and lease preference, then batch-loads authorized platform JSON for the selected `(tenant_id, workspace_id, client_id)` tuples. If authorization disappears between reads, the candidate is skipped rather than producing an empty executable plan.
|
||||
- Authorization bug fix: monitoring authorization now means a bound, non-deleted platform account for a supported AI platform. `platform_accounts.health` is treated as execution state, not authorization state, so non-live accounts no longer suppress daily scheduled task generation, dashboard authorization, desktop task targeting, or question-detail history display.
|
||||
- Immediate collection remains stricter than daily scheduled collection: it still requires the target desktop client to be online, and the target client is now the same client that owns the authorized platform accounts. Daily scheduled collection still materializes backend tasks while the desktop client is offline and dispatches when the client becomes available.
|
||||
|
||||
+226
@@ -387,6 +387,232 @@
|
||||
| 2026-03-31 | Playwright MCP browser open failed on `/.playwright-mcp` | 1 | Used preview HTTP verification instead of blocking on browser environment setup |
|
||||
| 2026-03-31 | Browser click on the login CTA did not submit the form, so Playwright stayed on `/login` | 1 | Added an explicit `@click=\"handleSubmit\"` binding to the CTA and re-ran the E2E flow successfully |
|
||||
|
||||
### Phase 22: Monitoring Daily Collection Review
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Read the existing planning files and restored context for the monitoring scheduler/runtime work.
|
||||
- Inspected `cmd/scheduler`, the shared cron helper, `ScheduleDispatchWorker`, monitoring handlers, monitoring service, phase2 desktop task dispatch, migrations, and deployment config.
|
||||
- Queried local Postgres/Monitoring Postgres containers for current task evidence.
|
||||
- Verified the data-detail/dashboard read path with existing 2026-04-24 monitoring data.
|
||||
- Findings so far:
|
||||
- Current DB evidence shows 2026-04-24 monitoring collection tasks are `desktop/manual/plugin_standard`, not daily automatic tasks.
|
||||
- `tenant_monitoring_quotas` is configured as `collect_frequency = daily`, but the inspected runtime does not contain a daily monitoring task generation worker.
|
||||
|
||||
### Phase 23: Monitoring Daily Task Generation Repair
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Added `server/internal/tenant/app/monitoring_daily_task_worker.go`.
|
||||
- Wired `MonitoringDailyTaskWorker` into `server/cmd/scheduler/main.go`.
|
||||
- Added `server/internal/tenant/app/monitoring_daily_task_worker_test.go` for midnight-after materialization, existing-task idempotency, and deterministic plan ordering.
|
||||
- Implemented stable jitter after midnight, global/plan/brand batch caps, daily hard-cap selection, and brand-level advisory locking.
|
||||
- Implemented due desktop dispatch claim with `FOR UPDATE SKIP LOCKED`, `last_dispatched_at`, `dispatch_attempts`, and a retry cooldown.
|
||||
- Kept automatic daily desktop tasks pending when no eligible desktop account/client is available; no legacy fallback is applied for this path.
|
||||
- Fixed the P0 rollout-disabled case: daily generation now requires `desktop_tasks` rollout plus `primary_client_id`; otherwise it skips the plan and records `SkippedPlanCount` rather than creating orphan `automatic/legacy` rows.
|
||||
- Removed Go runtime and dev-seed dependency on `tenant_monitoring_quotas`: daily plans now come from `desktop_clients`, dashboard/projection use default monitoring platforms, and primary client selection is derived from recent/registered desktop clients.
|
||||
- Fixed the P0 RabbitMQ publish context issue in daily worker: `publishMonitorDesktopTaskAvailable` now receives a 5-second child context derived from the worker tick context instead of `context.Background()`.
|
||||
- Verification:
|
||||
- `go test ./internal/tenant/app -run 'Test.*MonitoringDaily|Test.*Monitoring|TestEncodeMonitoringProjectionRebuildEnvelope|TestBuildMonitoringRawPayload' -count=1` passed.
|
||||
- `go test ./cmd/scheduler ./cmd/dev-seed ./internal/tenant/app -count=1` passed.
|
||||
- `docker exec geo-monitoring-postgres psql ... EXPLAIN ...` passed for the new claim SQL.
|
||||
|
||||
### Phase 24: Monitoring Daily Observability Hardening
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Completed the plan/brand scoped error-isolation path so transient failures increment summary failure counts and do not stop other tenants/brands in the same tick.
|
||||
- Added `monitoring_daily_task_metrics.go` with atomic counters/gauges for worker runs, fatal and partial failures, plan/brand failures, skipped plans, task stage counts, last timestamps, duration, business date, and last fatal error code/message.
|
||||
- Recorded metrics on every `runOnce` result, including zero-work ticks and fatal errors.
|
||||
- Exposed `/metrics` in Prometheus text format and `/api/internal/metrics/monitoring/daily-tasks` as JSON.
|
||||
- Started the scheduler's health/metrics HTTP server on `scheduler.http_port` default `8081`.
|
||||
- Reworked daily hard-cap accounting so existing materialized tasks are counted from DB, full candidate horizons are preserved for de-duplication, and remaining capacity is decremented only by rows inserted in the current tick.
|
||||
- Split daily materialization limit from desktop dispatch limit, so a full daily cap does not stop already-pending tasks from being dispatched.
|
||||
- Replaced the hard-coded daily plan defaults with subscription/config-driven rules: active subscription required, max brands from plan/config, platforms from live bound desktop accounts, and per-brand task cap from configured active questions capped by `brand_library`.
|
||||
- Changed dashboard and projection platform selection so unbound platforms are not silently shown or scheduled.
|
||||
- Verification:
|
||||
- `go test ./internal/tenant/app -run 'Test.*MonitoringDaily' -count=1` passed.
|
||||
- `go test ./internal/tenant/app -run 'Test.*MonitoringDaily|TestReconcile|TestResolve|TestIntersect|TestFilter' -count=1` passed.
|
||||
- `go test ./cmd/scheduler ./internal/bootstrap ./internal/shared/config -count=1` passed.
|
||||
- `go test ./cmd/scheduler ./cmd/dev-seed ./internal/tenant/app -count=1` passed.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 25: Heartbeat Primary Stability Hardening
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Replaced heartbeat primary resolution with a per-tenant/per-workspace `desktop_client_primary_leases` lease row.
|
||||
- The current primary renews the lease on heartbeat; other desktop clients are non-primary while the incumbent lease is live.
|
||||
- Failover now happens only when the lease has expired or the recorded primary desktop client is unavailable/revoked.
|
||||
- Updated daily monitoring plan selection to prefer the sticky lease primary before deterministic fallback.
|
||||
- Removed `last_seen_at` ordering from the current-user registered-client lookup so UI platform selection is not heartbeat-driven.
|
||||
- Added migration `20260424130000_add_desktop_primary_leases` with the lease table, backfill, and active desktop-client lookup indexes.
|
||||
- Added a unit test for the lease-claim decision matrix.
|
||||
- Verification:
|
||||
- `go test ./internal/tenant/app -run 'TestShouldClaimHeartbeatPrimaryLease|Test.*MonitoringDaily|TestReconcile|TestResolve|TestIntersect|TestFilter' -count=1` passed.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 26: Scheduler Metrics HTTP Hardening
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Added `scheduler.http_host` with default `127.0.0.1`.
|
||||
- Added `scheduler.internal_metrics_token`, with env overrides `SCHEDULER_INTERNAL_METRICS_TOKEN` / `SCHEDULER_METRICS_TOKEN`.
|
||||
- Protected `/metrics` and `/api/internal/metrics/monitoring/daily-tasks` with token auth.
|
||||
- Changed `scheduler.http_port < 0` to mean explicit HTTP disablement; `0` still defaults to `8081`.
|
||||
- Replaced `app.Engine.Run(... Fatalf ...)` with `http.Server.ListenAndServe`, non-fatal error logging, and graceful `Shutdown`.
|
||||
- Added scheduler HTTP auth/address tests and config normalization/env tests.
|
||||
- Verification:
|
||||
- `go test ./cmd/scheduler ./internal/shared/config -count=1` passed.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 27: Scheduler Worker Graceful Shutdown
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Added blocking `Run(ctx)` methods to scheduler workers while keeping existing `Start(ctx)` wrappers.
|
||||
- Updated `cmd/scheduler` to create workers explicitly and run them under a `sync.WaitGroup`.
|
||||
- On signal, scheduler now stops the metrics HTTP server, cancels worker loops, and waits up to 60 seconds for workers to finish.
|
||||
- Changed scheduler worker loops so shutdown cancellation prevents the next tick but does not immediately cancel an active `runOnce`; active work still has the existing per-worker timeout.
|
||||
- Added tests for worker wait completion and timeout behavior.
|
||||
- Verification:
|
||||
- `go test ./cmd/scheduler ./internal/scheduler ./internal/tenant/app -run 'TestScheduler|TestShouldClaimHeartbeatPrimaryLease|Test.*MonitoringDaily' -count=1` passed.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 28: Monitoring Platform Authorization UI Contract
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Added explicit monitoring platform authorization status to dashboard runtime and question-detail API responses.
|
||||
- Added shared frontend type `MonitoringPlatformAuthorizationStatus`.
|
||||
- Updated `TrackingView` to keep dashboard platform rows stable and disable collect-now for no-authorized or selected-unauthorized platform states without rendering a no-authorized dashboard alert.
|
||||
- Updated `TrackingQuestionDetailView` to show an explicit unavailable state and suppress the platform tabs/detail grid when no platforms are authorized.
|
||||
- Added i18n strings for the new no-desktop, no-authorized, and selected-unauthorized states.
|
||||
- Added a Go unit test for the authorization-status decision matrix.
|
||||
- Verification:
|
||||
- `go test ./internal/tenant/app -run 'TestMonitoringPlatformAuthorizationStatus|TestReconcileEnabledMonitoringPlatforms|TestFilterMonitoringPlatforms|TestResolveDashboardDateWindowAt|TestParseDetailDateRangeAt' -count=1` passed.
|
||||
- `pnpm typecheck:admin` passed.
|
||||
- Browser regression against admin-web with `platform_authorization_status = no_authorized_platforms` passed for dashboard and question detail; dashboard does not show a no-authorized alert.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `pnpm build:admin` passed; it still reports existing bundle/CSS warnings unrelated to this change.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 29: Monitoring Six-Platform Display Contract
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Changed `loadMonitoringProjectionPlatforms` to return the canonical six-platform catalog instead of deriving display rows from same-day task rows.
|
||||
- Changed dashboard platform breakdown to use the same six-platform display catalog, so platforms with zero tasks/samples still appear.
|
||||
- Added `authorized_monitoring_platform_ids` to dashboard runtime and shared frontend types.
|
||||
- Updated `TrackingView` collect-now gating to use the explicit authorized platform ID list instead of inferring authorization from visible platform rows.
|
||||
- Added unit coverage for the six-platform catalog copy and projection platform selection.
|
||||
- Verification:
|
||||
- `go test ./internal/tenant/app -run 'TestDefaultMonitoringPlatformMetadata|TestLoadMonitoringProjectionPlatforms|TestMonitoringPlatformAuthorizationStatus|TestReconcileEnabledMonitoringPlatforms|TestFilterMonitoringPlatforms' -count=1` passed.
|
||||
- `pnpm typecheck:admin` passed.
|
||||
- Browser regression confirmed a no-authorized-platform dashboard still renders all 6 platform names and keeps collect-now disabled.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `pnpm build:admin` passed; existing CSS/chunk warnings remain unrelated.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 30: Workspace Scope And Daily Config Visibility
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Added current-workspace context helpers and a `WorkspaceScope` middleware with `X-Workspace-ID` support plus membership validation for non-primary workspace requests.
|
||||
- Wired the tenant route group through `WorkspaceScope`.
|
||||
- Updated monitoring dashboard, question detail, and collect-now paths to use the current request workspace for quota/client/platform-account/access-state resolution.
|
||||
- Documented `findLatestRegisteredClientForUser` as created-at ordered and intentionally independent of online health.
|
||||
- Changed daily worker platform JSON decoding to return errors instead of swallowing malformed JSON.
|
||||
- Counted malformed daily platform config as a plan failure and logged it with tenant/workspace/business-date context.
|
||||
- Added unit coverage for current workspace fallback/override, workspace-id parsing, and daily platform JSON decode failures.
|
||||
- Verification:
|
||||
- `go test ./internal/shared/auth ./internal/shared/middleware ./internal/tenant/app -run 'TestCurrentWorkspaceID|TestWorkspaceScope|TestRequestedWorkspaceID|TestDecodeDailyMonitoringEnabledPlatforms|Test.*MonitoringDaily|TestMonitoringPlatformAuthorizationStatus|TestDefaultMonitoringPlatformMetadata|TestLoadMonitoringProjectionPlatforms' -count=1` passed.
|
||||
- `go test ./internal/tenant/transport -run TestProtectedRoutes -count=1` passed.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 31: Heartbeat Primary Lease Scalability
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Replaced per-heartbeat `BEGIN` / `SELECT ... FOR UPDATE` / `UPDATE` primary lease handling with read-mostly lease resolution.
|
||||
- Normal primary heartbeats now return from a plain read unless the renew interval has elapsed.
|
||||
- Non-primary heartbeats return from a plain read while the incumbent lease is live and available.
|
||||
- Missing, expired, unavailable, and race cases use single-statement `INSERT` or CAS-style `UPDATE` writes instead of an explicit lock transaction.
|
||||
- Added heartbeat primary lease metrics for hit/miss outcomes, DB writes, contention, errors, and duration.
|
||||
- Added token-protected tenant-api internal metrics endpoints for heartbeat primary lease JSON and Prometheus output.
|
||||
- Documented the renew throttle relative to the 25s desktop heartbeat and 90s TTL.
|
||||
- Verification:
|
||||
- `go test ./internal/tenant/app -run 'TestDecideHeartbeatPrimaryLease|TestMonitoringHeartbeatPrimaryLeaseMetrics|TestBuildMonitoringRawPayloadKimi' -count=1` passed.
|
||||
- `go test ./internal/tenant/transport -run 'TestProtectedRoutes|TestPublicRoutes' -count=1` passed.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 32: Monitoring Heartbeat Snapshot Scalability
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Changed `MonitoringCallbackService.Heartbeat` to batch-load current access snapshot state before deciding whether a monitoring DB transaction is required.
|
||||
- Stable unchanged platform access reports now skip snapshot upserts; unchanged rows refresh at most every 10 minutes.
|
||||
- Primary-client inaccessible-platform reconciliation now first checks whether matching pending tasks exist, so stable inaccessible reports with no pending work do not open a write transaction.
|
||||
- Added heartbeat snapshot metrics for platform reports, transaction opens, snapshot insert/update/refresh/skip counts, reconciled tasks, projection rebuilds, errors, and duration.
|
||||
- Exposed heartbeat snapshot metrics through token-protected tenant-api internal JSON and Prometheus endpoints.
|
||||
- Added a monitoring DB partial index for pending-platform reconciliation lookup.
|
||||
- Verification:
|
||||
- `go test ./internal/tenant/app -run 'TestDecideMonitoringHeartbeatSnapshot|TestMonitoringHeartbeatSnapshotMetrics|TestDecideHeartbeatPrimaryLease|TestMonitoringHeartbeatPrimaryLeaseMetrics' -count=1` passed.
|
||||
- `go test ./internal/tenant/app -count=1` passed.
|
||||
- `go test ./internal/tenant/transport -count=1` passed.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 33: Prometheus Metrics Exposition Hardening
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Added `prometheus/client_golang` and replaced custom hand-written Prometheus text builders with `prometheus.Collector` implementations plus `promhttp`.
|
||||
- Kept existing JSON snapshot endpoints unchanged.
|
||||
- Changed scheduler `/metrics` to serve a dedicated registry containing daily-task metrics plus Go runtime and process collectors.
|
||||
- Changed tenant-api internal heartbeat Prometheus endpoints to serve through `promhttp` as well.
|
||||
- Removed raw error-message labels from heartbeat snapshot Prometheus output; detailed messages remain available in JSON snapshots.
|
||||
- Added explicit carriage-return normalization for label values before creating const metrics, because the local client_golang text path does not escape raw CR.
|
||||
- Added tests covering scheduler runtime metrics and CR label normalization.
|
||||
- Verification:
|
||||
- `go test ./internal/tenant/app -run 'TestMonitoringDailyTaskMetrics|TestMonitoringDailyTaskPrometheus|TestMonitoringSchedulerMetricsPrometheusHandler|TestMonitoringHeartbeatPrimaryLeaseMetrics|TestMonitoringHeartbeatSnapshotMetrics' -count=1` passed.
|
||||
- `go test ./cmd/scheduler ./internal/tenant/transport -run 'TestScheduler|TestProtectedRoutes|TestPublicRoutes|TestDesktopClientRoutes' -count=1` passed.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 34: Daily Task Batch Materialization
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Changed `ensureDailyMonitoringCollectTasks` from per-candidate `tx.Exec` to a single batched `INSERT ... SELECT FROM unnest(...)`.
|
||||
- Batched arrays carry `question_id`, `question_hash`, `ai_platform_id`, and `dispatch_after`; tenant/brand/business date/collector/run mode/target client/shard key/execution owner remain constant per call.
|
||||
- Kept `ON CONFLICT DO NOTHING`, so duplicate materialization still returns only the number of newly inserted rows via `RowsAffected`.
|
||||
- Verification:
|
||||
- `go test ./internal/tenant/app -run 'Test.*MonitoringDaily|TestMonitoringDailyTaskMetrics|TestMonitoringDailyTaskPrometheus|TestMonitoringSchedulerMetricsPrometheusHandler' -count=1` passed.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 35: Daily Plan SQL Decomposition
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Split `loadDailyMonitoringPlans` into `loadDailyMonitoringPrimaryClientPlans`, `loadDailyMonitoringPlanPlatformJSON`, and a pure assembly helper.
|
||||
- Kept the first query focused on active subscription admission, lease-preferred primary client selection, workspace scoping, and subscription/config-derived brand/question limits.
|
||||
- Moved authorized platform aggregation to a second batched query keyed by `(tenant_id, workspace_id, client_id)`.
|
||||
- Added a helper test covering the race where a selected client loses its platform snapshot between the two reads; that candidate is skipped instead of becoming an empty executable plan.
|
||||
- Verification:
|
||||
- `go test ./internal/tenant/app -run 'TestBuildDailyMonitoringPlansFromSelectedClients|Test.*MonitoringDaily' -count=1` passed.
|
||||
- `go test ./internal/tenant/app -run 'Test.*MonitoringDaily|TestMonitoringDailyTaskMetrics|TestMonitoringDailyTaskPrometheus|TestMonitoringSchedulerMetricsPrometheusHandler' -count=1` passed.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 36: Monitoring Authorization And Detail Display Bug Fix
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
- Changed monitoring platform authorization to mean a bound, non-deleted platform account on a supported AI platform; `platform_accounts.health` no longer has to be `live`.
|
||||
- Removed `health = 'live'` from daily scheduled plan eligibility and platform aggregation, so daily automatic collection can materialize tasks for offline clients and non-live account states.
|
||||
- Kept immediate collection gated by desktop-client online state, and made the online check target the same desktop client that owns the authorized platform accounts.
|
||||
- Changed question detail to display the canonical platform history instead of hiding existing data behind current `platform_authorization_status`.
|
||||
- Updated frontend date validation to use the monitoring business date, so today's date such as 2026-04-24 is not rejected by local/browser timezone drift.
|
||||
- Verification:
|
||||
- `go test ./internal/tenant/app -run 'Test.*MonitoringDaily|TestMonitoringPlatformAuthorizationStatus|TestReconcileEnabledMonitoringPlatforms|TestFilterMonitoringPlatforms|TestDefaultMonitoringPlatformMetadata|TestLoadMonitoringProjectionPlatforms' -count=1` passed.
|
||||
- `pnpm --filter admin-web typecheck` passed.
|
||||
- `go test ./...` passed from `server/`.
|
||||
- `pnpm --filter admin-web build` passed.
|
||||
- `git diff --check` passed.
|
||||
|
||||
### Phase 21: Foreground Publish Admission
|
||||
- **Status:** complete
|
||||
- Actions taken:
|
||||
|
||||
+139
-1
@@ -4,7 +4,7 @@
|
||||
Continue the desktop AI monitoring implementation by first adding a client-side local scheduler with durable queueing and adaptive execution policy, then adding a reusable hidden Playwright CDP execution layer that attaches to Electron Chromium and reuses existing account partitions.
|
||||
|
||||
## Current Phase
|
||||
Phase 21
|
||||
Phase 36
|
||||
|
||||
## Phases
|
||||
### Phase 1: Progress Verification
|
||||
@@ -141,6 +141,122 @@ Phase 21
|
||||
- [x] Run targeted desktop verification
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 22: Monitoring Daily Collection Review
|
||||
- [x] Trace the scheduler entrypoints that could generate monitoring collect tasks
|
||||
- [x] Inspect current database evidence for automatic vs manual monitoring tasks
|
||||
- [x] Verify desktop dispatch/result/detail query behavior for existing monitoring data
|
||||
- [x] Run targeted compile/tests where practical
|
||||
- [x] Deliver findings on whether the daily scheduled collection path is working
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 23: Monitoring Daily Task Generation Repair
|
||||
- [x] Add a scheduler-owned daily monitoring task worker
|
||||
- [x] Materialize daily tasks after midnight with stable jitter and per-run/per-plan/per-brand caps
|
||||
- [x] Keep desktop-offline tasks pending instead of falling back to backend legacy execution
|
||||
- [x] Claim due desktop dispatch batches with `FOR UPDATE SKIP LOCKED` and retry cooldown
|
||||
- [x] Run targeted Go tests and SQL syntax verification
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 24: Monitoring Daily Observability Hardening
|
||||
- [x] Keep transient tenant/brand errors from aborting the entire worker tick
|
||||
- [x] Add daily worker counters/gauges for runs, failures, skips, task stages, and last-run state
|
||||
- [x] Expose scheduler-local metrics over HTTP for Prometheus scraping and JSON inspection
|
||||
- [x] Align daily hard-cap accounting with durable materialized task counts and actual inserts
|
||||
- [x] Restore product-policy enforcement for subscription limits, brand-library question limits, and user-authorized AI platforms
|
||||
- [x] Run targeted Go verification
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 25: Heartbeat Primary Stability Hardening
|
||||
- [x] Replace heartbeat primary selection by latest `last_seen_at` with a sticky per-workspace lease
|
||||
- [x] Keep the incumbent desktop client primary while its lease is live, and switch only after expiry/revocation
|
||||
- [x] Make daily monitoring plans prefer the sticky primary client before fallback selection
|
||||
- [x] Add desktop-client indexes for tenant/workspace heartbeat and online-client lookup paths
|
||||
- [x] Run targeted and full Go verification
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 26: Scheduler Metrics HTTP Hardening
|
||||
- [x] Stop binding scheduler metrics to all interfaces by default
|
||||
- [x] Require an internal metrics token for `/metrics` and internal JSON metrics
|
||||
- [x] Preserve explicit HTTP disablement with negative `http_port`
|
||||
- [x] Replace `Engine.Run`/`Fatalf` with `http.Server` and graceful shutdown
|
||||
- [x] Add targeted config and scheduler HTTP tests
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 27: Scheduler Worker Graceful Shutdown
|
||||
- [x] Add blocking `Run(ctx)` entrypoints for scheduler workers while preserving `Start(ctx)`
|
||||
- [x] Start scheduler workers under a process-level `sync.WaitGroup`
|
||||
- [x] Stop new ticks on shutdown but allow the current tick to finish under its own timeout
|
||||
- [x] Wait up to a bounded grace period before process exit and DB/MQ cleanup
|
||||
- [x] Add targeted scheduler lifecycle tests
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 28: Monitoring Platform Authorization UI Contract
|
||||
- [x] Expose explicit monitoring platform authorization status in dashboard and question-detail responses
|
||||
- [x] Update admin-web dashboard and question detail pages to render no-desktop/no-authorized-platform states instead of blank platform surfaces
|
||||
- [x] Disable collect-now for no-authorized and selected-unauthorized platform states
|
||||
- [x] Run backend, frontend build, diff, and browser no-authorized-platform regression checks
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 29: Monitoring Six-Platform Display Contract
|
||||
- [x] Restore projection platform rows to the canonical 6-platform display catalog
|
||||
- [x] Make dashboard platform breakdown use the 6-platform display catalog instead of authorized execution platforms
|
||||
- [x] Keep collect-now authorization checks based on explicit authorized platform IDs
|
||||
- [x] Verify dashboard still displays all 6 platforms when no platform is authorized
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 30: Workspace Scope And Daily Config Visibility
|
||||
- [x] Add request-scoped current workspace context with validated `X-Workspace-ID` support
|
||||
- [x] Update monitoring quota/runtime/access-state/collect-now paths to use current workspace instead of actor primary workspace
|
||||
- [x] Document latest-registered client ordering semantics
|
||||
- [x] Make daily monitoring platform JSON decode errors visible through logs and plan-failure metrics
|
||||
- [x] Run targeted backend tests, full `go test ./...`, and `git diff --check`
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 31: Heartbeat Primary Lease Scalability
|
||||
- [x] Remove per-heartbeat primary lease write transaction and row-lock read
|
||||
- [x] Convert primary lease handling to read-mostly plus conditional insert/update paths
|
||||
- [x] Add primary-hit, non-primary-miss, renew/claim, contention, error, and duration metrics
|
||||
- [x] Expose heartbeat primary lease metrics through token-protected tenant-api internal endpoints
|
||||
- [x] Run targeted backend tests, full `go test ./...`, and `git diff --check`
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 32: Monitoring Heartbeat Snapshot Scalability
|
||||
- [x] Avoid opening a monitoring DB transaction for stable heartbeat snapshot reports
|
||||
- [x] Refresh unchanged platform access snapshots at a bounded low frequency
|
||||
- [x] Reconcile inaccessible-platform pending tasks only when pending rows exist
|
||||
- [x] Add heartbeat snapshot write/skip/reconcile/projection metrics
|
||||
- [x] Run targeted backend tests, full `go test ./...`, and `git diff --check`
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 33: Prometheus Metrics Exposition Hardening
|
||||
- [x] Replace hand-written Prometheus text generation with `prometheus/client_golang`
|
||||
- [x] Register Go runtime and process collectors on scheduler `/metrics`
|
||||
- [x] Keep internal JSON metrics endpoints unchanged while serving Prometheus through `promhttp`
|
||||
- [x] Normalize carriage returns in label values before collector emission
|
||||
- [x] Run targeted backend tests, full `go test ./...`, and `git diff --check`
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 34: Daily Task Batch Materialization
|
||||
- [x] Replace per-candidate daily monitoring task INSERT calls with one batch INSERT
|
||||
- [x] Preserve conflict de-duplication, dispatch timing, target client, shard key, and execution-owner semantics
|
||||
- [x] Run targeted backend tests, full `go test ./...`, and `git diff --check`
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 35: Daily Plan SQL Decomposition
|
||||
- [x] Split `loadDailyMonitoringPlans` into primary-client selection and platform-list loading
|
||||
- [x] Keep lease preference, subscription-derived brand limits, workspace scoping, and platform authorization semantics
|
||||
- [x] Add a focused helper test for plan assembly when platform snapshots disappear between queries
|
||||
- [x] Run targeted backend tests, full `go test ./...`, and `git diff --check`
|
||||
- **Status:** complete
|
||||
|
||||
### Phase 36: Monitoring Authorization And Detail Display Bug Fix
|
||||
- [x] Treat bound platform accounts as authorization without requiring `health = 'live'`
|
||||
- [x] Keep daily scheduled task generation independent of desktop-client online state
|
||||
- [x] Keep immediate collection gated by the authorized target desktop client being online
|
||||
- [x] Stop question detail from hiding historical data behind current authorization status
|
||||
- [x] Run targeted backend tests, full `go test ./...`, admin-web typecheck/build, and `git diff --check`
|
||||
- **Status:** complete
|
||||
|
||||
## Key Questions
|
||||
1. How should the desktop client defer and locally optimize monitor-task execution without violating one-task-per-platform serialism?
|
||||
2. What is the smallest durable local cache that allows same-day resume while safely dropping stale next-day monitor tasks?
|
||||
@@ -169,6 +285,28 @@ Phase 21
|
||||
| Treat media publish as foreground and AI monitoring as background | The user explicitly wants publish to have the highest priority and never be starved by monitor execution |
|
||||
| Reserve runtime capacity for publish instead of relying on queue priority alone | Running monitor tasks cannot be preempted by queue ordering, so production-grade priority needs admission control and reserved capacity |
|
||||
| Keep publish scheduling in memory with per-platform locks/cooldowns | Publish needs foreground platform admission but not monitor-style durable recovery, cross-day pruning, or question cooldown semantics |
|
||||
| Generate daily monitoring tasks after midnight with stable jitter and hard caps | The backend needs a durable daily plan without creating a single scheduler spike |
|
||||
| Keep daily automatic monitoring pending when desktop is offline | Offline clients should defer desktop work, not force backend legacy execution |
|
||||
| Claim due desktop dispatch batches in the database | `FOR UPDATE SKIP LOCKED` plus dispatch cooldown prevents multi-instance duplicate dispatch storms |
|
||||
| Skip daily generation when `desktop_tasks` rollout is disabled or no primary client is configured | Creating `automatic/legacy` tasks would be an orphan path and hide a disabled execution channel as successful scheduling |
|
||||
| Do not use `tenant_monitoring_quotas` in current monitoring runtime | Current product scope can derive execution plans from registered desktop clients and use canonical default platform coverage |
|
||||
| Bound daily worker RabbitMQ publish with the worker context plus a per-publish timeout | Scheduler ticks must not leave unbounded goroutines when the broker stalls |
|
||||
| Expose daily-worker metrics from the scheduler process itself | The worker runs in `cmd/scheduler`, so observing only `tenant-api` would still miss cron stalls and generation drops |
|
||||
| Count daily cap against durable materialized tasks, not candidate history | Existing candidates should not consume later brands' new-task budget, while already-pending tasks must still be dispatchable after the cap is full |
|
||||
| Source daily monitoring policy from subscriptions, brand_library config, and bound platform_accounts | Hard-coded `1 brand / 6 platforms / 36 tasks` bypassed SaaS entitlement and user authorization; account `health` is execution state, not authorization state |
|
||||
| Use `desktop_client_primary_leases` for monitoring heartbeat leadership | A TTL lease gives sticky primary semantics without reusing `tenant_monitoring_quotas`, and prevents multiple desktop clients from alternating primary on every heartbeat |
|
||||
| Use negative `scheduler.http_port` as the explicit metrics-server disable sentinel | `0` remains the default-port shorthand while `-1` can intentionally disable scheduler HTTP without being normalized back to `8081` |
|
||||
| Require a scheduler internal metrics token before starting HTTP metrics | Failing closed avoids exposing tenant/brand counts and worker error state through unauthenticated endpoints |
|
||||
| Separate worker lifecycle cancellation from per-tick operation contexts | Signal handling should stop the next tick, but an in-flight DB transaction or RabbitMQ publish should finish or hit its own timeout before the process exits |
|
||||
| Treat no authorized monitoring platforms as an explicit API/UI state | Empty platform arrays are a weak contract; dashboard and detail pages should render a stable state and block collection actions |
|
||||
| Separate monitoring display platforms from executable platforms | Product expects today's 6 AI platform columns to remain visible; task generation and collect-now use bound platform accounts, while collect-now still requires the target desktop client to be online |
|
||||
| Resolve monitoring desktop state from the request-scoped workspace | Multi-workspace SaaS views must not borrow desktop clients or platform account state from the actor's primary workspace |
|
||||
| Keep heartbeat primary lease resolution read-mostly | Stable heartbeat traffic should not create a row-lock/write-transaction stream; writes are reserved for lease lifecycle transitions and throttled renewals |
|
||||
| Treat monitoring access snapshots as semantic state, not heartbeat liveness | Stable platform reports skip snapshot writes; unchanged rows refresh at most every 10 minutes, and pending-task reconciliation only opens a transaction when pending rows exist |
|
||||
| Use `prometheus/client_golang` for Prometheus text exposition | Avoid hand-written text-format escaping/typing drift; scheduler `/metrics` now also registers Go runtime and process collectors while JSON metric snapshots remain unchanged |
|
||||
| Materialize daily monitoring tasks with batched array INSERT | The per-brand cap is currently small, but one `INSERT ... SELECT FROM unnest(...) ON CONFLICT DO NOTHING` keeps DB round trips fixed if caps or retry batches grow |
|
||||
| Split daily monitoring plan loading into selection and aggregation steps | Primary-client election and platform authorization aggregation have different explainability and testability concerns; two smaller queries are easier to inspect and later move to sqlc |
|
||||
| Treat platform-account health as execution state, not authorization state | `health != live` should not suppress daily task planning, dashboard authorization, or detail history; only desktop-client online state gates immediate collection |
|
||||
|
||||
## Errors Encountered
|
||||
| Error | Attempt | Resolution |
|
||||
|
||||
Reference in New Issue
Block a user