Files
geo/findings.md
T
root f3679e6176 feat(deploy): add k3s deployment foundation and complete offline images
Introduce a self-contained kustomize stack under deploy/k3s covering
namespace, config, placeholder secrets, infra StatefulSets, app
Deployments/Services, migration and minio-init Jobs, and Traefik
Ingress for both tenant and ops hosts. Add Dockerfile.ops-web so the
operations frontend can ship as an nginx image, and extend the offline
package script to build and bundle ops-api, kol-assist-worker, and
ops-web alongside the k3s manifests. Wire the new ops/security knobs
through .env.example and the compose stack.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 23:19:41 +08:00

42 KiB

Findings & Decisions

Requirements

  • Verify the repository against the 9-step implementation plan provided by the user.
  • Use /Users/liangxu/Documents/test/geo-rankly/docs/superpowers/specs/2026-03-31-admin-web-backend-core-design.md as the source of truth for the next development slice.
  • Continue development immediately after identifying the earliest unfinished step.
  • For desktop AI monitoring, put scheduling authority on the client, keep scraping silent/background-only, allow stale cross-day tasks to be dropped, and avoid fanning one question to all models at once.

Research Findings

  • Current review scope: determine whether the monitoring data detail path and the daily scheduled generation of question collection tasks are working.
  • cmd/scheduler starts ScheduleDispatchWorker, monitoring result recovery, lease recovery, received inspection, and knowledge cleanup; it does not start a worker that generates daily monitoring_collect_tasks.
  • The only production code path found so far that inserts monitoring_collect_tasks is MonitoringService.CollectNow via ensureCollectNowTasks; this writes trigger_source = 'manual'.
  • Current local Monitoring PG data confirms recent task creation is manual: on 2026-04-24 there are 3 desktop/manual/plugin_standard tasks; on 2026-04-23 there are 6 desktop/manual/plugin_standard tasks. The latest automatic monitoring collect tasks in this local DB are from 2026-04-17, not today.
  • Current local business PG has tenant_monitoring_quotas.collect_frequency = 'daily' and task_daily_hard_cap = 36, but those fields are configuration only in the code paths inspected so far; no scheduler reads them to generate today's monitoring tasks.
  • The repository already contains core backend scaffolding under server/, including cmd/tenant-api, internal/bootstrap, shared, tenant, migrations, and SQL query files.
  • server/Makefile already defines dev-init, dev-api, migrate-up, sqlc-generate, seed, lint, and test.
  • The bootstrap layer already exposes /api/health/live and /api/health/ready.
  • The design doc defines a sequential 9-step delivery plan and expects sqlc generated code under internal/tenant/repository/generated/, plus repository wrappers around it.
  • The repository has 19 migration .up.sql files, matching the plan's schema count, and cmd/dev-seed seeds admin@geo.local plus plan/quota/template data.
  • internal/tenant/repository/ currently contains only query SQL, sqlc.yaml, tx.go, and placeholder integration tests; there is no generated/ directory and no repository wrapper implementations.
  • auth_service.go, workspace_service.go, template_service.go, article_service.go, and brand_service.go still issue SQL directly against pgxpool.Pool, which contradicts the design doc's Step 4 target.
  • Running make sqlc-generate currently fails because server/internal/tenant/repository/sqlc.yaml sets schema: "../../migrations/", which resolves to a non-existent path. The correct repository-relative path should point to server/migrations.
  • After fixing sqlc.yaml, make sqlc-generate succeeds and produces typed code in server/internal/tenant/repository/generated/.
  • Repository wrappers now exist for auth, workspace, template, article, quota, and audit queries, and the auth/workspace/template services consume them on the main code path.
  • Refresh token rotation is now atomic in Redis via a Lua script in session_store.go, with tests covering successful rotation and hash mismatch rejection.
  • make test and make lint both pass locally after cleaning up existing lint violations.
  • A GitHub Actions workflow now exists at .github/workflows/backend-ci.yml; it runs sqlc generation, verifies generated code is committed, executes the tenant SQL scope guard, runs tests, and runs golangci-lint.
  • The server/Makefile now exposes tenant-scope-guard, so the tenant filter check is available both locally and in CI.
  • The server/Makefile no longer depends on a preinstalled migrate binary; it now runs golang-migrate via go run -tags postgres, which allowed make dev-init to run successfully on this machine.
  • make dev-init now completes end-to-end: Docker services start, all 19 migrations apply, and seed data creates tenant/user/plan/template records.
  • Runtime verification succeeded for /api/health/live, /api/health/ready, POST /api/auth/login, GET /api/auth/me, POST /api/auth/refresh, and POST /api/auth/logout.
  • Negative auth checks also passed at runtime: the old access token returns 40103 token_revoked after logout, and the old refresh token returns 40121 refresh_session_expired after rotation.
  • Workspace runtime verification passed for all four first-screen endpoints: overview, recent-articles, quota-summary, and template-cards.
  • Template generation runtime verification passed end-to-end: creating a generation task produced an article, article detail returned generate_status=completed, article versions returned one version, and quota balance decreased from 100 to 99.
  • Brand runtime verification passed for create keyword/question/competitor flows, question version history returned two versions after update, and soft-deleted brands can be recreated with the same name.
  • The repository still has no apps/ or packages/ directories, so neither admin-web nor ops-admin-web has been started.
  • The frontend architecture doc recommends pnpm workspace + Vue 3 + TypeScript + ant-design-vue + Vite, with shared packages for UI, shared DTOs, HTTP client, and TS config.
  • The current backend already exposes enough stable endpoints for a first admin-web slice: POST /api/auth/login, GET /api/auth/me, and the four workspace endpoints under /api/tenant/workspace/*.
  • Backend responses use a consistent envelope: code, message, data, and request_id, which makes a shared frontend API client straightforward.
  • Local toolchain support for the frontend is present on this machine: node v22.20.0, npm 10.9.3, and pnpm 10.28.2.
  • A new pnpm workspace now exists at the repo root, with apps/admin-web plus shared packages under packages/shared-types, packages/http-client, and packages/tsconfig.
  • apps/admin-web is now a working Vue 3 + TypeScript + Vite 8 app with Ant Design Vue, Pinia, Vue Router, and Vue Query wired in.
  • The frontend implements a persistent auth session store, login flow, route guard, auto-refresh-capable API client, and a tenant-facing application shell.
  • The workspace page is wired to live backend data through the four dashboard endpoints and renders template cards, stats, quota, and recent articles.
  • Placeholder routes now exist for template creation, custom generation, optimization, media, brands, tracking, and knowledge so the navigation shell is ready for incremental page delivery.
  • pnpm build:admin passes successfully, producing a production build under apps/admin-web/dist/.
  • Static preview verification passed via vite preview and HTTP checks against http://127.0.0.1:4173/login, which returned HTTP/1.1 200 OK plus the built asset references.
  • Gemini CLI was tested but then abandoned for this turn: gemini-3.1-pro-preview consistently returned 429 MODEL_CAPACITY_EXHAUSTED, and the user explicitly requested not to use Gemini afterward.
  • After the user indicated Playwright was fixed, browser-level verification succeeded through the Playwright CLI path rather than the broken MCP path.
  • Browser testing exposed one real frontend bug: the login CTA rendered correctly but did not submit the form on click until it was explicitly bound to handleSubmit.
  • After the login-button fix, a Playwright end-to-end flow successfully loaded /login, signed in with the seeded account, reached /workspace, and verified the presence of the workspace heading plus the template and recent-content sections.
  • The production bundle size improved substantially after switching Ant Design Vue registration from full-plugin mode to per-component registration; the main JS chunk dropped from roughly 1.4 MB to about 747 kB before gzip.
  • The currently exposed tenant-facing backend interfaces fall into four frontend-covered groups: auth, workspace, templates/articles, and brands.
  • The reference screenshots were re-read using position-aware OCR extraction, which made the layout intent clear beyond the raw text labels.
  • docs/refer/工作台.png places the main content in a left-heavy column with template cards above recent articles, while the right side holds the compact stats cluster; the quota/plan card sits in the left sidebar footer rather than the main canvas.
  • docs/refer/模板创作.png uses a top page hero, a horizontal filter/action strip, an article table below it, and a separate template-selection layer before entering a 3-step generation flow.
  • docs/refer/文擎GEO/模版创作-新建模版-创建topx文章-步骤一.png and ...步骤二.png show the generation flow as a wizard: step 1 basic info plus brand/keyword/competitor context, step 2 title and structure selection, step 3 generation.
  • docs/refer/品牌词库.png uses a top brand card rail, then a split content area with keyword navigation on the left and question management on the right; the competitor library lives under a sibling tab.
  • docs/refer/媒体管理.png confirms the overall visual system: page hero under the content header, three-step guide card near the top, and white cards on a cool gray background with the same left navigation shell.
  • The user explicitly asked for i18n and styling discipline during implementation: text should move behind translation keys, and component styles should live in <style> blocks rather than inline attributes.
  • pnpm dev:admin regressed temporarily after the route graph referenced a missing BrandsView.vue; adding the missing view, installing the new frontend dependency set, and rerunning checks restored the Vite startup path.
  • pnpm typecheck:admin now passes after introducing the i18n scaffold, restoring AppShell.vue to the user's preferred version, and removing inline styles from the Vue SFC templates.
  • The currently used hard-coded prompts are concentrated in four runtime areas: cmd/dev-seed platform template seeds, template_prompt.go generation fallback sections, template_assist.go analyze/title/outline fallback prompts, and prompt_generate_service.go plus knowledge_service.go prompt supplements.
  • The seed path stores four full article prompt templates plus twelve wizard prompt templates inside large inline JSON blobs, which makes prompt iteration awkward and easy to miss during later tuning.
  • Runtime prompt assembly still embeds section titles, writing rules, output requirements, output examples, and knowledge-reference instructions directly inside application functions.
  • A dedicated prompt hub now exists under server/internal/tenant/prompts/: template_seeds.go owns the four platform template seeds and their wizard prompt templates, while runtime.go owns generation fallback prompts, assist fallback prompts, prompt-rule supplements, and knowledge prompt instructions.
  • cmd/dev-seed now reads platform templates from prompts.PlatformTemplateSeeds() instead of carrying 400+ lines of inline prompt/card-config text in the command itself.
  • template_prompt.go, template_assist.go, prompt_generate_service.go, and knowledge_service.go now assemble prompts from the centralized prompt package, so runtime business logic no longer embeds the long prompt text directly.
  • The prompt centralization also corrected invalid JSON quoting in the seed card config custom_placeholder fields, which previously contained unescaped inner quotes.
  • Targeted verification passed with go test ./internal/tenant/... ./cmd/dev-seed from server/, confirming the refactor compiles cleanly across the affected runtime and seed paths.
  • desktop-client now includes an AI platform management view with all six target platforms and the existing bind/open/unbind flows wired through the desktop bridge.
  • account-binder.ts already supports all six AI platforms through generic binding definitions, but identity detection is heuristic and not yet platform-specific.
  • The current desktop runtime executes tasks through a simple in-memory FIFO guarded by a single currentTaskId, so all desktop tasks are effectively global-serial today.
  • Monitor-task dropping by business_date is already implemented in the runtime controller; stale tasks are returned as unknown with a client-drop reason instead of being retried on the next day.
  • There is currently no durable local monitor-task queue/cache in desktop-client; if the process exits, queued work only survives on the server side.
  • process-metrics.ts already samples Electron process CPU and memory, which gives us enough local telemetry to drive a small adaptive-concurrency policy without introducing a new metrics dependency.
  • session-registry.ts already persists per-account Electron session partitions to desktop-session-partitions.json, showing that lightweight file-backed persistence is already accepted in this runtime.
  • The only implemented monitor adapter is doubao, and it still uses the existing hidden WebContentsView path rather than Playwright.
  • playwright-core is installed in apps/desktop-client, but there is no connectOverCDP() runtime integration yet.
  • Hidden execution primitives already exist in two forms: hidden BrowserWindow instances for silent account checks and pooled hidden WebContentsView instances for existing adapters.
  • Runtime diagnostics already expose scheduler and process-metrics state through the desktop snapshot, so new scheduler/CDP fields can be surfaced without inventing a new renderer path.
  • A new local monitor scheduler now persists queued monitor tasks to app.getPath("userData")/desktop-monitor-scheduler.json, recovers same-day tasks after restart, and drops stale cross-day tasks during hydration.
  • The new scheduler is intentionally conservative when a task has no question-level metadata: unknown-question monitor tasks do not consume the second concurrency slot, which avoids accidental same-question fan-out until server event metadata is available.
  • Electron process metrics are sufficient to drive a two-tier adaptive concurrency policy: default to 1, raise to 2 only when the machine looks healthy enough and multiple live AI accounts exist.
  • Desktop task SSE events now carry optional business_date, scheduler_group_key, question_text, and title metadata derived from the task payload, allowing the local scheduler to defer same-question fan-out before leasing.
  • A new hidden Playwright manager now connects to Electron Chromium through a local CDP endpoint and leases hidden pages bound to the existing account session partition, while leaving the legacy hidden-view path available for current adapters.
  • The browser-extension qwen adapter does not rely on stable DOM selectors; it drives Tongyi's internal text/chat managers from __TONGYI_PORTSL_GLOBAL_CONTAINER__ and reads results from __qianwenChatAPI, which makes Playwright page evaluation a better desktop port than raw HTTP.
  • Qwen binding was previously too strict for desktop because the generic AI bind flow waited for obvious post-login UI signals like avatar/name; in practice Qwen can already have a durable session footprint before those signals render, so the bind window may fail to auto-close.
  • Anonymous curl inspection of https://www.qianwen.com/ currently returns an XSRF-TOKEN cookie even before login, so CSRF-only cookies should not count as authenticated AI session evidence.
  • On this machine's real pending-qwen-* partition, a logged-in Qwen session persists cookies including tongyi_sso_ticket, tongyi_sso_ticket_hash, and b-user-id; these are materially stronger auth signals than generic DOM nickname/avatar detection.
  • The user clarified that only yuanbao, kimi, and deepseek require login for monitor execution; other AI platforms may still ask questions and collect responses anonymously, so preflight auth gating must not block those monitor tasks.
  • The admin-web tracking page still used the removed browser-plugin monitoring bridge for 立即采集, even though the current monitoring architecture is desktop-client driven and runs silently in the background.
  • The user clarified that 立即采集 must only be available when the current logged-in user's own desktop client is online; another user's online client in the same workspace must not make the action available.
  • MonitoringService.CollectNow previously fell back to the most recently online workspace client regardless of user_id, so the backend rule did not match the desired UI rule until this turn.
  • Media publishing is now treated as the desktop runtime foreground channel, while AI monitoring remains background/best-effort. This is implemented through admission control rather than queue ordering alone.
  • A new in-memory publish-scheduler.ts owns publish queue selection, per-platform active locks, and 3-6 second post-completion platform cooldowns. It intentionally does not persist state or copy monitor scheduler question/cross-day semantics.
  • runtime-controller.ts now uses per-kind lease-in-flight tracking, so a monitor lease request no longer blocks a publish lease request at the runtime-controller guard.
  • Total runtime concurrency is now derived from hardware class, current Electron process health, and optional GEO_DESKTOP_MAX_TOTAL_CONCURRENCY, while monitor capacity is capped to leave a reserved foreground publish slot.
  • When publish is queued or publish lease is in flight, monitor admission returns zero; even a same-platform/cooldown-blocked publish backlog prevents monitor from consuming the publish reserve.
  • Desktop runtime diagnostics now expose publishScheduler alongside monitorScheduler in the runtime snapshot.

Technical Decisions

Decision Rationale
Verify by code inspection first, then run targeted commands Faster way to establish step coverage before making changes
Use the design doc as the acceptance reference when repository behavior is ambiguous User explicitly pointed to this document for next-step development
Treat Step 4 as the current development target unless command verification disproves it Earlier steps have visible artifacts, but repository/sqlc acceptance is not yet met
Fix sqlc generation and introduce repository wrappers before touching later roadmap items This is the first broken acceptance gate and an architectural dependency for the service layer
Close Step 9 by adding repository-scoped CI rather than a repo-wide generic pipeline The remaining missing acceptance item was backend CI plus the tenant scope guard
Resume into admin-web instead of ops-admin-web Only tenant-facing APIs exist in the repository today, so this is the only frontend slice that can be meaningfully wired end to end
Keep the initial frontend page set broad in navigation but shallow in implementation This gives the repo a usable admin shell now without inventing missing backend modules for later pages
Use templates/articles and brands as the primary implementation targets for this turn They are the remaining backend-backed pages still missing real frontend coverage
Use screenshot-derived spatial structure as an implementation constraint Needed to satisfy the user's requirement to follow layout and placement, not just labels
Introduce a minimal vue-i18n foundation now instead of leaving hard-coded strings in new pages The user explicitly called out the current i18n approach as non-standard
Keep AppShell.vue on the user-preferred version and avoid structural/style rewrites there The user said that file is managed elsewhere and only allowed inline-style cleanup
Create a dedicated internal/tenant/prompts package for prompt text and seed definitions This keeps prompt tuning in one place while minimizing refactor risk in the current dirty worktree
Add the local scheduler before wiring more monitor adapters This aligns with the user's requirement that the client, not the server, decides actual execution timing
Keep publish-task execution behavior intact while evolving monitor-task scheduling Publish is already working and should not be destabilized by monitor-specific policy changes
Build the hidden Playwright layer as reusable infrastructure instead of burying it inside one adapter Qwen is the immediate motivation, but the same layer will likely be needed by other complex AI platforms
Port Qwen by evaluating in-page managers/state instead of reverse-engineering private APIs The existing extension path already proved this is the stable way to drive Qwen's encrypted web client
Exclude CSRF-only cookie names from generic AI auth fingerprinting Qwen sets XSRF-TOKEN on anonymous visits, which would otherwise create false-positive auth evidence
Only enforce active auth preflight for monitor tasks on yuanbao, kimi, and deepseek The user explicitly allowed anonymous execution for the other AI monitoring platforms
Gate tracking collect-now on the current actor's online desktop client, not any workspace client The user explicitly wants the action tied to the current logged-in account's client presence
Expose current-user desktop-client availability through the monitoring dashboard response TrackingView already loads the dashboard, so returning the runtime bit there avoids a second frontend probe and keeps button gating aligned with backend validation
Implement publish priority through runtime admission control, not just queue sort order Already-running monitor tasks cannot be moved by queue priority; publish needs reserved execution capacity to avoid starvation
Keep monitor hard preemption out of the first publish-priority slice Soft priority with reserved capacity avoids browser half-submit and lease/result inconsistency while still preventing monitor saturation

Issues Encountered

Issue Resolution
Repository is fully untracked in git status output Avoid relying on git history; verify via filesystem and test commands instead
make sqlc-generate fails before any code is generated Repair the sqlc.yaml schema path and then build repository wrappers around generated queries
go run migrate initially failed with unknown driver postgres Added -tags postgres to the Makefile-managed migrate command
Playwright MCP could not open a browser session locally because it tried to create /.playwright-mcp Use vite preview plus curl verification for this turn instead of spending time on browser tool plumbing
desktop-client has no existing SQLite/local DB layer despite better-sqlite3 being installed Decide between a tiny file-backed queue or adding a new DB-backed task store during implementation

Resources

  • /Users/liangxu/Documents/test/geo-rankly/docs/superpowers/specs/2026-03-31-admin-web-backend-core-design.md
  • /Users/liangxu/Documents/test/geo-rankly/server/Makefile
  • /Users/liangxu/Documents/test/geo-rankly/server/cmd/tenant-api/main.go
  • /Users/liangxu/Documents/test/geo-rankly/server/internal/bootstrap/bootstrap.go
  • /Users/liangxu/Documents/test/geo-rankly/server/internal/shared/auth/session_store.go
  • /Users/liangxu/Documents/test/geo-rankly/server/internal/tenant/repository/sqlc.yaml
  • /Users/liangxu/Documents/test/geo-rankly/server/scripts/check_tenant_scope.sh
  • /Users/liangxu/Documents/test/geo-rankly/.github/workflows/backend-ci.yml
  • /Users/liangxu/Documents/test/geo-rankly/server/docker-compose.yaml
  • /Users/liangxu/Documents/test/geo-rankly/apps/admin-web
  • /Users/liangxu/Documents/test/geo-rankly/packages/shared-types/src/index.ts
  • /Users/liangxu/Documents/test/geo-rankly/packages/http-client/src/index.ts

Dongchedi Desktop Publish Debug - 2026-04-28

  • Local desktop_tasks evidence for article 95 showed the latest dongchedi failures changed from request_failed_403 to platform response server exception after switching the adapter to page-context publish.
  • Raw article_versions.markdown_content stores diff-match-patch patches by design; repository reconstruction for article 95 produced a normal 8219-byte article body, so raw DB diff text is not the publish payload.
  • The reconstructed article contains editor image asset references from /api/public/assets/...; Dongchedi /spice/image?sk=dcd cannot fetch localhost/relative desktop assets. Leaving those image URLs in final HTML is a likely cause of Dongchedi server exception.
  • After image URL replacement was fixed, Dongchedi still returned server exception; local reconstruction of article 95 showed the submitted HTML contained GFM <table> output. The Dongchedi publish API likely rejects or crashes on table markup, so the desktop adapter now downgrades tables to paragraph rows before submit.
  • After table downgrading, Dongchedi still returned server exception; the adapter now further reduces content to conservative tags (p, strong, img), strips editor image wrapper attributes, converts headings/list items to paragraphs, and runs ImageX-uploaded local body images through spice/image before inserting the content URL.
  • Latest retry after conservative HTML still returned server exception with content_table_count = 0, content_has_local_asset_url = false, and content_image_count = 1, so the remaining suspects are the body image URL shape, table-derived HTML entities, or cover metadata rather than raw local asset/table markup.
  • Follow-up hardening removes &nbsp; from Dongchedi body HTML and tries Dongchedi /spice/image?sk=dcd in both session and page context.
  • A retry showed content_image_upload_strategies = ['imagex_direct_fallback'] and still returned server exception, confirming direct ImageX body URLs are not safe for Dongchedi正文. The adapter now removes local body images that cannot be converted to a Dongchedi spice URL and submits the article with the uploaded cover only.
  • A no-body-image retry still returned server exception, leaving extra.content_word_cnt as the most suspicious remaining mismatch because it was still hardcoded to the reference extension's 267 while the final submitted content length is much larger. The adapter now computes content_word_cnt from the final HTML text and logs both submitted and recomputed counts.

Visual/Browser Findings

  • Reviewed the reference screenshots for 工作台.png and 模板创作.png and matched the new UI to the same left-nav plus airy card/table composition.
  • Browser automation verification was attempted but blocked by the local Playwright MCP directory error; preview HTTP verification succeeded instead.
  • Later browser verification succeeded with Playwright CLI, and the captured login/workspace screenshots matched the intended layout and data density.
  • Position-aware OCR of the reference images confirmed the sidebar navigation occupies roughly the left 10-12% of the frame, with content titles aligned at the upper-left of the main canvas and most action buttons aligned to the right edge of page headers.
  • The workspace screenshot places stats on the upper-right and the recent-articles table directly beneath the template card block, so the existing all-grid treatment should be tightened into a more asymmetric two-column composition.

Monitoring Daily Collection Review

  • Data-detail/dashboard reads are working for existing monitoring data: the detail API returned 2026-04-24 question samples, and the dashboard composite showed manual collection progress for brand 3.
  • The daily automatic path was not working before this repair: cmd/scheduler started article scheduled generation plus monitoring recovery/inspection workers, but no worker generated monitoring_collect_tasks from tenant_monitoring_quotas.collect_frequency = daily.
  • Local monitoring DB evidence matched the code gap: 2026-04-24 and 2026-04-23 tasks were manual/plugin-standard; the latest automatic tasks were older 2026-04-17 and 2026-04-10 rows.
  • MonitoringService.CollectNow was the only production path found that inserted monitoring collect tasks before this repair, and it writes trigger_source = manual.
  • The repair adds MonitoringDailyTaskWorker under internal/tenant/app and starts it from cmd/scheduler.
  • Daily task generation now materializes durable monitoring_collect_tasks after midnight using stable jitter, per-run/per-plan/per-brand caps, daily hard-cap enforcement, tenant/brand advisory locks, and the existing idempotent unique key.
  • Desktop offline is now a normal deferred state: automatic daily tasks stay pending with execution_owner = desktop_tasks; missing live desktop account/client no longer falls back to legacy backend execution.
  • Due desktop dispatch is claimed with FOR UPDATE SKIP LOCKED, last_dispatched_at, and dispatch_attempts, so multiple scheduler instances do not repeatedly fan out the same pending tasks.
  • P0 follow-up: when desktop_tasks rollout is disabled or the tenant has no primary_client_id, the daily worker now skips the plan instead of inserting automatic/legacy rows. This prevents orphan daily tasks that no intended worker will dispatch.
  • Follow-up simplification: Go runtime and dev seed code no longer depend on tenant_monitoring_quotas. Daily plans are derived from registered desktop_clients; platform coverage uses the canonical default monitoring platform set; dashboard/projection no longer read quota rows; and heartbeat/collect-now no longer write a primary client pointer into that table.
  • P0 follow-up: daily worker RabbitMQ publish no longer uses context.Background(). It now derives a 5-second child context from the worker tick context, so RabbitMQ publish is bounded by both the per-publish timeout and the worker's 45-second deadline.
  • P0 follow-up: one tenant/brand failure no longer aborts the whole tick. Plan/brand load and generation failures increment summary failure counts, log scoped fields, and continue unless the worker context is canceled or reaches its deadline.
  • Observability follow-up: the daily worker now records real counters/gauges for completed/failed/partial runs, plan/brand failures, skipped plans, planned/due/created/desktop/deferred task counts, last timestamps, duration, scope counts, business date, and last fatal error code.
  • cmd/scheduler now starts a lightweight HTTP server on scheduler.http_port default 8081; /metrics returns Prometheus text for scraping and /api/internal/metrics/monitoring/daily-tasks returns the JSON snapshot from the scheduler process.
  • Fairness follow-up: daily hard-cap accounting now starts from the durable count of already materialized automatic desktop tasks for that tenant/date, keeps the full hard-cap candidate horizon for existing-key filtering, and only decrements in-memory remaining capacity by rows actually inserted in the current tick. Existing candidates no longer consume later brands' new-task capacity.
  • Product-policy follow-up: the daily worker no longer treats any registered desktop client as eligible. Daily plans now require an active tenant subscription, use subscription/config brand limits, use only live desktop platform_accounts as authorized AI platforms, and compute per-brand task capacity from min(active configured questions, brand_library.max_keywords * max_questions_per_keyword) * authorized_platform_count.
  • Dashboard/projection follow-up: platform lists no longer backfill the full six-platform catalog. Tenant UI quota resolution is based on the current user's registered desktop client and bound platform accounts; projection platform rows are derived from actual task platforms for that brand/date.
  • Verification passed with targeted Go tests and a live Postgres EXPLAIN of the new SQL claim statement.
  • Heartbeat primary follow-up: selecting the primary desktop client by ORDER BY last_seen_at DESC is unsafe because the current heartbeat has not touched last_seen_at yet and multiple desktop clients can alternate primary. The fix uses a per-tenant/per-workspace desktop_client_primary_leases row with a TTL: the incumbent renews its lease, non-primary clients cannot steal a live lease, and failover only happens after lease expiry or primary client revocation.
  • Daily-plan follow-up: daily automatic task generation now prefers the sticky primary lease when choosing primary_client_id; if no lease exists or the leased client has no bound platform accounts, it falls back to a deterministic eligible desktop client.
  • UI quota follow-up: current-user registered client lookup no longer sorts by last_seen_at; it now uses registration order so heartbeat cadence cannot make platform authorization display flip between the same user's desktop clients.
  • Index follow-up: desktop_clients did not have a tenant/workspace/last_seen covering index. A migration now adds tenant/workspace and tenant/workspace/user partial indexes for active desktop-client lookup paths.
  • Scheduler metrics hardening: /metrics and /api/internal/metrics/monitoring/daily-tasks now require a configured internal metrics token and accept either Authorization: Bearer <token> or X-Internal-Metrics-Token. If the token is missing, the HTTP metrics server does not start.
  • Scheduler bind hardening: scheduler.http_host defaults to 127.0.0.1 instead of 0.0.0.0; deployments that need pod/network scraping must explicitly set SCHEDULER_HTTP_HOST=0.0.0.0 or config equivalent and provide SCHEDULER_INTERNAL_METRICS_TOKEN.
  • Scheduler shutdown hardening: the metrics server now uses http.Server with Shutdown, and ListenAndServe errors are logged without Fatalf, so a port bind failure does not kill all scheduler workers or bypass deferred cleanup.
  • Scheduler disablement: scheduler.http_port: -1 is preserved as an explicit disable sentinel, while omitted/zero ports still normalize to 8081.
  • Scheduler worker shutdown follow-up: the scheduler process now starts every long-running worker through a sync.WaitGroup and waits on shutdown before returning from main, so deferred DB/MQ cleanup does not race active worker goroutines.
  • Worker lifecycle follow-up: scheduler workers expose blocking Run(ctx) entrypoints. Their loops still stop when the process context is canceled, but active runOnce work uses its own timeout context and is not interrupted immediately by the signal.
  • Authorization-contract follow-up: dashboard runtime and question-detail responses now expose platform_authorization_status with authorized, no_desktop_client, and no_authorized_platforms. Dashboard runtime also exposes authorized_monitoring_platform_count.
  • Frontend no-auth follow-up: TrackingView does not render a dashboard no-authorized/no-desktop alert; it keeps the platform matrix stable and disables collect-now for no authorized platforms and for a selected platform outside the authorized set. TrackingQuestionDetailView still renders an explicit unavailable state when detail data has no executable platforms.
  • Browser regression: admin-web was exercised with a no-authorized-platform response. Dashboard rendered all six platform rows, blocked collection, and did not show the removed no-authorized alert; question detail showed the unavailable state and did not render the answer/detail grid.
  • Product display follow-up: the dashboard/projection display contract is now fixed at the canonical six AI platforms (yuanbao, kimi, wenxin, deepseek, doubao, qwen). This is separate from executable platform authorization, so a revoked/offline/unbound platform remains visible with zero samples/pending status instead of disappearing.
  • Authorization execution follow-up: dashboard runtime now returns authorized_monitoring_platform_ids; frontend collect-now uses this explicit list to decide whether a selected platform is executable, not the six display rows.
  • Workspace-scope follow-up: monitoring quota/runtime/access-state and collect-now client resolution now use auth.CurrentWorkspaceID(ctx) instead of actor.PrimaryWorkspaceID, so a future multi-workspace switch can scope desktop clients and platform accounts correctly.
  • Workspace request scope is set by WorkspaceScope: absent X-Workspace-ID falls back to the actor's primary workspace; a non-primary workspace id must be present in workspace_memberships for the current tenant/user.
  • Daily-plan config follow-up: malformed enabled_platforms JSON now returns a decode error, logs a scoped Warn with tenant/workspace/business date, increments the plan failure counter through FailedPlanCount, and skips only that plan.
  • Heartbeat primary lease follow-up: the primary decision path no longer runs a write transaction or FOR UPDATE on every heartbeat. Normal primary/non-primary heartbeats are read-only; writes happen only for missing lease insert, throttled primary renew, expired/unavailable primary claim, or CAS race handling.
  • Heartbeat lease observability now records resolution outcomes, DB write operations, CAS contention, errors, and duration; tenant-api exposes JSON and Prometheus-format internal endpoints under /api/internal/metrics/monitoring/heartbeat-primary-lease when scheduler.internal_metrics_token is configured.
  • Monitoring heartbeat snapshot follow-up: access snapshots are now handled as semantic state instead of a per-heartbeat liveness write. The heartbeat path batch-loads current platform states, skips fresh unchanged rows, refreshes unchanged rows at most every 10 minutes, and only opens a monitoring DB transaction if a snapshot insert/update/refresh or pending-task reconciliation is actually needed.
  • Inaccessible-platform reconciliation now does a read-only pending-task platform check first. Stable not_logged_in / unavailable reports with no pending tasks do not open a transaction; if pending rows exist, the primary client updates only those rows to skipped.
  • Monitoring heartbeat snapshot observability now tracks reported platforms, opened transactions, snapshot inserts/updates/refreshes/skips, reconciled skipped tasks, projection rebuilds, errors, and duration. Tenant-api exposes token-protected JSON and Prometheus endpoints under /api/internal/metrics/monitoring/heartbeat-snapshots.
  • Added a monitoring DB partial index for the reconciliation check: idx_collect_tasks_pending_platform_reconcile on (tenant_id, business_date, ai_platform_id) where collector_type = 'desktop' and status = 'pending'.
  • Prometheus exposition follow-up: daily-task, heartbeat-primary-lease, and heartbeat-snapshot Prometheus text output now uses prometheus/client_golang collectors plus promhttp instead of hand-written Fprintf formatting.
  • Scheduler /metrics now uses a dedicated Prometheus registry with the custom daily-task collector plus collectors.NewGoCollector() and collectors.NewProcessCollector(). Process metrics are emitted on supported production OSes such as Linux; local Darwin tests assert runtime metrics and keep process assertion OS-gated.
  • client_golang v1.20.5 still does not escape raw carriage returns in label values before exposition on this local toolchain, so the custom collector path normalizes \r to a literal \\r sequence before passing label values into ConstMetric.
  • Daily task materialization follow-up: ensureDailyMonitoringCollectTasks no longer loops with one tx.Exec per candidate. It now builds arrays of question IDs, question hashes, platform IDs, and dispatch times, then performs one INSERT ... SELECT FROM unnest(...) ON CONFLICT DO NOTHING per batch. This preserves idempotency and created-row counting while keeping transaction round trips fixed.
  • Daily plan loading follow-up: loadDailyMonitoringPlans no longer uses one SQL statement for lease-aware client election and platform JSON aggregation. It first selects the primary client candidate per (tenant_id, workspace_id) with subscription policy and lease preference, then batch-loads authorized platform JSON for the selected (tenant_id, workspace_id, client_id) tuples. If authorization disappears between reads, the candidate is skipped rather than producing an empty executable plan.
  • Authorization bug fix: monitoring authorization now means a bound, non-deleted platform account for a supported AI platform. platform_accounts.health is treated as execution state, not authorization state, so non-live accounts no longer suppress daily scheduled task generation, dashboard authorization, desktop task targeting, or question-detail history display.
  • Immediate collection remains stricter than daily scheduled collection: it still requires the target desktop client to be online, and the target client is now the same client that owns the authorized platform accounts. Daily scheduled collection still materializes backend tasks while the desktop client is offline and dispatches when the client becomes available.

k3s Deployment Foundation - 2026-04-30

  • The existing deploy topology is a good base for k3s: app services are tenant-api, worker-generate, scheduler, ops-api, and frontend; repo runtime also contains kol-assist-worker and ops-web, so the k3s stack includes them too.
  • The stateful dependencies needed for a self-contained k3s install are postgres, monitoring-postgres, rabbitmq, redis, qdrant, and minio.
  • deploy/k3s now uses kustomize and renders a namespace, config map, placeholder secret, infra StatefulSets, one-shot migrate and minio-init Jobs, app Deployments/Services, and Traefik Ingress rules for geo-rankly.local and ops.geo-rankly.local.
  • Kustomize cannot reference files above deploy/k3s under its default load restrictions, so the k3s overlay now carries its own config/config.yaml, config/prompts.yml, and config/ops-config.yaml.
  • Tenant services mount /app/configs/config.yaml, /app/configs/prompts.yml, and secret-backed /app/configs/config.local.yaml, allowing sensitive overrides without rebuilding images.
  • ops-api currently reads a single ops config file and does not merge an ops-config.local.yaml, so k3s documentation calls out that database password changes must also be reflected in deploy/k3s/config/ops-config.yaml.
  • The existing offline package script did not build ops-api, kol-assist-worker, or ops-web; it now builds and saves those images so k3s can run the full service set.
  • Dockerfile.ops-web now builds the operations frontend and serves it through its existing nginx proxy to ops-api.
  • Verification succeeded with kubectl kustomize deploy/k3s, local YAML parsing of 30 rendered Kubernetes objects, bash -n for deploy scripts, git diff --check, pnpm --filter ops-web build, and pnpm --filter admin-web typecheck.
  • Live kubectl apply --dry-run=client validation could not complete locally because the configured kube-apiserver at 127.0.0.1:26443 is not running; run that validation on the target k3s node.