root
b6858d2605
Add task_plan/findings/progress notes for knowledge fact fidelity, brand description context, media supply favorites, and the workspace sync-to-39 deployment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
64 lines
7.1 KiB
Markdown
64 lines
7.1 KiB
Markdown
# Findings: RAG Numeric Fact Fidelity
|
|
|
|
## Reported Symptom
|
|
- Generated article says Liaoning Muge Creative was founded in 2014 and has 12 years of local service.
|
|
- The user reports the knowledge base contains a different founding time.
|
|
- Required invariant: when selected knowledge contains a basic numeric fact, generated copy must not contradict it.
|
|
|
|
## Local Trace
|
|
- Generation code is under `server/internal/tenant/app`; retrieval is under `server/internal/shared/retrieval`; dispatch is under `server/internal/worker/generate`.
|
|
- Prompt config describes knowledge as optional reference material and constrains dates through natural-language instructions only.
|
|
- `KnowledgeContext.PreciseFacts` exists and is rendered into a high-precision prompt section.
|
|
- `ResolveContext` loads precise facts from up to 24 completed items in selected groups, prioritizing retrieved items.
|
|
- Founding/establishment/registration labels and phrases are absent from the precise-fact extractor, so `company founded in 20xx` is not elevated.
|
|
- Custom tasks snapshot only the rendered knowledge prompt string, not typed fact records.
|
|
- Template, custom, and imitation paths all accept model output through `sanitizeGeneratedArticleMarkdown` and then persist it without factual validation.
|
|
|
|
## Production Trace
|
|
- Three worker-generate pods and all core dependencies are healthy in namespace `geo-rankly`.
|
|
- Worker selector is `app.kubernetes.io/name=worker-generate`; deployed image is `geo-rankly/worker-generate:uncommitted-server-0acd3791-20260709232655`.
|
|
- A production imitation task logged `candidate_count=0`, `selected_count=0`, and `precise_fact_count=22`, confirming group-wide precise facts are actively rendered even when vector retrieval returns no search points.
|
|
- The reported Huludao article was not identifiable in the last 24 hours of filtered worker logs; locate it through PostgreSQL next.
|
|
- Read-only `psql` access to the production `geo` database is available through the PostgreSQL pod; queries will minimize returned article/knowledge content.
|
|
- Exact reproduction record: article `554`, current version `742`, tenant `2`, brand `4`, title `2026葫芦岛全屋定制本地5家实力公司汇总，口碑+落地+性价比深度评测`, completed at `2026-07-10 09:40:22 UTC`.
|
|
- The target article source/task type is `kol`, corresponding to worker log task `528`; the fix must cover the KOL generation path in addition to template/custom/imitation paths.
|
|
- Task `528` selected knowledge groups `[17,27,28,30,32,47]` and subscription prompt `1140`; its payload contains rendered prompt/schema/card snapshots but no typed knowledge facts.
|
|
- Version `742` contains the reported claim (percent-encoded in stored markdown): `品牌成立于2014年，拥有12年本地服务经验`.
|
|
- Production knowledge is internally inconsistent: newest completed item `31` says `成立于2020年1月`, while older completed items `13` and `14` say `成立于2019年`. Both are in the selected recursive group scope.
|
|
- KOL worker resolves knowledge with `brand_id` read from task input. Task `528` has no `brand_id` key even though article `554` belongs to brand `4`, so it resolves with brand `0`; this likely excluded the entire knowledge scope for the reported generation.
|
|
- KOL output follows the same unvalidated pattern as other paths: model result is trimmed, titled, and persisted directly.
|
|
|
|
## Ranked Hypotheses
|
|
1. **Missing KOL brand scope:** if this is the primary cause, a task payload without `brand_id` will resolve no brand-4 knowledge, and task `528` will have no corresponding knowledge-resolution log.
|
|
2. **Founding facts are not extracted:** if true, `extractKnowledgePreciseFacts("公司成立于2020年1月")` will omit the line.
|
|
3. **No deterministic output gate:** if true, a body contradicting a supplied founding fact will pass every current post-generation function and be persisted.
|
|
4. **Conflicting source values are not canonicalized:** if true, once founding extraction is enabled both 2020 and 2019 enter prompt context without a unique authoritative value.
|
|
|
|
## Hypothesis Results
|
|
- Hypothesis 1 confirmed at the creator: normal KOL submission builds `taskInput` with `knowledge_group_ids` but omits `brand_id`; only scheduled KOL input conditionally attaches it. The worker always reads `brand_id` from the payload.
|
|
- Existing tests cover precise prompt extraction/rendering but do not cover KOL task scope propagation or output fact validation.
|
|
- Normal KOL submission already resolves the current brand and passes it into `enqueue`; the bug is localized to serializing the task snapshot, so no API contract change is needed.
|
|
- The shared `llm.Client.Generate` signature is compatible with a reusable generate/validate/repair helper used by tenant services and the KOL worker.
|
|
- Template, imitation, and KOL paths retain the resolved `KnowledgeContext` in the same execution frame and can pass its constraints directly to the guard.
|
|
- Custom prompt-rule generation snapshots the rendered knowledge prompt before queueing and reconstructs only a prompt string in the worker, so it also needs a JSON-safe constraint snapshot for deterministic validation.
|
|
|
|
## Proposed Conflict Policy
|
|
- Prefer the newest completed active knowledge item for the same typed fact and subject as the canonical value.
|
|
- Preserve source metadata and emit an observable conflict when older active items disagree.
|
|
- Validate generated output against the canonical value before persistence; retry once with correction instructions, then fail rather than publish a contradiction.
|
|
- Review note: the current group-item SQL prioritizes retrieved item IDs before `updated_at`. Canonical fact selection must not accidentally treat an older retrieved item as newer; source timestamp/order semantics need explicit coverage.
|
|
|
|
## Design Constraint
|
|
- A global numeric-token whitelist is unsafe: article years, rankings, section numbers, derived values, and user-provided facts are legitimate.
|
|
- Enforcement should use typed fact identity, normalized values, subject/field-aware contradiction detection, and an explicit retry/fail policy.
|
|
- Initial protected types will cover company founding date and explicitly sourced experience/service duration; the structure must be extensible to other business numeric types.
|
|
- With active fact constraints, generation should withhold draft deltas, validate the full result, repair once from canonical facts, and reject the task if the repaired result still conflicts.
|
|
|
|
## Final Outcome
|
|
- Immediate and scheduled KOL tasks now serialize the same brand/knowledge scope consumed by the worker.
|
|
- Founding dates and explicit service/experience durations become typed, subject-aware constraints.
|
|
- Source conflicts choose the newest completed item; older conflicting values are removed from the prompt and logged without exposing source content.
|
|
- Template, custom prompt-rule, imitation, and KOL generation validate before persistence, repair once, and fail instead of saving a second conflicting draft.
|
|
- Custom queued tasks snapshot constraints; pre-upgrade queued tasks resolve missing constraints at execution time.
|
|
- Production was inspected read-only. No deployment was performed from the dirty worktree.
|