geo/findings.rag-numeric-fidelity.md

Findings: RAG Numeric Fact Fidelity

Reported Symptom

• Generated article says Liaoning Muge Creative was founded in 2014 and has 12 years of local service.
• The user reports the knowledge base contains a different founding time.
• Required invariant: when selected knowledge contains a basic numeric fact, generated copy must not contradict it.

Local Trace

• Generation code is under server/internal/tenant/app; retrieval is under server/internal/shared/retrieval; dispatch is under server/internal/worker/generate.
• Prompt config describes knowledge as optional reference material and constrains dates through natural-language instructions only.
• KnowledgeContext.PreciseFacts exists and is rendered into a high-precision prompt section.
• ResolveContext loads precise facts from up to 24 completed items in selected groups, prioritizing retrieved items.
• Founding/establishment/registration labels and phrases are absent from the precise-fact extractor, so company founded in 20xx is not elevated.
• Custom tasks snapshot only the rendered knowledge prompt string, not typed fact records.
• Template, custom, and imitation paths all accept model output through sanitizeGeneratedArticleMarkdown and then persist it without factual validation.

Production Trace

• Three worker-generate pods and all core dependencies are healthy in namespace geo-rankly.
• Worker selector is app.kubernetes.io/name=worker-generate; deployed image is geo-rankly/worker-generate:uncommitted-server-0acd3791-20260709232655.
• A production imitation task logged candidate_count=0, selected_count=0, and precise_fact_count=22, confirming group-wide precise facts are actively rendered even when vector retrieval returns no search points.
• The reported Huludao article was not identifiable in the last 24 hours of filtered worker logs; locate it through PostgreSQL next.
• Read-only psql access to the production geo database is available through the PostgreSQL pod; queries will minimize returned article/knowledge content.
• Exact reproduction record: article 554, current version 742, tenant 2, brand 4, title 2026葫芦岛全屋定制本地5家实力公司汇总，口碑+落地+性价比深度评测, completed at 2026-07-10 09:40:22 UTC.
• The target article source/task type is kol, corresponding to worker log task 528; the fix must cover the KOL generation path in addition to template/custom/imitation paths.
• Task 528 selected knowledge groups [17,27,28,30,32,47] and subscription prompt 1140; its payload contains rendered prompt/schema/card snapshots but no typed knowledge facts.
• Version 742 contains the reported claim (percent-encoded in stored markdown): 品牌成立于2014年，拥有12年本地服务经验.
• Production knowledge is internally inconsistent: newest completed item 31 says 成立于2020年1月, while older completed items 13 and 14 say 成立于2019年. Both are in the selected recursive group scope.
• KOL worker resolves knowledge with brand_id read from task input. Task 528 has no brand_id key even though article 554 belongs to brand 4, so it resolves with brand 0; this likely excluded the entire knowledge scope for the reported generation.
• KOL output follows the same unvalidated pattern as other paths: model result is trimmed, titled, and persisted directly.

Ranked Hypotheses

1. Missing KOL brand scope: if this is the primary cause, a task payload without brand_id will resolve no brand-4 knowledge, and task 528 will have no corresponding knowledge-resolution log.
2. Founding facts are not extracted: if true, extractKnowledgePreciseFacts("公司成立于2020年1月") will omit the line.
3. No deterministic output gate: if true, a body contradicting a supplied founding fact will pass every current post-generation function and be persisted.
4. Conflicting source values are not canonicalized: if true, once founding extraction is enabled both 2020 and 2019 enter prompt context without a unique authoritative value.

Hypothesis Results

• Hypothesis 1 confirmed at the creator: normal KOL submission builds taskInput with knowledge_group_ids but omits brand_id; only scheduled KOL input conditionally attaches it. The worker always reads brand_id from the payload.
• Existing tests cover precise prompt extraction/rendering but do not cover KOL task scope propagation or output fact validation.
• Normal KOL submission already resolves the current brand and passes it into enqueue; the bug is localized to serializing the task snapshot, so no API contract change is needed.
• The shared llm.Client.Generate signature is compatible with a reusable generate/validate/repair helper used by tenant services and the KOL worker.
• Template, imitation, and KOL paths retain the resolved KnowledgeContext in the same execution frame and can pass its constraints directly to the guard.
• Custom prompt-rule generation snapshots the rendered knowledge prompt before queueing and reconstructs only a prompt string in the worker, so it also needs a JSON-safe constraint snapshot for deterministic validation.

Proposed Conflict Policy

• Prefer the newest completed active knowledge item for the same typed fact and subject as the canonical value.
• Preserve source metadata and emit an observable conflict when older active items disagree.
• Validate generated output against the canonical value before persistence; retry once with correction instructions, then fail rather than publish a contradiction.
• Review note: the current group-item SQL prioritizes retrieved item IDs before updated_at. Canonical fact selection must not accidentally treat an older retrieved item as newer; source timestamp/order semantics need explicit coverage.

Design Constraint

• A global numeric-token whitelist is unsafe: article years, rankings, section numbers, derived values, and user-provided facts are legitimate.
• Enforcement should use typed fact identity, normalized values, subject/field-aware contradiction detection, and an explicit retry/fail policy.
• Initial protected types will cover company founding date and explicitly sourced experience/service duration; the structure must be extensible to other business numeric types.
• With active fact constraints, generation should withhold draft deltas, validate the full result, repair once from canonical facts, and reject the task if the repaired result still conflicts.

Final Outcome

• Immediate and scheduled KOL tasks now serialize the same brand/knowledge scope consumed by the worker.
• Founding dates and explicit service/experience durations become typed, subject-aware constraints.
• Source conflicts choose the newest completed item; older conflicting values are removed from the prompt and logged without exposing source content.
• Template, custom prompt-rule, imitation, and KOL generation validate before persistence, repair once, and fail instead of saving a second conflicting draft.
• Custom queued tasks snapshot constraints; pre-upgrade queued tasks resolve missing constraints at execution time.
• Production was inspected read-only. No deployment was performed from the dirty worktree.