geo/findings.rag-numeric-fidelity.md

# Findings: RAG Numeric Fact Fidelity

## Reported Symptom
- Generated article says Liaoning Muge Creative was founded in 2014 and has 12 years of local service.
- The user reports the knowledge base contains a different founding time.
- Required invariant: when selected knowledge contains a basic numeric fact, generated copy must not contradict it.

## Local Trace
- Generation code is under `server/internal/tenant/app`; retrieval is under `server/internal/shared/retrieval`; dispatch is under `server/internal/worker/generate`.
- Prompt config describes knowledge as optional reference material and constrains dates through natural-language instructions only.
- `KnowledgeContext.PreciseFacts` exists and is rendered into a high-precision prompt section.
- `ResolveContext` loads precise facts from up to 24 completed items in selected groups, prioritizing retrieved items.
- Founding/establishment/registration labels and phrases are absent from the precise-fact extractor, so `company founded in 20xx` is not elevated.
- Custom tasks snapshot only the rendered knowledge prompt string, not typed fact records.
- Template, custom, and imitation paths all accept model output through `sanitizeGeneratedArticleMarkdown` and then persist it without factual validation.

## Production Trace
- Three worker-generate pods and all core dependencies are healthy in namespace `geo-rankly`.
- Worker selector is `app.kubernetes.io/name=worker-generate`; deployed image is `geo-rankly/worker-generate:uncommitted-server-0acd3791-20260709232655`.
- A production imitation task logged `candidate_count=0`, `selected_count=0`, and `precise_fact_count=22`, confirming group-wide precise facts are actively rendered even when vector retrieval returns no search points.
- The reported Huludao article was not identifiable in the last 24 hours of filtered worker logs; locate it through PostgreSQL next.
- Read-only `psql` access to the production `geo` database is available through the PostgreSQL pod; queries will minimize returned article/knowledge content.
- Exact reproduction record: article `554`, current version `742`, tenant `2`, brand `4`, title `2026葫芦岛全屋定制本地5家实力公司汇总，口碑+落地+性价比深度评测`, completed at `2026-07-10 09:40:22 UTC`.
- The target article source/task type is `kol`, corresponding to worker log task `528`; the fix must cover the KOL generation path in addition to template/custom/imitation paths.
- Task `528` selected knowledge groups `[17,27,28,30,32,47]` and subscription prompt `1140`; its payload contains rendered prompt/schema/card snapshots but no typed knowledge facts.
- Version `742` contains the reported claim (percent-encoded in stored markdown): `品牌成立于2014年，拥有12年本地服务经验`.
- Production knowledge is internally inconsistent: newest completed item `31` says `成立于2020年1月`, while older completed items `13` and `14` say `成立于2019年`. Both are in the selected recursive group scope.
- KOL worker resolves knowledge with `brand_id` read from task input. Task `528` has no `brand_id` key even though article `554` belongs to brand `4`, so it resolves with brand `0`; this likely excluded the entire knowledge scope for the reported generation.
- KOL output follows the same unvalidated pattern as other paths: model result is trimmed, titled, and persisted directly.

## Ranked Hypotheses
1. **Missing KOL brand scope:** if this is the primary cause, a task payload without `brand_id` will resolve no brand-4 knowledge, and task `528` will have no corresponding knowledge-resolution log.
2. **Founding facts are not extracted:** if true, `extractKnowledgePreciseFacts("公司成立于2020年1月")` will omit the line.
3. **No deterministic output gate:** if true, a body contradicting a supplied founding fact will pass every current post-generation function and be persisted.
4. **Conflicting source values are not canonicalized:** if true, once founding extraction is enabled both 2020 and 2019 enter prompt context without a unique authoritative value.

## Hypothesis Results
- Hypothesis 1 confirmed at the creator: normal KOL submission builds `taskInput` with `knowledge_group_ids` but omits `brand_id`; only scheduled KOL input conditionally attaches it. The worker always reads `brand_id` from the payload.
- Existing tests cover precise prompt extraction/rendering but do not cover KOL task scope propagation or output fact validation.
- Normal KOL submission already resolves the current brand and passes it into `enqueue`; the bug is localized to serializing the task snapshot, so no API contract change is needed.
- The shared `llm.Client.Generate` signature is compatible with a reusable generate/validate/repair helper used by tenant services and the KOL worker.
- Template, imitation, and KOL paths retain the resolved `KnowledgeContext` in the same execution frame and can pass its constraints directly to the guard.
- Custom prompt-rule generation snapshots the rendered knowledge prompt before queueing and reconstructs only a prompt string in the worker, so it also needs a JSON-safe constraint snapshot for deterministic validation.

## Proposed Conflict Policy
- Prefer the newest completed active knowledge item for the same typed fact and subject as the canonical value.
- Preserve source metadata and emit an observable conflict when older active items disagree.
- Validate generated output against the canonical value before persistence; retry once with correction instructions, then fail rather than publish a contradiction.
- Review note: the current group-item SQL prioritizes retrieved item IDs before `updated_at`. Canonical fact selection must not accidentally treat an older retrieved item as newer; source timestamp/order semantics need explicit coverage.

## Design Constraint
- A global numeric-token whitelist is unsafe: article years, rankings, section numbers, derived values, and user-provided facts are legitimate.
- Enforcement should use typed fact identity, normalized values, subject/field-aware contradiction detection, and an explicit retry/fail policy.
- Initial protected types will cover company founding date and explicitly sourced experience/service duration; the structure must be extensible to other business numeric types.
- With active fact constraints, generation should withhold draft deltas, validate the full result, repair once from canonical facts, and reject the task if the repaired result still conflicts.

## Final Outcome
- Immediate and scheduled KOL tasks now serialize the same brand/knowledge scope consumed by the worker.
- Founding dates and explicit service/experience durations become typed, subject-aware constraints.
- Source conflicts choose the newest completed item; older conflicting values are removed from the prompt and logged without exposing source content.
- Template, custom prompt-rule, imitation, and KOL generation validate before persistence, repair once, and fail instead of saving a second conflicting draft.
- Custom queued tasks snapshot constraints; pre-upgrade queued tasks resolve missing constraints at execution time.
- Production was inspected read-only. No deployment was performed from the dirty worktree.