geo/task_plan.rag-numeric-fidelity.md

Task Plan: RAG Numeric Fact Fidelity

Goal

Ensure numeric facts explicitly present in retrieved knowledge, especially company founding dates, cannot be silently contradicted in generated articles.

Current Phase

Complete

Phases

Phase 1: Reproduce and Trace

• Trace retrieval, prompt rendering, model output, and persistence locally
• Identify the exact generated article and KOL task in production
• Confirm the reported source/output contradiction against production knowledge data
• Build deterministic failing tests at the payload, fact, and generation seams

Phase 2: Test Hypotheses

• Rank 4 falsifiable causes
• Confirm production task type and payload behavior
• Confirm the immediate KOL creator omits the brand scope consumed by its worker
• Isolate the minimum enforceable fact-validation contract

Phase 3: Regression Test and Fix

• Add failing coverage for source-backed founding-date contradictions
• Implement typed fact extraction and deterministic validation
• Confirm focused tests pass

Phase 4: Verification and Cleanup

• Run focused, package-level, and full Go tests against the production-shaped regression
• Check production data/logs read-only and review the scoped diff
• Confirm no temporary instrumentation remains

Decisions

Decision	Rationale
Use task-specific planning files	Shared planning files were concurrently rewritten by another task; isolation preserves user work.
Do not globally whitelist every numeric token	Rankings, article year, derived durations, and section numbers can be valid without appearing in RAG.
Treat prompt-only controls as insufficient	Model instructions are probabilistic and current output is persisted without validation.
Guard protected facts before persistence	Retry once with the canonical facts, then fail instead of saving a contradictory article.
Suppress draft streaming when fact constraints exist	Prevent a known-invalid first draft from being exposed before the repair/validation result.
Canonicalize source conflicts by newest completed item	Gives corrected knowledge precedence while retaining an observable conflict for cleanup.

Errors

Error	Attempt	Resolution
Worker selector app=worker-generate matched no resources	1	Inspect the real deployment selector/pod names before retrying.
Shared planning append disappeared during concurrent edits	1	Switched to task-specific planning files and will not modify shared plan files again.
First green run had a helper-name collision and a cross-package private test reference	1	Renamed text parsing separately from payload decoding and used producer-package task parsers in the payload test.
Used the outer exec-cell wait helper for an inner terminal session	1	Polled the returned terminal session with write_stdin; the full test run completed successfully.
Full go vet ./... later hit a syntax error in concurrently edited media_supply_favorites_test.go:160	1	Preserve the unrelated user change; vet the affected production packages with -tests=false and retain the earlier successful full test result.