51 lines
3.2 KiB
Markdown
51 lines
3.2 KiB
Markdown
|
# Task Plan: RAG Numeric Fact Fidelity
|
||
|
|
|
||
|
|
## Goal
|
||
|
|
Ensure numeric facts explicitly present in retrieved knowledge, especially company founding dates, cannot be silently contradicted in generated articles.
|
||
|
|
|
||
|
|
## Current Phase
|
||
|
|
Complete
|
||
|
|
|
||
|
|
## Phases
|
||
|
|
|
||
|
|
### Phase 1: Reproduce and Trace
|
||
|
|
- [x] Trace retrieval, prompt rendering, model output, and persistence locally
|
||
|
|
- [x] Identify the exact generated article and KOL task in production
|
||
|
|
- [x] Confirm the reported source/output contradiction against production knowledge data
|
||
|
|
- [x] Build deterministic failing tests at the payload, fact, and generation seams
|
||
|
|
|
||
|
|
### Phase 2: Test Hypotheses
|
||
|
|
- [x] Rank 4 falsifiable causes
|
||
|
|
- [x] Confirm production task type and payload behavior
|
||
|
|
- [x] Confirm the immediate KOL creator omits the brand scope consumed by its worker
|
||
|
|
- [x] Isolate the minimum enforceable fact-validation contract
|
||
|
|
|
||
|
|
### Phase 3: Regression Test and Fix
|
||
|
|
- [x] Add failing coverage for source-backed founding-date contradictions
|
||
|
|
- [x] Implement typed fact extraction and deterministic validation
|
||
|
|
- [x] Confirm focused tests pass
|
||
|
|
|
||
|
|
### Phase 4: Verification and Cleanup
|
||
|
|
- [x] Run focused, package-level, and full Go tests against the production-shaped regression
|
||
|
|
- [x] Check production data/logs read-only and review the scoped diff
|
||
|
|
- [x] Confirm no temporary instrumentation remains
|
||
|
|
|
||
|
|
## Decisions
|
||
|
|
| Decision | Rationale |
|
||
|
|
|----------|-----------|
|
||
|
|
| Use task-specific planning files | Shared planning files were concurrently rewritten by another task; isolation preserves user work. |
|
||
|
|
| Do not globally whitelist every numeric token | Rankings, article year, derived durations, and section numbers can be valid without appearing in RAG. |
|
||
|
|
| Treat prompt-only controls as insufficient | Model instructions are probabilistic and current output is persisted without validation. |
|
||
|
|
| Guard protected facts before persistence | Retry once with the canonical facts, then fail instead of saving a contradictory article. |
|
||
|
|
| Suppress draft streaming when fact constraints exist | Prevent a known-invalid first draft from being exposed before the repair/validation result. |
|
||
|
|
| Canonicalize source conflicts by newest completed item | Gives corrected knowledge precedence while retaining an observable conflict for cleanup. |
|
||
|
|
|
||
|
|
## Errors
|
||
|
|
| Error | Attempt | Resolution |
|
||
|
|
|-------|---------|------------|
|
||
|
|
| Worker selector `app=worker-generate` matched no resources | 1 | Inspect the real deployment selector/pod names before retrying. |
|
||
|
|
| Shared planning append disappeared during concurrent edits | 1 | Switched to task-specific planning files and will not modify shared plan files again. |
|
||
|
|
| First green run had a helper-name collision and a cross-package private test reference | 1 | Renamed text parsing separately from payload decoding and used producer-package task parsers in the payload test. |
|
||
|
|
| Used the outer exec-cell wait helper for an inner terminal session | 1 | Polled the returned terminal session with `write_stdin`; the full test run completed successfully. |
|
||
|
|
| Full `go vet ./...` later hit a syntax error in concurrently edited `media_supply_favorites_test.go:160` | 1 | Preserve the unrelated user change; vet the affected production packages with `-tests=false` and retain the earlier successful full test result. |
|