Adds a new browser-fetch microservice that wraps Lightpanda for rendering
JS-heavy pages, and wires it into the knowledge URL parser as a fallback
when the upstream link reader returns 403/429/5xx for an allow-listed
domain (e.g. zhuanlan.zhihu.com). Includes config/env plumbing, hot-reload
diff support, and full deploy assets (Dockerfile target, docker-compose,
k3s manifests, image-build/package scripts).
Poll the knowledge item list every 5s while any item is still pending
or processing so the UI reflects ingestion progress without manual
refresh, and exclude items that have no active chunks from group
counters so groups with no usable content do not appear populated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- shared/cache: add Options/L1/async/metrics/prefix decorators, multi-key ops, Redis pool tuning, and JSON readthrough metrics
- worker-generate: claim tasks via DB lease + heartbeat, requeue stale queued tasks, expire dead leases with refund/cache invalidation
- tenant: version article cache keys so worker recovery invalidations propagate cleanly
- shared/config: expand Redis (pool/timeouts/TLS) and Generation (lease/recovery) configs with defaults
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add POST /api/tenant/knowledge/items/retry/:id to re-queue failed parse
tasks, plus a 20-rune cap on text item names with matching frontend
validation. Wires the retry button into the knowledge table with a
colored status tag for clearer state feedback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace single Load() with a watching Store that re-reads config files
(and config.local.yaml) and fans out a ReloadEvent with a per-field diff
so consumers can decide whether the change is hot-applicable or requires
a process restart.
- Wrap llm, retrieval, vector store, and object storage clients in
Reloadable* shells so the bootstrap can swap their underlying impls when
config changes without re-instantiating handlers.
- Make jwt.Manager and ops TokenIssuer mutable under a lock so secrets and
TTLs can be rotated live; thread default plan code through a setter on
the ops AdminUserService.
- Wire ConfigStore through bootstrap and every cmd/main.go, scheduler /
worker / tenant-api / ops-api start the watcher; services and handlers
take a config.Provider so they always read current values for things
like generation.stream_enabled, scheduler dispatch, retrieval, etc.
- Switch shared/config decoding off viper to a Kratos-derived runtime
package so env placeholders (\${VAR:default}) resolve consistently and
the same source machinery powers both the loader and the watcher.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Logging the full candidate and selected snippet bodies per resolve was
inflating log volume without adding signal we use.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Walk knowledge group children recursively so selecting a parent group
pulls snippets from its descendants too.
- Filter knowledge items to status='completed' across the prompt and
active-snippet queries to avoid leaking in-progress documents.
- Dedupe vector candidates by point id and normalized text, then pick
rerank+vector results with a per-item cap so prompts stay diverse.
- Tag each rendered snippet with a [Kn] reference id for prompt citing.
- Build the rule knowledge query from task params (title/keywords/etc.)
before falling back to rule prompt content so retrieval matches the
user's actual ask.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Collect per-request auth diagnostics (header presence, scheme, token
fingerprint, failure stage/reason, subject, expiry) in the auth
middleware and expose them through RequestLogContextFromGin so the
shared logger middleware can emit them.
- Add route, query, user-agent, and referer fields to the request log.
- Log knowledge resolve outcome per stage (no_search_points,
no_candidate_text, no_active_candidates, rerank_failed_fallback,
resolved) with candidate/selected snippet previews and precise facts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Added `EnableWebSearch` and `KnowledgeGroupIDs` fields to `KolGenerationSubmitRequest`.
- Updated `Submit` method in `KolGenerationService` to handle new request fields.
- Integrated web search and knowledge group validation based on card configuration.
- Introduced caching mechanisms in `KolPackageService`, `KolPromptService`, and `KolSubscriptionAdminService` to improve performance.
- Implemented knowledge context resolution in `KolGenerationWorker` to enrich prompts with relevant knowledge snippets.
- Added utility functions for handling card configuration in both backend and frontend.
- Created tests for knowledge extraction and rendering to ensure accuracy and reliability.
Replace in-process generation with queue-based async execution via
RabbitMQ. Add generation task runtime for lifecycle management, article
generation payload/queue abstractions, and template-assist queue publisher.
Refactor prompt generation service to support batch generation with
per-item error tracking. Update stream hub to bridge queue events to SSE.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Implemented KnowledgeWebsiteMarkdownPrompt for formatting webpage content into Markdown.
- Added GetItemDetail method in KnowledgeHandler to retrieve knowledge item details.
- Introduced KnowledgeDetailDrawer component for displaying knowledge item details in the admin web interface.
- Created MarkdownPreview component for rendering Markdown content.
- Added knowledge chunking and URL parsing logic to handle webpage content extraction and formatting.
- Updated database schema to include markdown_content in knowledge_items table.
- Implemented KnowledgeHandler for managing knowledge groups and items, including listing, creating, updating, and deleting operations.
- Added database migration scripts to create necessary tables for knowledge management, including knowledge_groups, knowledge_items, knowledge_parse_tasks, and knowledge_chunks_meta.
- Introduced prompt_rule_knowledge_groups table to associate prompt rules with knowledge groups.
root