feat(browser-fetch): add lightpanda-backed fetch service and knowledge URL fallback
Adds a new browser-fetch microservice that wraps Lightpanda for rendering JS-heavy pages, and wires it into the knowledge URL parser as a fallback when the upstream link reader returns 403/429/5xx for an allow-listed domain (e.g. zhuanlan.zhihu.com). Includes config/env plumbing, hot-reload diff support, and full deploy assets (Dockerfile target, docker-compose, k3s manifests, image-build/package scripts).
This commit is contained in:
@@ -129,8 +129,8 @@ llm:
|
||||
provider: ark
|
||||
base_url: https://ark.cn-beijing.volces.com/api/v3
|
||||
api_key: "7fb6c66b-129c-4935-9617-709236c4205a"
|
||||
model: doubao-seed-2-0-lite-260215
|
||||
knowledge_url_model: doubao-seed-2-0-mini-260215
|
||||
model: doubao-seed-2-0-pro-260215 # doubao-seed-2-0-lite-260215 #
|
||||
knowledge_url_model: doubao-seed-2-0-mini-260215 #doubao-seed-2-0-mini-260215
|
||||
timeout: 2m
|
||||
max_output_tokens: 16000
|
||||
temperature: 0.7
|
||||
@@ -172,3 +172,22 @@ generation:
|
||||
task_recovery_batch_size: 100
|
||||
task_queued_stale_after: 2m
|
||||
task_max_attempts: 3
|
||||
|
||||
browser_fetch:
|
||||
enabled: true
|
||||
provider: lightpanda
|
||||
service_url: http://browser-fetch:8082
|
||||
queue_size: 64
|
||||
worker_concurrency: 2
|
||||
request_timeout: 45s
|
||||
cache_ttl: 6h
|
||||
allowed_domains:
|
||||
- zhuanlan.zhihu.com
|
||||
- zhihu.com
|
||||
trigger_status:
|
||||
- 403
|
||||
- 429
|
||||
- 500
|
||||
- 502
|
||||
- 503
|
||||
- 504
|
||||
|
||||
Reference in New Issue
Block a user