feat(llm): add configurable request/token rate limiting for LLM client

Introduce process-local and Redis-backed global rate limiting for the
shared LLM client, plus a dedicated generation consumer prefetch knob.

- Add request (RPM) and token (TPM) rate limits with burst controls,
  process or global scope, and fail-closed/process-degraded failover
- Wire the LLM client through NewWithRedis so limits can be shared
  across replicas via a Redis token bucket
- Add generation_consumer_prefetch to tune RabbitMQ generation
  consumers independently of the default prefetch
- Bump generation worker_concurrency to 24 and promote golang.org/x/time
  to a direct dependency
- Drop t.Parallel() from a few middleware/bootstrap tests

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-10 00:14:40 +08:00
parent 0acd37918e
commit 347cea1296
16 changed files with 1016 additions and 30 deletions
+1 -1
View File
@@ -206,7 +206,7 @@ metadata:
labels:
app.kubernetes.io/name: worker-generate
spec:
replicas: 1
replicas: 3
selector:
matchLabels:
app.kubernetes.io/name: worker-generate
+12 -1
View File
@@ -83,6 +83,7 @@ rabbitmq:
compliance_review_dlq: compliance.review.run.dlq
compliance_review_dlq_routing_key: compliance.review.run.dlq
consumer_prefetch: 10
generation_consumer_prefetch: 1
publish_channel_pool_size: 32
scheduler:
@@ -200,6 +201,16 @@ llm:
temperature: 0.7
reasoning_effort: minimal
web_search_limit: 3
# Global Redis-backed budget shared by all replicas.
rate_limit_scope: global
rate_limit_key: ark:doubao-seed-2-0
rate_limit_failover_mode: process_degraded
rate_limit_failover_replicas: 10
request_rate_limit_rpm: 30000
request_rate_limit_burst: 3000
token_rate_limit_tpm: 5000000
token_rate_limit_burst: 500000
estimated_chars_per_token: 1
compliance:
enabled: true
@@ -228,7 +239,7 @@ retrieval:
generation:
queue_size: 128
worker_concurrency: 8
worker_concurrency: 24
stream_enabled: false
article_timeout: 8m
task_lease_ttl: 10m