feat(llm): add configurable request/token rate limiting for LLM client
Introduce process-local and Redis-backed global rate limiting for the shared LLM client, plus a dedicated generation consumer prefetch knob. - Add request (RPM) and token (TPM) rate limits with burst controls, process or global scope, and fail-closed/process-degraded failover - Wire the LLM client through NewWithRedis so limits can be shared across replicas via a Redis token bucket - Add generation_consumer_prefetch to tune RabbitMQ generation consumers independently of the default prefetch - Bump generation worker_concurrency to 24 and promote golang.org/x/time to a direct dependency - Drop t.Parallel() from a few middleware/bootstrap tests Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -206,7 +206,7 @@ metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: worker-generate
|
||||
spec:
|
||||
replicas: 1
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: worker-generate
|
||||
|
||||
@@ -83,6 +83,7 @@ rabbitmq:
|
||||
compliance_review_dlq: compliance.review.run.dlq
|
||||
compliance_review_dlq_routing_key: compliance.review.run.dlq
|
||||
consumer_prefetch: 10
|
||||
generation_consumer_prefetch: 1
|
||||
publish_channel_pool_size: 32
|
||||
|
||||
scheduler:
|
||||
@@ -200,6 +201,16 @@ llm:
|
||||
temperature: 0.7
|
||||
reasoning_effort: minimal
|
||||
web_search_limit: 3
|
||||
# Global Redis-backed budget shared by all replicas.
|
||||
rate_limit_scope: global
|
||||
rate_limit_key: ark:doubao-seed-2-0
|
||||
rate_limit_failover_mode: process_degraded
|
||||
rate_limit_failover_replicas: 10
|
||||
request_rate_limit_rpm: 30000
|
||||
request_rate_limit_burst: 3000
|
||||
token_rate_limit_tpm: 5000000
|
||||
token_rate_limit_burst: 500000
|
||||
estimated_chars_per_token: 1
|
||||
|
||||
compliance:
|
||||
enabled: true
|
||||
@@ -228,7 +239,7 @@ retrieval:
|
||||
|
||||
generation:
|
||||
queue_size: 128
|
||||
worker_concurrency: 8
|
||||
worker_concurrency: 24
|
||||
stream_enabled: false
|
||||
article_timeout: 8m
|
||||
task_lease_ttl: 10m
|
||||
|
||||
Reference in New Issue
Block a user