diff --git a/deploy/OFFLINE_PACKAGE.md b/deploy/OFFLINE_PACKAGE.md index b7779c3..073e67f 100644 --- a/deploy/OFFLINE_PACKAGE.md +++ b/deploy/OFFLINE_PACKAGE.md @@ -20,6 +20,8 @@ bash deploy.sh docker compose -f docker-compose.yaml -f docker-compose.offline.yaml --env-file .env up -d ``` +Runtime services connect to Postgres through PgBouncer in `session` pooling mode. The one-shot `migrate` job still connects directly to `postgres` and `monitoring-postgres` so DDL and seed work do not go through the pooler. + On a k3s host: ```bash diff --git a/deploy/k3s/README.md b/deploy/k3s/README.md index c5c095a..9427c95 100644 --- a/deploy/k3s/README.md +++ b/deploy/k3s/README.md @@ -2,7 +2,7 @@ 这套清单用于把当前 Docker Compose 拓扑搬到单节点或小集群 k3s 中。默认包含： -- infra: `postgres`, `monitoring-postgres`, `rabbitmq`, `redis`, `qdrant`, `minio` +- infra: `postgres`, `monitoring-postgres`, `pgbouncer`, `monitoring-pgbouncer`, `rabbitmq`, `redis`, `qdrant`, `minio` - jobs: `migrate`, `minio-init` - apps: `tenant-api`, `worker-generate`, `kol-assist-worker`, `scheduler`, `ops-api`, `frontend`, `ops-web` - ingress: `geo-rankly.local` -> `frontend`, `ops.geo-rankly.local` -> `ops-web` @@ -38,7 +38,7 @@ sudo k3s ctr images import images.tar - `config/config.yaml` 里的 LLM / SiliconFlow API key - `config.local.yaml` 里对应的数据库、RabbitMQ、MinIO、JWT 配置 -注意：`ops-api` 目前从 `deploy/k3s/config/ops-config.yaml` 读取数据库连接信息；如果改了 Postgres 密码，也同步改这个文件。 +注意：应用运行时通过 `pgbouncer` / `monitoring-pgbouncer` 访问数据库；`migrate` 任务仍然直连 `postgres` / `monitoring-postgres`。数据库密码统一来自 `POSTGRES_PASSWORD` secret，应用配置和 PgBouncer 都会跟随这个 secret。 ## 3. 部署 diff --git a/findings.md b/findings.md index 211ead7..539e5da 100644 --- a/findings.md +++ b/findings.md @@ -236,3 +236,12 @@ - Gitea Registry secret name is `REGISTRY_PASSWORD`; Gitea rejects secret names beginning with `GITEA_`. - CD workflows verify required commit8 image tags in Gitea Registry before pulling or deploying. Missing images are treated as a failed prerequisite, not as a signal to build during CD. - Offline packaging pulls previously published Gitea Registry images into the package; it does not rebuild business images when `SOURCE_IMAGE_REGISTRY` is set by the workflow. + +## Stable PgBouncer Deployment - 2026-05-03 +- Before this change the deployment had no PgBouncer/pooler service. Runtime services used `pgxpool` and connected directly to `postgres` / `monitoring-postgres`. +- Stable-first decision: add PgBouncer in `session` pooling mode, not transaction pooling, so current `pgx` behavior and any session-level assumptions remain compatible. +- Runtime services now target `pgbouncer` and `monitoring-pgbouncer`; migration and seed jobs continue to connect directly to the underlying Postgres services. +- Application-side pools are intentionally smaller: tenant runtime config uses 12 main DB connections and 8 monitoring DB connections per process; ops runtime uses 5 for each DB. +- Offline packages must include `edoburu/pgbouncer:v1.24.1-p1` alongside the existing infra images. +- Postgres passwords are passed through environment placeholders instead of hard-coded in deploy configs. The Go DSN builder now URL-escapes credentials so strong passwords containing `@`, `:`, `/`, or `?` remain valid. +- Local smoke validation used the existing `server/docker-compose.yaml` Postgres containers on `server_default`; PgBouncer successfully returned `SELECT 1` through the pooler and the temporary test container was removed. diff --git a/progress.md b/progress.md index f99e319..0149d56 100644 --- a/progress.md +++ b/progress.md @@ -706,3 +706,20 @@ - Added commit8 image verification to compose/k3s/offline deployment paths so CD fails if CI has not produced the required image. - Changed offline package defaults to use commit8 instead of tag names and to pull business images from the Gitea Registry. - Verification passed: workflow YAML parse, deploy script `bash -n`, and `sh -n` over all workflow run blocks. + +## 2026-05-03T03:03:09Z - Stable PgBouncer Deployment + +- Confirmed the existing repository topology uses application-side `pgxpool` and direct Postgres service names; no PgBouncer service/config existed before this change. +- Started Phase 40 to add stable PgBouncer deployment with session pooling, app runtime traffic through PgBouncer, and migration jobs remaining direct to Postgres. +- Added `pgbouncer` and `monitoring-pgbouncer` to production Docker Compose and k3s using `edoburu/pgbouncer:v1.24.1-p1`, `POOL_MODE=session`, and `AUTH_TYPE=scram-sha-256`. +- Changed runtime app configs to target PgBouncer while keeping Docker/k3s migration jobs direct to `postgres` / `monitoring-postgres`. +- Lowered application-side Postgres pools: tenant runtime uses 12 main / 8 monitoring connections per process; ops runtime uses 5 / 5. +- Made deployment passwords flow from `POSTGRES_PASSWORD` / `OPS_POSTGRES_PASSWORD` placeholders and changed `DatabaseConfig.DSN()` to URL-escape credentials for stronger production passwords. +- Included PgBouncer in offline packaging and docs. +- Verification passed: + - `docker compose -f deploy/docker-compose.yaml --env-file deploy/.env.example config` + - `kubectl kustomize deploy/k3s` plus local YAML parse + - `bash -n deploy/scripts/package.sh` + - `git diff --check` + - `go test ./internal/shared/config ./internal/shared/repository/postgres ./internal/ops/config` + - Temporary PgBouncer container attached to existing `server_default` network returned `SELECT 1` through PgBouncer, then was removed. diff --git a/task_plan.md b/task_plan.md index 7149305..1a7831b 100644 --- a/task_plan.md +++ b/task_plan.md @@ -4,7 +4,7 @@ Continue the desktop AI monitoring implementation by first adding a client-side local scheduler with durable queueing and adaptive execution policy, then adding a reusable hidden Playwright CDP execution layer that attaches to Electron Chromium and reuses existing account partitions. ## Current Phase -Phase 37 +Phase 40 ## Phases ### Phase 1: Progress Verification @@ -276,6 +276,14 @@ Phase 37 - [x] Run workflow YAML and shell syntax checks - **Status:** complete +### Phase 40: Stable PgBouncer Deployment +- [x] Add PgBouncer infrastructure to Docker Compose and k3s +- [x] Route runtime services through PgBouncer while keeping migrations direct to Postgres +- [x] Lower application-side connection pools for stability +- [x] Include PgBouncer in offline packaging and deployment docs +- [x] Validate rendered deployment/config files +- **Status:** complete + ## Key Questions 1. How should the desktop client defer and locally optimize monitor-task execution without violating one-task-per-platform serialism? 2. What is the smallest durable local cache that allows same-day resume while safely dropping stale next-day monitor tasks? @@ -343,6 +351,7 @@ Phase 37 | `connectOverCDP()` is not wired yet even though `playwright-core` is installed | 1 | Add a dedicated hidden-browser manager instead of pushing CDP code directly into adapters or the runtime controller | | `pnpm --filter @geo/desktop-client typecheck` fails in `electron.vite.config.ts` due to a pre-existing Vite 5/8 plugin type mismatch | 1 | Verified this change with `pnpm --filter @geo/desktop-client build` and `pnpm --filter @geo/desktop-client test`; leave config dependency alignment as a separate fix | | `kubectl apply --dry-run=client` still attempted to contact the local kube-apiserver, which is not running at `127.0.0.1:26443` | 1 | Verified with `kubectl kustomize`, parsed the rendered YAML locally, and recorded that live API validation should run on the target k3s cluster | +| Production compose PgBouncer smoke test could not start bundled Postgres containers because existing `server/docker-compose.yaml` containers already own `geo-postgres` / `geo-monitoring-postgres` names | 1 | Did not stop user/local DB containers; verified PgBouncer with a temporary test container attached to the existing `server_default` network and removed it after `SELECT 1` passed | ## Notes - Re-check task_plan.md before major implementation decisions.