DB maintenance + health observability + Postgres integration CI lane #72

Merged

bvandeusen merged 5 commits from dev into main 2026-06-14 20:54:34 -04:00

Conversation 0 Commits 5 Files Changed 12

+915 -4

bvandeusen commented 2026-06-14 20:54:24 -04:00

Owner

Copy Link

Copy Source

Rolls up dev into main. Headline of this batch is basic Postgres maintenance + the visibility to use it, plus a real-DB CI lane.

Database maintenance (#838)

• Daily targeted VACUUM (ANALYZE) over the six high-churn tables the retention/purge sweeps churn (app_logs, notifications, the token tables, notes, note_versions), complementing autovacuum.
• services/db_maintenance.py runs over a closed allowlist via an AUTOCOMMIT connection (VACUUM can't run in a txn); per-table summary; one failure doesn't abort the rest.
• services/db_maintenance_scheduler.py: BackgroundScheduler, default 04:00 UTC (after the 03:30 trash purge); enable-gate at fire time; live reschedule on hour change.
• Admin Settings card: enable toggle, run-hour, Run-now, last-run summary.

Table-health observability (#839)

• Read-only get_table_health() over pg_stat_user_tables + size functions: per-table size, live/dead tuples, dead-tuple ratio (the bloat signal), last (auto)vacuum/(auto)analyze, total DB size.
• "Table health" readout in the maintenance card — all tables sorted by dead tuples, ≥20% flagged; auto-refreshes after Run-now.

Postgres integration CI lane

• New non-gating integration job (rules 6/79–82): postgres:16-alpine service, bridge-IP discovery, busybox-safe readiness wait, alembic upgrade head, pytest -m integration. Unit test job now runs -m "not integration".
• test_integration_db_maintenance.py runs run_maintenance()/get_table_health() against real Postgres — the guard that catches the sync/async API class of bug unit mocks can't (an un-awaited execution_options, fixed here, that reported VACUUM 0/6).

Also brings along any earlier dev work not yet on main. All CI-green on dev head 4f31890.

🤖 Generated with Claude Code

Rolls up `dev` into `main`. Headline of this batch is basic Postgres maintenance + the visibility to use it, plus a real-DB CI lane. ## Database maintenance (#838) - Daily targeted `VACUUM (ANALYZE)` over the six high-churn tables the retention/purge sweeps churn (`app_logs`, `notifications`, the token tables, `notes`, `note_versions`), complementing autovacuum. - `services/db_maintenance.py` runs over a **closed allowlist** via an AUTOCOMMIT connection (VACUUM can't run in a txn); per-table summary; one failure doesn't abort the rest. - `services/db_maintenance_scheduler.py`: BackgroundScheduler, default 04:00 UTC (after the 03:30 trash purge); enable-gate at fire time; live reschedule on hour change. - Admin Settings card: enable toggle, run-hour, Run-now, last-run summary. ## Table-health observability (#839) - Read-only `get_table_health()` over `pg_stat_user_tables` + size functions: per-table size, live/dead tuples, **dead-tuple ratio** (the bloat signal), last (auto)vacuum/(auto)analyze, total DB size. - "Table health" readout in the maintenance card — all tables sorted by dead tuples, ≥20% flagged; auto-refreshes after Run-now. ## Postgres integration CI lane - New non-gating `integration` job (rules 6/79–82): `postgres:16-alpine` service, bridge-IP discovery, busybox-safe readiness wait, `alembic upgrade head`, `pytest -m integration`. Unit `test` job now runs `-m "not integration"`. - `test_integration_db_maintenance.py` runs `run_maintenance()`/`get_table_health()` against real Postgres — the guard that catches the sync/async API class of bug unit mocks can't (an un-awaited `execution_options`, fixed here, that reported VACUUM 0/6). Also brings along any earlier `dev` work not yet on `main`. All CI-green on `dev` head 4f31890. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 5 commits 2026-06-14 20:54:25 -04:00

feat(db): scheduled DB maintenance — daily targeted VACUUM (ANALYZE) ... c4553d937c

Adds a daily off-hours VACUUM (ANALYZE) over the high-churn tables the
retention/purge sweeps churn (app_logs, notifications, token tables, notes,
note_versions), on top of Postgres autovacuum, to reclaim bloat left by the
nightly bulk DELETEs and keep planner stats fresh.

- services/db_maintenance.py: run_maintenance() over a closed table allowlist
  via an AUTOCOMMIT connection (VACUUM can't run in a txn); per-table summary
  persisted as the db_maintenance_last_run admin setting.
- services/db_maintenance_scheduler.py: BackgroundScheduler cron (default
  04:00 UTC, after the 03:30 trash purge); enabled-gate checked at fire time;
  live reschedule on hour change. Wired into app.py start/stop.
- routes/admin.py: admin-only GET/PUT /api/admin/db-maintenance + POST /run.
- settings.py: set_admin_setting() (write-side of get_admin_setting) for
  out-of-request writes.
- SettingsView.vue: admin 'Database maintenance' card — enable toggle, run-hour
  (UTC), Run-now, last-run summary.
- Tests: allowlist is closed, VACUUM issued per table, one failure doesn't
  abort the rest, summary persisted; route/scheduler/service surface.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(db): table-health readout — per-table bloat metrics in admin card ... 96079d5b77

You can't decide what to maintain without seeing what's bloating. Adds a
read-only health panel driven by Postgres' own statistics views.

- services/db_maintenance.py: get_table_health() queries pg_stat_user_tables +
  pg_total_relation_size + pg_database_size — per-table size, live/dead tuples,
  dead-tuple ratio (the bloat signal), and last (auto)vacuum/(auto)analyze.
- routes/admin.py: admin-only GET /api/admin/db-maintenance/health.
- SettingsView.vue: 'Table health' table in the maintenance card, all tables
  sorted by dead tuples, rows >=20% dead-ratio flagged; total DB size shown;
  refreshes after a Run-now so the dead-tuple drop is visible.
- Tests: health row/size shaping + null-timestamp passthrough; route + service
  surface.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

fix(db): await AsyncConnection.execution_options in run_maintenance ... e6c89f6b88

execution_options() is a coroutine on AsyncConnection and must be awaited;
the un-awaited call returned a coroutine, so exec_driver_sql() blew up with
AttributeError and every table's VACUUM was skipped (Run-now reported 0/6).
A prior change had wrongly dropped the await. Fix it and make the test mock
execution_options async so this call shape is actually exercised.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

test(ci): add Postgres integration lane + real run_maintenance guard ... 2ad2e943f3

The unit suite can't catch sync/async API mismatches against SQLAlchemy (an
un-awaited execution_options passed green CI but failed at runtime: VACUUM 0/6).
Add a real-Postgres integration lane modelled on the family pattern (rules
6/79-82): a new CI 'integration' job with a postgres:16 service, bridge-IP
discovery, busybox-safe readiness wait, and 'alembic upgrade head', running
pytest -m integration. Non-gating, like the unit lane.

- tests/test_integration_db_maintenance.py: runs run_maintenance() and
  get_table_health() against real Postgres; asserts all allowlisted tables
  vacuum OK (the await regression makes this fail) and health reports real stats.
- pyproject: register the 'integration' marker.
- conftest: integration-marked tests use the real DATABASE_URL, not the stub.
- ci.yml: unit 'test' job now runs -m 'not integration'.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

test(ci): dispose engine between integration tests (per-loop pool) ... 4f31890bde

First integration run proved the lane works (run_maintenance test passed against
real Postgres), but the health test failed with 'Future attached to a different
loop': pytest-asyncio uses a fresh loop per test while the app's module-level
engine pools a connection from the prior test's loop. Dispose the engine in each
test's teardown so the next test starts with an empty pool on its own loop.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

bvandeusen merged commit 6cdac307af into main 2026-06-14 20:54:34 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledScribe#72