DB maintenance + health observability + Postgres integration CI lane #72

Merged
bvandeusen merged 5 commits from dev into main 2026-06-14 20:54:34 -04:00

5 Commits

Author SHA1 Message Date
bvandeusen 4f31890bde test(ci): dispose engine between integration tests (per-loop pool)
CI & Build / Python lint (push) Successful in 2s
CI & Build / integration (push) Successful in 14s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / Python tests (push) Successful in 43s
CI & Build / Build & push image (push) Successful in 13s
First integration run proved the lane works (run_maintenance test passed against
real Postgres), but the health test failed with 'Future attached to a different
loop': pytest-asyncio uses a fresh loop per test while the app's module-level
engine pools a connection from the prior test's loop. Dispose the engine in each
test's teardown so the next test starts with an empty pool on its own loop.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 19:24:09 -04:00
bvandeusen 2ad2e943f3 test(ci): add Postgres integration lane + real run_maintenance guard
CI & Build / Python lint (push) Successful in 2s
CI & Build / integration (push) Failing after 19s
CI & Build / TypeScript typecheck (push) Successful in 20s
CI & Build / Python tests (push) Successful in 45s
CI & Build / Build & push image (push) Successful in 1m1s
The unit suite can't catch sync/async API mismatches against SQLAlchemy (an
un-awaited execution_options passed green CI but failed at runtime: VACUUM 0/6).
Add a real-Postgres integration lane modelled on the family pattern (rules
6/79-82): a new CI 'integration' job with a postgres:16 service, bridge-IP
discovery, busybox-safe readiness wait, and 'alembic upgrade head', running
pytest -m integration. Non-gating, like the unit lane.

- tests/test_integration_db_maintenance.py: runs run_maintenance() and
  get_table_health() against real Postgres; asserts all allowlisted tables
  vacuum OK (the await regression makes this fail) and health reports real stats.
- pyproject: register the 'integration' marker.
- conftest: integration-marked tests use the real DATABASE_URL, not the stub.
- ci.yml: unit 'test' job now runs -m 'not integration'.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 19:19:18 -04:00
bvandeusen e6c89f6b88 fix(db): await AsyncConnection.execution_options in run_maintenance
CI & Build / Python lint (push) Successful in 2s
CI & Build / TypeScript typecheck (push) Successful in 20s
CI & Build / Python tests (push) Successful in 48s
CI & Build / Build & push image (push) Successful in 1m0s
execution_options() is a coroutine on AsyncConnection and must be awaited;
the un-awaited call returned a coroutine, so exec_driver_sql() blew up with
AttributeError and every table's VACUUM was skipped (Run-now reported 0/6).
A prior change had wrongly dropped the await. Fix it and make the test mock
execution_options async so this call shape is actually exercised.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 18:31:30 -04:00
bvandeusen 96079d5b77 feat(db): table-health readout — per-table bloat metrics in admin card
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 32s
CI & Build / Python tests (push) Successful in 44s
CI & Build / Build & push image (push) Successful in 52s
You can't decide what to maintain without seeing what's bloating. Adds a
read-only health panel driven by Postgres' own statistics views.

- services/db_maintenance.py: get_table_health() queries pg_stat_user_tables +
  pg_total_relation_size + pg_database_size — per-table size, live/dead tuples,
  dead-tuple ratio (the bloat signal), and last (auto)vacuum/(auto)analyze.
- routes/admin.py: admin-only GET /api/admin/db-maintenance/health.
- SettingsView.vue: 'Table health' table in the maintenance card, all tables
  sorted by dead tuples, rows >=20% dead-ratio flagged; total DB size shown;
  refreshes after a Run-now so the dead-tuple drop is visible.
- Tests: health row/size shaping + null-timestamp passthrough; route + service
  surface.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 17:53:24 -04:00
bvandeusen c4553d937c feat(db): scheduled DB maintenance — daily targeted VACUUM (ANALYZE)
CI & Build / Python lint (push) Successful in 3s
CI & Build / TypeScript typecheck (push) Successful in 33s
CI & Build / Python tests (push) Successful in 48s
CI & Build / Build & push image (push) Successful in 54s
Adds a daily off-hours VACUUM (ANALYZE) over the high-churn tables the
retention/purge sweeps churn (app_logs, notifications, token tables, notes,
note_versions), on top of Postgres autovacuum, to reclaim bloat left by the
nightly bulk DELETEs and keep planner stats fresh.

- services/db_maintenance.py: run_maintenance() over a closed table allowlist
  via an AUTOCOMMIT connection (VACUUM can't run in a txn); per-table summary
  persisted as the db_maintenance_last_run admin setting.
- services/db_maintenance_scheduler.py: BackgroundScheduler cron (default
  04:00 UTC, after the 03:30 trash purge); enabled-gate checked at fire time;
  live reschedule on hour change. Wired into app.py start/stop.
- routes/admin.py: admin-only GET/PUT /api/admin/db-maintenance + POST /run.
- settings.py: set_admin_setting() (write-side of get_admin_setting) for
  out-of-request writes.
- SettingsView.vue: admin 'Database maintenance' card — enable toggle, run-hour
  (UTC), Run-now, last-run summary.
- Tests: allowlist is closed, VACUUM issued per table, one failure doesn't
  abort the rest, summary persisted; route/scheduler/service surface.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 16:42:04 -04:00