bvandeusenandClaude Opus 5
dc8af8b1a7
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 30s
Build images / build-web (push) Successful in 55s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m45s
Build images / promote (push) Skipped
CI / integration (push) Successful in 1m49s
Nothing in FabledCurator knew what was SUPPOSED to be running. `celery inspect` reports the workers that ANSWER, so a dead worker was a shorter list rather than a red light, and grep for any notion of expected services returned nothing. That is why Portainer was the only place an operator could see it: Portainer knows the intended set. `service_seen` is the memory that makes an absence observable — every part that has checked in, and when it last did. **Keyed on the queue set, not the worker hostname.** Celery's worker names here are `celery@<container id>`, minted fresh on every deploy. Keyed on those, this table would record a death and a birth every time the stack updates — and a status page that goes red on every deploy is a status page nobody reads, which is worse than not having one. CELERY_QUEUES is assigned per role in compose and survives container replacement, so it is the stable identity. Two replicas of a role are therefore ONE row, which is right: the question is whether the role is served, not how many containers exist. The GPU agent is keyed on agent_id, the identity its lease protocol already uses. gpu.py received it on both lease and heartbeat and threw it away — an idle agent with nothing to lease left no trace and was indistinguishable from one switched off a week ago. Now recorded on the calls that were already happening. **Who observes, corrected from the plan.** The plan said "record from the existing inspect path", which would only run when someone opened the Activity tab. Two other candidates and why they lost: - A beat sweep. If the scheduler dies the sweep stops, every row goes stale, and the page says everything is down when one thing is. An alarm that cannot distinguish "a part died" from "the observer died" is worse than none. - A background task in web. hypercorn runs --workers 4, so that is four concurrent inspect loops per container, forever. Taken instead: refresh on demand, rate-limited by the newest last_seen_at that every process can already see. The observer is then the thing serving the page — if web is down you get a browser error, not a confidently green page — and it self-limits with no coordination, since a race costs one redundant inspect that writes identical values. Migration 0090 is the first written on the collapsed baseline (milestone 328), so it is also the first evidence the chain steps FORWARD from 0089 rather than merely reproducing the schema. No secondary indexes: one row per moving part means every read is a handful of rows, and #3301 is the record of what speculative indexes cost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA
89 lines
3.9 KiB
Python
89 lines
3.9 KiB
Python
"""service_seen — the learned roster of FabledCurator's own moving parts.
|
|
|
|
Nothing else in this application knows what is SUPPOSED to be running.
|
|
`celery inspect` reports the workers that answer, so a stopped worker is a
|
|
shorter list rather than a red light, and Postgres and Redis have no
|
|
representation at all. That is why the only place an operator could see a
|
|
dead service was Portainer, which knows the intended set (milestone 365).
|
|
|
|
This table is the memory that makes an absence observable: every part that
|
|
has ever checked in, and when it last did. A row that stops advancing is a
|
|
part that stopped.
|
|
|
|
## Why the key is not the hostname
|
|
|
|
`_read_workers_sync()` returns celery's worker names, which here are
|
|
`celery@<container id>`. Those are minted fresh on every deploy. Keyed on
|
|
them, this table would record a death and a birth every time the stack is
|
|
updated — and a status page that goes red on every deploy is a status page
|
|
nobody reads, which is worse than not having one.
|
|
|
|
So a celery role is keyed on its **queue set**, which is assigned per role in
|
|
docker-compose.yml (`CELERY_QUEUES`) and survives container replacement:
|
|
|
|
default,import,thumbnail,download -> worker
|
|
maintenance,scan -> scheduler (celery worker --beat)
|
|
ml -> ml-worker
|
|
|
|
Two replicas of one role share a queue set and are therefore ONE row — which
|
|
is right, because the question being answered is "is that role being served",
|
|
not "how many containers exist". The replica count and their hostnames go in
|
|
`details`, where they can change without the identity changing.
|
|
|
|
The GPU agent is keyed on its `agent_id`, the identity its lease protocol
|
|
already uses (`api/gpu.py`).
|
|
|
|
## What is NOT in here
|
|
|
|
Postgres and Redis. They are always expected and never learned, and a
|
|
last-seen for them would be actively misleading — that one answered thirty
|
|
seconds ago says nothing about now. They are probed live at request time.
|
|
|
|
## kind
|
|
|
|
Plain `String`, not a Postgres ENUM and not CHECK-gated, matching
|
|
`gpu_job.status` and `backup_run.status`. The value set here is expected to
|
|
grow as parts are added, and a constraint swap per new kind (rule 36) would
|
|
be cost with no invariant behind it.
|
|
|
|
celery — a worker role, keyed on its queue set
|
|
agent — a GPU agent, keyed on its agent_id
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import JSON, DateTime, String, func
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from .base import Base
|
|
|
|
|
|
class ServiceSeen(Base):
|
|
__tablename__ = "service_seen"
|
|
|
|
# No indexes beyond the primary key, deliberately. This table holds one row
|
|
# per moving part — a handful, forever — so every query against it is a
|
|
# full read of a few rows and an index would be write cost buying nothing
|
|
# (the lesson of #3301, which removed seven redundant ones).
|
|
key: Mapped[str] = mapped_column(String(128), primary_key=True)
|
|
kind: Mapped[str] = mapped_column(String(16), nullable=False)
|
|
|
|
# What to call it in the UI. Derived from the queue set where it is
|
|
# recognised, and falling back to the raw queue list where it is not — a
|
|
# deployment that slices its queues differently should still show something
|
|
# true rather than a name this code invented for it.
|
|
display_name: Mapped[str] = mapped_column(String(64), nullable=False)
|
|
|
|
first_seen_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now(),
|
|
)
|
|
last_seen_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now(),
|
|
)
|
|
|
|
# The parts that change without changing identity: replica hostnames,
|
|
# active task counts, the queues actually being served. Kept as a blob
|
|
# because it is displayed and never queried — giving it columns would
|
|
# invite filtering on it, which is what the activity endpoints are for.
|
|
details: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|