bvandeusenandClaude Opus 5
dc8af8b1a7
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 30s
Build images / build-web (push) Successful in 55s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m45s
Build images / promote (push) Skipped
CI / integration (push) Successful in 1m49s
Nothing in FabledCurator knew what was SUPPOSED to be running. `celery inspect` reports the workers that ANSWER, so a dead worker was a shorter list rather than a red light, and grep for any notion of expected services returned nothing. That is why Portainer was the only place an operator could see it: Portainer knows the intended set. `service_seen` is the memory that makes an absence observable — every part that has checked in, and when it last did. **Keyed on the queue set, not the worker hostname.** Celery's worker names here are `celery@<container id>`, minted fresh on every deploy. Keyed on those, this table would record a death and a birth every time the stack updates — and a status page that goes red on every deploy is a status page nobody reads, which is worse than not having one. CELERY_QUEUES is assigned per role in compose and survives container replacement, so it is the stable identity. Two replicas of a role are therefore ONE row, which is right: the question is whether the role is served, not how many containers exist. The GPU agent is keyed on agent_id, the identity its lease protocol already uses. gpu.py received it on both lease and heartbeat and threw it away — an idle agent with nothing to lease left no trace and was indistinguishable from one switched off a week ago. Now recorded on the calls that were already happening. **Who observes, corrected from the plan.** The plan said "record from the existing inspect path", which would only run when someone opened the Activity tab. Two other candidates and why they lost: - A beat sweep. If the scheduler dies the sweep stops, every row goes stale, and the page says everything is down when one thing is. An alarm that cannot distinguish "a part died" from "the observer died" is worse than none. - A background task in web. hypercorn runs --workers 4, so that is four concurrent inspect loops per container, forever. Taken instead: refresh on demand, rate-limited by the newest last_seen_at that every process can already see. The observer is then the thing serving the page — if web is down you get a browser error, not a confidently green page — and it self-limits with no coordination, since a race costs one redundant inspect that writes identical values. Migration 0090 is the first written on the collapsed baseline (milestone 328), so it is also the first evidence the chain steps FORWARD from 0089 rather than merely reproducing the schema. No secondary indexes: one row per moving part means every read is a handful of rows, and #3301 is the record of what speculative indexes cost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA
65 lines
2.4 KiB
Python
65 lines
2.4 KiB
Python
"""service_seen — the learned roster that makes a stopped part observable.
|
|
|
|
Milestone 365. Nothing in FabledCurator knew what was SUPPOSED to be running:
|
|
`celery inspect` reports the workers that answer, so a dead worker was a
|
|
shorter list rather than a red light, and the only surface that could tell an
|
|
operator otherwise was Portainer. This table is the memory that turns an
|
|
absence into something the app can see.
|
|
|
|
Keyed on the queue set for a celery role and on agent_id for the GPU agent —
|
|
NOT on the celery worker name, which here is `celery@<container id>` and is
|
|
minted fresh on every deploy. See the model docstring for why that choice is
|
|
the whole design.
|
|
|
|
## First migration on the collapsed baseline
|
|
|
|
0089 is the single generated baseline that replaced revisions 0001..0089
|
|
(milestone 328). This is the first revision written on top of it, so it is
|
|
also the first evidence that the chain steps forward from the collapse rather
|
|
than merely reproducing the schema — which nothing had demonstrated yet.
|
|
|
|
An existing install is at 0089 because it ran the real 0089; a fresh one is at
|
|
0089 because it ran the baseline. Both arrive here identically, which was the
|
|
property the collapse was designed around.
|
|
|
|
Revision ID: 0090
|
|
Revises: 0089
|
|
Create Date: 2026-09-02
|
|
|
|
"""
|
|
from typing import Sequence, Union
|
|
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
|
|
revision: str = "0090"
|
|
down_revision: Union[str, None] = "0089"
|
|
branch_labels: Union[str, Sequence[str], None] = None
|
|
depends_on: Union[str, Sequence[str], None] = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
op.create_table(
|
|
"service_seen",
|
|
sa.Column("key", sa.String(length=128), nullable=False),
|
|
sa.Column("kind", sa.String(length=16), nullable=False),
|
|
sa.Column("display_name", sa.String(length=64), nullable=False),
|
|
sa.Column(
|
|
"first_seen_at", sa.DateTime(timezone=True),
|
|
server_default=sa.text("now()"), nullable=False,
|
|
),
|
|
sa.Column(
|
|
"last_seen_at", sa.DateTime(timezone=True),
|
|
server_default=sa.text("now()"), nullable=False,
|
|
),
|
|
sa.Column("details", sa.JSON(), nullable=False),
|
|
sa.PrimaryKeyConstraint("key", name=op.f("pk_service_seen")),
|
|
)
|
|
# No secondary indexes, deliberately: one row per moving part means every
|
|
# read is a handful of rows and an index would be write cost buying
|
|
# nothing (#3301 removed seven of exactly that shape).
|
|
|
|
|
|
def downgrade() -> None:
|
|
op.drop_table("service_seen")
|