FabledCurator can now tell you one of its own parts has stopped #250

Merged

bvandeusen merged 3 commits from dev into main 2026-09-02 18:01:39 -04:00

Conversation 0 Commits 3 Files Changed 12

+793 -8

bvandeusen commented 2026-09-02 18:01:32 -04:00

Owner

Copy Link

Copy Source

Milestone 365, all four steps. Operator, on the live instance: "the main UI doesn't have a surface or path for checking and rendering the health state of the other parts of curator. the only UI for knowing if things are running out is through my docker management surface portainer."

Why the existing surfaces could not answer that

More existed than nothing. /api/system/activity/workers runs a live celery inspect, QueuesTable shows per-queue depth beside worker counts and even warns when a queue has depth with no workers.

All of it reports PRESENCE. celery inspect returns the workers that answer, so a dead worker is a shorter list, not a red light. There was no roster anywhere in the codebase, so nothing could observe an absence. Portainer can, because Portainer knows what is supposed to be running.

dc8af8b — the learned roster, keyed on what does not churn

service_seen remembers every part that has checked in and when it last did.

Keyed on the queue set, not the worker hostname. Celery's worker names here are celery@<container id>, minted fresh on every deploy — keyed on those, this table would record a death and a birth every time the stack updates, and a status page that reddens on every deploy is one nobody reads. CELERY_QUEUES is assigned per role in compose and survives container replacement. Two replicas of a role are therefore ONE row, which is right: the question is whether the role is served.

The GPU agent is keyed on agent_id. It cannot be polled — it is HTTP-only and pulls work from web — and gpu.py was already receiving that id on lease and heartbeat and discarding it, so an idle agent left no trace and was indistinguishable from one switched off a week ago.

Migration 0090 is the first written on the collapsed baseline (milestone 328), so it is also the first evidence the chain steps forward from 0089 rather than merely reproducing the schema.

fe4e0f2 — /api/system/health

Learned parts from the table; Postgres and Redis probed live, because a last-seen for them would be actively misleading.

The endpoint must never fail because something it checks has failed. That inversion destroys the feature exactly when it is needed — a 500 when Redis is down instead of redis: down. Every probe is wrapped and deadlined.

Staleness thresholds are deliberately generous (90s / 300s). The constraint is a deploy, not a crash: docker compose up -d rolls start-first, so a role is briefly served by two containers and then neither.

5084ba6 — the dot beside the brand

There were already three partial surfaces: TopNav's health dot, PipelineStatusChip's pulse, the Activity tab. A fourth would have made the question harder to answer.

TopNav's dot read /api/health — a no-DB liveness check proving only that the web container is serving. Green there while a worker was dead is exactly what it looked like, and a green dot beside the product name is read as "everything is fine". It now carries the whole-stack verdict, names the actual problem in its tooltip, and links to /system.

Corrections to the plan, made while building

• The observer is the web process, refreshing on demand and rate-limited by the newest last_seen_at. A beat sweep would mean a dead scheduler makes everything look dead — an alarm that cannot tell "a part died" from "the observer died". A before_serving task would be four concurrent inspect loops, since hypercorn runs --workers 4.
• Re-used the existing indicator rather than adding one, per rule 28.

Verification

CI green on 5084ba6, all five lanes; 0090 applies from empty in the integration lane.

Not yet verified against a real stopped service, and it needs an operator: deploy, stop a worker, watch the dot change without navigating — then redeploy and confirm it does NOT redden, since the hostname-churn trap is the most likely way this ships broken and only shows on a deploy. Frontend CI has no Vue type-check or visual regression, so the page needs a look rather than a green lane.

Prod runs 0421fd3 and is unaffected until the next deploy.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

Milestone 365, all four steps. Operator, on the live instance: *"the main UI doesn't have a surface or path for checking and rendering the health state of the other parts of curator. the only UI for knowing if things are running out is through my docker management surface portainer."* ## Why the existing surfaces could not answer that More existed than nothing. `/api/system/activity/workers` runs a live `celery inspect`, `QueuesTable` shows per-queue depth beside worker counts and even warns when a queue has depth with no workers. **All of it reports PRESENCE.** `celery inspect` returns the workers that answer, so a dead worker is a *shorter list*, not a red light. There was no roster anywhere in the codebase, so nothing could observe an absence. Portainer can, because Portainer knows what is supposed to be running. ## `dc8af8b` — the learned roster, keyed on what does not churn `service_seen` remembers every part that has checked in and when it last did. **Keyed on the queue set, not the worker hostname.** Celery's worker names here are `celery@<container id>`, minted fresh on every deploy — keyed on those, this table would record a death and a birth every time the stack updates, and a status page that reddens on every deploy is one nobody reads. `CELERY_QUEUES` is assigned per role in compose and survives container replacement. Two replicas of a role are therefore ONE row, which is right: the question is whether the role is served. The GPU agent is keyed on `agent_id`. It cannot be polled — it is HTTP-only and *pulls* work from web — and `gpu.py` was already receiving that id on lease and heartbeat and discarding it, so an idle agent left no trace and was indistinguishable from one switched off a week ago. **Migration `0090` is the first written on the collapsed baseline** (milestone 328), so it is also the first evidence the chain steps *forward* from `0089` rather than merely reproducing the schema. ## `fe4e0f2` — `/api/system/health` Learned parts from the table; Postgres and Redis probed live, because a last-seen for them would be actively misleading. **The endpoint must never fail because something it checks has failed.** That inversion destroys the feature exactly when it is needed — a 500 when Redis is down instead of `redis: down`. Every probe is wrapped and deadlined. Staleness thresholds are deliberately generous (90s / 300s). The constraint is a deploy, not a crash: `docker compose up -d` rolls start-first, so a role is briefly served by two containers and then neither. ## `5084ba6` — the dot beside the brand There were already **three** partial surfaces: TopNav's health dot, PipelineStatusChip's pulse, the Activity tab. A fourth would have made the question harder to answer. TopNav's dot read `/api/health` — a no-DB liveness check proving only that the *web container* is serving. Green there while a worker was dead is exactly what it looked like, and a green dot beside the product name is read as "everything is fine". It now carries the whole-stack verdict, names the actual problem in its tooltip, and links to `/system`. ## Corrections to the plan, made while building - **The observer is the web process**, refreshing on demand and rate-limited by the newest `last_seen_at`. A beat sweep would mean a dead scheduler makes everything look dead — an alarm that cannot tell "a part died" from "the observer died". A `before_serving` task would be four concurrent inspect loops, since hypercorn runs `--workers 4`. - **Re-used the existing indicator** rather than adding one, per rule 28. ## Verification CI green on `5084ba6`, all five lanes; `0090` applies from empty in the integration lane. **Not yet verified against a real stopped service**, and it needs an operator: deploy, stop a worker, watch the dot change without navigating — then redeploy and confirm it does NOT redden, since the hostname-churn trap is the most likely way this ships broken and only shows on a deploy. Frontend CI has no Vue type-check or visual regression, so the page needs a look rather than a green lane. Prod runs `0421fd3` and is unaffected until the next deploy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

bvandeusen added 3 commits 2026-09-02 18:01:32 -04:00

feat: a learned roster, so a stopped part is observable (milestone 365 steps 1-2) ... dc8af8b1a7

Nothing in FabledCurator knew what was SUPPOSED to be running. `celery
inspect` reports the workers that ANSWER, so a dead worker was a shorter list
rather than a red light, and grep for any notion of expected services returned
nothing. That is why Portainer was the only place an operator could see it:
Portainer knows the intended set.

`service_seen` is the memory that makes an absence observable — every part
that has checked in, and when it last did.

**Keyed on the queue set, not the worker hostname.** Celery's worker names
here are `celery@<container id>`, minted fresh on every deploy. Keyed on those,
this table would record a death and a birth every time the stack updates — and
a status page that goes red on every deploy is a status page nobody reads,
which is worse than not having one. CELERY_QUEUES is assigned per role in
compose and survives container replacement, so it is the stable identity. Two
replicas of a role are therefore ONE row, which is right: the question is
whether the role is served, not how many containers exist.

The GPU agent is keyed on agent_id, the identity its lease protocol already
uses. gpu.py received it on both lease and heartbeat and threw it away — an
idle agent with nothing to lease left no trace and was indistinguishable from
one switched off a week ago. Now recorded on the calls that were already
happening.

**Who observes, corrected from the plan.** The plan said "record from the
existing inspect path", which would only run when someone opened the Activity
tab. Two other candidates and why they lost:

- A beat sweep. If the scheduler dies the sweep stops, every row goes stale,
  and the page says everything is down when one thing is. An alarm that cannot
  distinguish "a part died" from "the observer died" is worse than none.
- A background task in web. hypercorn runs --workers 4, so that is four
  concurrent inspect loops per container, forever.

Taken instead: refresh on demand, rate-limited by the newest last_seen_at that
every process can already see. The observer is then the thing serving the page
— if web is down you get a browser error, not a confidently green page — and
it self-limits with no coordination, since a race costs one redundant inspect
that writes identical values.

Migration 0090 is the first written on the collapsed baseline (milestone 328),
so it is also the first evidence the chain steps FORWARD from 0089 rather than
merely reproducing the schema. No secondary indexes: one row per moving part
means every read is a handful of rows, and #3301 is the record of what
speculative indexes cost.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

feat: /api/system/health — one verdict for the whole stack (milestone 365 step 3) ... fe4e0f2b71

The single endpoint the nav indicator and the System page will both read.
Composing a verdict is this module's job, not the UI's.

Two kinds of part, answered differently. LEARNED — celery roles and the GPU
agent, out of service_seen, where the question is "how long since it checked
in" and the answer can be "it has not". PROBED — Postgres and Redis, always
expected, never learned, because a last-seen for them would be actively
misleading: that Redis answered thirty seconds ago says nothing about now.

**The endpoint must never fail because something it checks has failed.** That
inversion is easy to write by accident and it destroys the feature exactly
when it is needed — a 500 when Redis is down instead of `redis: down`. Every
probe is wrapped, every wait carries a deadline (rule 156), and the roster
refresh swallows its own errors. The worst case is a part reported `unknown`,
which is a true statement about the system.

Postgres is probed first and gates the rest, because if it is unreachable
nothing else can be read — and "the database is down" is the most useful
single thing this can ever say.

The staleness thresholds are the design risk, not the code, and they are
deliberately generous: 90s to doubt, 300s to disbelieve. The constraint is a
deploy rather than a crash — `docker compose up -d` rolls start-first, so a
role is briefly served by two containers and then by neither while the old one
drains. Thresholds tight enough to catch a crash in seconds would paint the
page red on every update, and an alarm that cries wolf on every deploy is one
nobody reads. Tune down only after watching a real deploy pass through. The
numbers ship in the response so the UI can explain a `stale` without keeping a
second copy of them.

States are described in sentences rather than left as chips: "Scheduler has
not checked in for 6 min — treat it as stopped" is what someone needs at the
moment they are deciding whether to go and open Portainer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

feat: the dot beside the brand now means the whole stack (milestone 365 step 4) ... 5084ba666b

The ask was a surface AND a path. The path is the part that was missing —
everything that could answer "is it running" lived inside Settings, which you
only open once you already suspect something.

**Re-used the indicator that already existed rather than adding a fourth.**
There were three partial surfaces: TopNav's health dot, PipelineStatusChip's
pulse, and the Settings Activity tab. None answered "is every part alive", and
a fourth would have made the question harder to answer, not easier.

TopNav's dot read /api/health — a no-DB liveness check proving only that the
WEB container is serving. Green there while a worker was dead is exactly what
it looked like, and a green dot beside the product name gets read as
"everything is fine". It now reflects the whole-stack verdict, and it is a
link: the place someone already looks when they suspect something is now also
the way to the detail.

The tooltip names the actual problem. "Scheduler has not checked in for 6 min"
sends someone somewhere; "something is unhealthy" sends them hunting.

/system is deliberately NOT in the nav row — TopNav builds that from routes
with a meta.title, and a sixth top-level tab for a page visited twice a year
costs more attention than it returns. It is reached from the dot.

The page lists every learned part with its state as a sentence rather than a
chip, and prints the staleness thresholds it was judged by, taken from the
endpoint so the UI keeps no second copy of them. PipelineStatusChip still
hand-rolls its own 3-minute scheduler window; that is now a duplicate of a
threshold the server owns, and worth collapsing once this has been watched
working.

The stores stay separate on purpose: system.js is "can I reach the API",
systemActivity.js is "what is the pipeline doing", systemHealth.js is "is
anything broken". Running and alive fail independently — an idle stack with a
dead worker looks identical to a healthy one on every activity surface, which
is the whole reason this milestone exists.

Not yet verified against a real stopped service. Rule 12 keeps a local stack
out of it, and frontend CI has no Vue type-check or visual regression, so this
needs an operator look rather than a green lane.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

bvandeusen merged commit a8fdbd86bc into main 2026-09-02 18:01:39 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#250