18032a4 re-checked a "not back in service" job every minute with no end,
so any later swarm wobble on the host (radarr's failed task during the
downloads move, say) flipped maintenance_failed back to 1: five false
firings and ten emails for Docker02 overnight on 10-19. And the overview's
finding never learned the rule at all, so it still said Docker02's
maintenance failed three days on.
The job now keeps came_back_at, set the first time its host is found back.
maintenance.still_failed(job) is the one answer: the figure, the overview,
the cadence page's run rows and the host panel all ask it. A failed update
or a timed-out job still flags its host until a job succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host page gains a "Who's sending" panel, and the MCP a tool, that
put the host's outbound (or inbound) traffic down to its containers and
NFS mounts: read in shared time buckets, so the host's busiest bucket
says who was sending in it. A swarm service's tasks add up under its
name; what none of them accounts for is the host itself.
Also fixes CI 9488: per-container net_rx_bps / net_tx_bps join the
docker alert catalog, so a rule can catch a container flooding the
uplink.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On 2026-10-18 an update on VDNT-Docker02 restarted the Docker daemon six
seconds in, before the drain that came only with the reboot, and every
container on the node stopped. Docker's own packages restart the daemon from
their install scripts; needrestart's list-only mode doesn't stop that.
- The helper gains a `check` action (agent 1.24.0): a dry run of the update
for each package manager, whether Docker is among it, and the running
containers outside the swarm with their restart policies.
- A swarm node is drained before an update that upgrades Docker, else only
before a reboot, and at most once a job.
- After a drain the job settles 5 minutes before the update or reboot.
- Once the work is over, the containers that ran outside the swarm must be
seen running again within 10 minutes, or the job fails naming them.
- An older agent gets no check and its job runs as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new plan's last_slot_at holds the moment it was made, so its first
slot is the next one; the tool returned that moment, a Wednesday
evening for a Sunday plan, beside latest_run: null. cadence.fired_slot
returns it only when it falls on one of the plan's slots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each source in an anomaly's `around` was cut to its oldest 40 with
nothing in the reply to say so; on a busy host the events nearest the
occurrence were the ones dropped. Each source now keeps the 40 nearest
the occurrence's start, and `around_left_out` counts the rest. Sources
return everything their bounded query read (Docker and the host agent
no longer cut at 40 and 20 themselves), so the count is honest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_anomaly gives each occurrence against what is usual at that hour, the
metric around the newest one with its normal, what else happened on the
resource's host around each occurrence (alert transitions, Ansible runs,
other anomalies there and a count elsewhere, plus plugins' own events
through a new mcp_events hook: the host agent's maintenance jobs and
downtime, Docker's container starts, deaths and OOM kills), and how earlier
anomalies on the same metric were judged.
get_metric_normal reads the learned slots and the level that would be
flagged. review_anomaly takes one id or several, all or nothing, with a
note. get_anomaly_settings and (admin) set_anomaly_settings read and change
the review window and each normal's floor, as Settings → Thresholds does.
metric_series.points takes an optional `until`; core.alerts gains
host_of_resource; floor_setting names its floor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5834 "The host agent is always on, and the MCP sees a host's
maintenance", step 1 of milestone 520 "The MCP sees what Steward has
grown".
The host agent is a core system (operator, 2026-10-15: "it's very core
to the functionality of Steward as a whole and shouldn't be considered
something that can be toggled off"). ALWAYS_ON_PLUGINS in core settings
makes to_plugins_cfg report it enabled whatever its stored row says, so
it always loads and its MCP tools are always listed. Settings -> Plugins
shows it as "Built in: always on" with no switch, and a save keeps it
enabled. Its code and URLs stay under plugins/host_agent: every
installed agent posts to /plugins/host_agent/ingest.
Maintenance through the MCP:
- list_maintenance_plans: each plan's group, weekly slot, reboot policy,
next slot, and its latest run (hosts, how far it got, who it skipped
and why, how it ended).
- get_maintenance_job(host | job): a host's jobs, newest and running
first, or one job: state, packages updated, reboot, drained by, why it
failed, when its current state times out.
- run_maintenance(host, reboot="if_required" | "never"): admin, as the
host page's Update now; refuses a busy host or one with no helper;
audited "<user> via MCP".
- The overview names a host whose latest job in the last week failed or
timed out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of milestone 514 "Updates and reboots on a cadence, carried out by
each host's agent…": #5765 "The agent reports pending updates and whether
a reboot is required".
- Agent 1.16.0: an hourly background check (apt-check, apt list, or dnf -C
with updateinfo and needs-restarting), each command bounded at 120 s, no
root and no network. Debian's reboot-required marker and its packages
are read every sample.
- Ingest records updates_pending, updates_security and reboot_required for
rules, and keeps the newest on the registration (host_agent_005_updates).
- The Agent card leads with "Reboot required since …", then the updates
waiting or "Up to date", with when the check ran. list_agents carries
them too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MCP-5 of milestone 461 "MCP server: Steward as an LLM-inspectable,
LLM-operable hub". Contributed through each plugin's register_mcp_tools
hook, so they exist only while the plugin is enabled.
docker (viewer, read-only):
- list_containers: filter by host, status, compose project, or problem
(restarting, crash_looping, unhealthy, oom_killed, exited_nonzero);
problems sort first and each row names what is wrong.
- get_container: state, CPU/memory history across the raw/hourly seam,
lifecycle events. A unique part of a swarm task name is enough.
- list_swarm_services: replicas per node (agent-less placements as ghosts)
and the swarm's nodes.
- search_container_logs: text (% and _ literal), stream, range, context
lines, paging by a full-precision `before` cursor; over-long lines keep
both ends and say how much of the middle was cut.
- get_docker_disk_usage: bytes per host, reclaimable first.
- mcp_overview: failing containers and under-replicated swarm services
join get_overview.
host_agent: list_agents (reporting / stale / pending, version, OS), and an
overview finding for agents registered over 30 minutes ago that never
checked in.
Settings > Thresholds gains the TLS-expiry warn/crit days (were 14/3 in
code) and the crash-loop deaths-per-hour count (3), which get_overview
and the docker tools now read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bvandeusenandClaude Opus 5.5