feat(host_agent): Steward hands a host a maintenance job and follows it through the reboot (#5767)
#5767 "Steward hands a host a maintenance job and follows it to the end, through its own restarts", milestone 514. A host_maintenance_jobs row per job: queued → updating → [draining →] rebooting → [undraining →] done, or failed / timed_out. Steward's answer to an ingest post carries the step that host's agent should carry out now; the agent (1.18.0) leaves each step id for the root helper once, and the helper refuses an id it already ran, so repeats are harmless. Results come back as the #5766 maintenance events, matched by step id; the reboot completes when the host's `started` event has a boot_at after the reboot was handed out. The job reboots only when the update says one is needed (or never, if asked). A swarm node is drained first by itself if it's a manager or by a manager with the helper, and made active after; with no manager to drain it the job is refused. A minute tick times out a job past its state's deadline. A maintenance window over host_agent/<host> lasts the rest of the job. Everything is on the row, so a Steward restart, or the reboot of the host Steward runs on, resumes from the next post. Ingest runs it in its own savepoint so a job going wrong never costs the samples. The Agent panel leads with a running job and says what a finished one did; Manage agent has Update now (with "never reboot"), or says why a job can't start. Integration test: hand off, Steward restarts, the post-boot events close the job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -73,3 +73,36 @@ class HostDowntime(Base):
|
||||
started_at = Column(DateTime(timezone=True), nullable=False)
|
||||
ended_at = Column(DateTime(timezone=True), nullable=True)
|
||||
created_at = Column(DateTime(timezone=True), nullable=False, default=_utcnow)
|
||||
|
||||
|
||||
class HostMaintenanceJob(Base):
|
||||
"""One host's update, and its reboot when the update needs one (#5767).
|
||||
|
||||
Steward keeps the job here and the host's agent carries out each step
|
||||
through its root helper; maintenance.py has the states and transitions.
|
||||
step_id is the helper request id of the current step, and step_seen says a
|
||||
result for it has arrived, so it stops being handed out. node_id and
|
||||
drain_host_id are set when the host is a swarm node: the node to drain and
|
||||
the manager whose agent drains it.
|
||||
"""
|
||||
__tablename__ = "host_maintenance_jobs"
|
||||
|
||||
id = Column(String(36), primary_key=True, default=_uuid)
|
||||
host_id = Column(String(36), ForeignKey("hosts.id", ondelete="CASCADE"),
|
||||
nullable=False, index=True)
|
||||
state = Column(String(16), nullable=False, default="queued", index=True)
|
||||
state_at = Column(DateTime(timezone=True), nullable=True)
|
||||
reboot_policy = Column(String(16), nullable=False, default="if_required")
|
||||
step_id = Column(String(64), nullable=True, index=True)
|
||||
step_seen = Column(Boolean, nullable=False, default=False)
|
||||
node_id = Column(String(64), nullable=True)
|
||||
drain_host_id = Column(String(36), ForeignKey("hosts.id", ondelete="SET NULL"), nullable=True)
|
||||
packages = Column(Integer, nullable=True)
|
||||
reboot_needed = Column(Boolean, nullable=True)
|
||||
back_at = Column(DateTime(timezone=True), nullable=True)
|
||||
detail = Column(Text, nullable=True)
|
||||
window_id = Column(String(36), ForeignKey("maintenance_windows.id", ondelete="SET NULL"),
|
||||
nullable=True)
|
||||
created_by = Column(String(36), ForeignKey("users.id", ondelete="SET NULL"), nullable=True)
|
||||
created_at = Column(DateTime(timezone=True), nullable=False, default=_utcnow)
|
||||
finished_at = Column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
Reference in New Issue
Block a user