fix(ansible): the swarm prune never runs a docker prune that touches images; it lives in its own folder (#6399)
The first applied run lost VDNT-Docker03 29 images (10.4 GB) before its script ran. Its `docker system prune` removes dangling images, and Docker counts an image with no tag as dangling: every image a swarm task pulled by digest. Docker03 was left holding only its own tasks' images, as under the old prune. Nothing went down, since a running container's image can't be removed. - Step 1 now prunes containers, networks, build cache and (prune_volumes) volumes one by one. Every image decision is left to the script, which keeps the images services are pinned to. - A test guards against any image-pruning command in the playbook. - The playbook moves to maintenance/swarm_prune/, with its script in that folder's files/, so a helper's name can't collide with another playbook's (the operator's direction for playbooks with helpers). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+1
-1
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""Steward's swarm prune (maintenance/swarm_prune.yml), run on each node.
|
"""Steward's swarm prune (maintenance/swarm_prune/swarm_prune.yml), run on each node.
|
||||||
|
|
||||||
Removes the images on this node that no swarm service is pinned to and no
|
Removes the images on this node that no swarm service is pinned to and no
|
||||||
container uses (by id, or by the reference it was made from), and keeps every image a service is pinned to even when its
|
container uses (by id, or by the reference it was made from), and keeps every image a service is pinned to even when its
|
||||||
+19
-11
@@ -4,10 +4,11 @@
|
|||||||
# steward:confirm: true
|
# steward:confirm: true
|
||||||
# Run on the swarm's nodes; each must be a manager (only a manager can list
|
# Run on the swarm's nodes; each must be a manager (only a manager can list
|
||||||
# services). On each node:
|
# services). On each node:
|
||||||
# 1. docker system prune: stopped containers, unused networks, dangling
|
# 1. Stopped containers, unused networks and build cache are pruned
|
||||||
# images, build cache (+ prune_volumes=true: unused anonymous volumes).
|
# (+ prune_volumes=true: unused anonymous volumes). No image is: Docker
|
||||||
# 2. files/swarm_prune.py: removes every other image no container uses and
|
# calls an untagged image dangling, and swarm pulls by digest, untagged.
|
||||||
# no swarm service is pinned to, wherever that service's task runs. A
|
# 2. files/swarm_prune.py: removes every image no container uses and no
|
||||||
|
# swarm service is pinned to, wherever that service's task runs. A
|
||||||
# node keeps the images for the services running elsewhere, so it can
|
# node keeps the images for the services running elsewhere, so it can
|
||||||
# start them at once when the other node fails or is drained.
|
# start them at once when the other node fails or is drained.
|
||||||
# A node that can't list the swarm's services removes no images and the run
|
# A node that can't list the swarm's services removes no images and the run
|
||||||
@@ -19,13 +20,19 @@
|
|||||||
vars:
|
vars:
|
||||||
prune_volumes: false
|
prune_volumes: false
|
||||||
tasks:
|
tasks:
|
||||||
- name: Prune stopped containers, unused networks, dangling images and build cache
|
# Never `docker system prune` or `docker image prune`: Docker counts an
|
||||||
|
# image with no tag as dangling, and every image a swarm task pulled by
|
||||||
|
# digest has none. Images are left to the script below.
|
||||||
|
- name: Prune stopped containers, unused networks and build cache
|
||||||
ansible.builtin.command:
|
ansible.builtin.command:
|
||||||
argv: >-
|
argv: "{{ item }}"
|
||||||
{{ ['docker', 'system', 'prune', '-f']
|
loop: >-
|
||||||
+ (['--volumes'] if prune_volumes | bool else []) }}
|
{{ [['docker', 'container', 'prune', '-f'],
|
||||||
register: system_prune
|
['docker', 'network', 'prune', '-f'],
|
||||||
changed_when: "'Total reclaimed space: 0B' not in system_prune.stdout"
|
['docker', 'builder', 'prune', '-f']]
|
||||||
|
+ ([['docker', 'volume', 'prune', '-f']] if prune_volumes | bool else []) }}
|
||||||
|
register: cleanup
|
||||||
|
changed_when: "'Total reclaimed space: 0B' not in cleanup.stdout and cleanup.stdout | trim | length > 0"
|
||||||
|
|
||||||
- name: Remove images no container uses and no swarm service is pinned to
|
- name: Remove images no container uses and no swarm service is pinned to
|
||||||
ansible.builtin.script:
|
ansible.builtin.script:
|
||||||
@@ -39,5 +46,6 @@
|
|||||||
- name: Report what went
|
- name: Report what went
|
||||||
ansible.builtin.debug:
|
ansible.builtin.debug:
|
||||||
msg: >-
|
msg: >-
|
||||||
{{ (system_prune.stdout_lines | default([]) | select | list)[-1:]
|
{{ (cleanup.results | default([]) | map(attribute='stdout', default='')
|
||||||
|
| select('search', 'reclaimed') | list)
|
||||||
+ [image_prune.stdout | from_json] }}
|
+ [image_prune.stdout | from_json] }}
|
||||||
@@ -16,7 +16,7 @@ BUILTIN = Path(get_sources({"sources": []})[0]["path"])
|
|||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
def prune():
|
def prune():
|
||||||
spec = importlib.util.spec_from_file_location(
|
spec = importlib.util.spec_from_file_location(
|
||||||
"swarm_prune", BUILTIN / "maintenance" / "files" / "swarm_prune.py")
|
"swarm_prune", BUILTIN / "maintenance" / "swarm_prune" / "files" / "swarm_prune.py")
|
||||||
mod = importlib.util.module_from_spec(spec)
|
mod = importlib.util.module_from_spec(spec)
|
||||||
spec.loader.exec_module(mod)
|
spec.loader.exec_module(mod)
|
||||||
return mod
|
return mod
|
||||||
@@ -27,13 +27,19 @@ def _image(id_, tags=(), digests=(), size=100):
|
|||||||
|
|
||||||
|
|
||||||
def test_the_playbook_is_listed_and_asks_before_it_runs():
|
def test_the_playbook_is_listed_and_asks_before_it_runs():
|
||||||
assert "maintenance/swarm_prune.yml" in discover_playbooks(str(BUILTIN))
|
assert "maintenance/swarm_prune/swarm_prune.yml" in discover_playbooks(str(BUILTIN))
|
||||||
content = (BUILTIN / "maintenance" / "swarm_prune.yml").read_text()
|
content = (BUILTIN / "maintenance" / "swarm_prune" / "swarm_prune.yml").read_text()
|
||||||
meta = discover_playbook_meta(content)
|
meta = discover_playbook_meta(content)
|
||||||
assert meta["confirm"] is True and meta["category"] == "maintenance" and meta["description"]
|
assert meta["confirm"] is True and meta["category"] == "maintenance" and meta["description"]
|
||||||
play = yaml.safe_load(content)[0]
|
play = yaml.safe_load(content)[0]
|
||||||
script = next(t for t in play["tasks"] if "ansible.builtin.script" in t)
|
script = next(t for t in play["tasks"] if "ansible.builtin.script" in t)
|
||||||
assert script["check_mode"] is False and "--dry-run" in script["ansible.builtin.script"]["cmd"]
|
assert script["check_mode"] is False and "--dry-run" in script["ansible.builtin.script"]["cmd"]
|
||||||
|
# No docker prune touches images: an image swarm pulled by digest has no
|
||||||
|
# tag, and Docker calls that dangling (on 2026-10-25 a `system prune`
|
||||||
|
# took 29 a node was keeping for the other's services).
|
||||||
|
commands = yaml.safe_dump([t for t in play["tasks"] if "ansible.builtin.command" in t])
|
||||||
|
assert "'system'" not in commands and "'image'" not in commands
|
||||||
|
assert "'container', 'prune'" in commands and "'builder', 'prune'" in commands
|
||||||
|
|
||||||
|
|
||||||
def test_services_are_matched_by_digest_or_else_by_tag(prune):
|
def test_services_are_matched_by_digest_or_else_by_tag(prune):
|
||||||
|
|||||||
Reference in New Issue
Block a user