bvandeusen/FabledSteward
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(docker): the move of task-named history rests between passes (#6718)
CI / lint (push) Successful in 2s
Details
CI / unit (push) Successful in 1m34s
Details
CI / integration (push) Successful in 1m49s
Details
CI / publish (push) Successful in 20s
Details

Browse Source 
#6718 "A swarm container's metrics, normal and anomalies are keyed by its task
name, so they start over whenever the task moves". Live, the move took
VDNT-Docker02's I/O pressure (psi_io_full_avg60) from about 1% to about 23%,
and passes of 5000 raw samples outran their 15 s slot. Each moved sample
rewrites its row and every index on it.

Batches are 1000 samples, and after a pass the task waits four times as long
as it took, so it holds the disk for about a fifth of the time and the move
slows if the disk does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
bvandeusen
2026-10-30 20:01:10 -04:00
co-authored by Claude Opus 5.5
parent 4689386b69
commit acc487f9c3
1 changed files with 8 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plugins/docker/rekey.py
+8 -3
View File
@@ -32,9 +32,10 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
BATCH = 5000 # raw samples moved in one go BATCH = 1000 # raw samples moved in one go: each rewrites its row and every index on it
EVERY_S = 15 EVERY_S = 15
PASS_S = 2 # a key with only a few rows takes milliseconds, and there can be many PASS_S = 2 # a key with only a few rows takes milliseconds, and there can be many
REST = 4 # after a pass, wait this many times as long as it took, so ingest keeps the disk
TRAFFIC = ("net_rx_bps", "net_tx_bps") TRAFFIC = ("net_rx_bps", "net_tx_bps")
_TASK_KEY = r"^[^/]+/.+\.([0-9]+|[a-z0-9]{25})\.[a-z0-9]{25}$" _TASK_KEY = r"^[^/]+/.+\.([0-9]+|[a-z0-9]{25})\.[a-z0-9]{25}$"
_NOT_TRAFFIC = "metric_name NOT IN ('net_rx_bps', 'net_tx_bps')" _NOT_TRAFFIC = "metric_name NOT IN ('net_rx_bps', 'net_tx_bps')"
@@ -167,10 +168,12 @@ async def move_key(session, old: str) -> bool:
def make_rekey_task(app: "Quart") -> ScheduledTask: def make_rekey_task(app: "Quart") -> ScheduledTask:
remaining: list[str] | None = None remaining: list[str] | None = None
busy = asyncio.Lock() busy = asyncio.Lock()
resume_at = 0.0
async def rekey() -> None: async def rekey() -> None:
nonlocal remaining nonlocal remaining, resume_at
if remaining == [] or busy.locked(): started = time.monotonic()
if remaining == [] or busy.locked() or started < resume_at:
return return
async with busy, app.db_sessionmaker() as session: async with busy, app.db_sessionmaker() as session:
if remaining is None: if remaining is None:
@@ -185,5 +188,7 @@ def make_rekey_task(app: "Quart") -> ScheduledTask:
remaining.pop() remaining.pop()
if not remaining: if not remaining:
logger.info("Swarm history under task names: all moved") logger.info("Swarm history under task names: all moved")
now = time.monotonic()
resume_at = now + REST * (now - started)
return ScheduledTask(name="docker_rekey", coro_factory=rekey, interval_seconds=EVERY_S) return ScheduledTask(name="docker_rekey", coro_factory=rekey, interval_seconds=EVERY_S)