bvandeusen/FabledScribe
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 50 Wiki Activity

feat(events): replace end_dt column with duration_minutes (#160)

Browse Source 
Structural fix for the "end before start" bug class observed on prod
2026-04-29. Bad data became inexpressible at the schema level instead
of getting trapped in defensive read-path filters.

The hotfix that landed earlier today (94b169f) is reverted by the
preceding revert commit; this commit supersedes it cleanly with a
proper data-model change.

## Schema (migration 0043)

- Add `duration_minutes INTEGER NULLABLE` column on `events`.
- CHECK constraint: ``duration_minutes IS NULL OR duration_minutes >= 0``.
- Backfill from existing `end_dt`:
    - end_dt valid (end > start) → duration_minutes = total minutes
    - end_dt == start → duration_minutes = 0 (zero-duration point)
    - end_dt NULL or end_dt < start → duration_minutes = NULL
      (the corrupt prod row collapses cleanly to a point event)
- Drop the `end_dt` column. The wire format is preserved — `to_dict()`
  emits `end_dt` as a derived `start_dt + duration_minutes`. Existing
  API consumers (Flutter app, web frontend, CalDAV sync) keep
  receiving the same response shape; they just no longer have a way
  to PUT a stored `end_dt` that disagrees with `start_dt`.

## Service layer

- `Event.end_dt` becomes a `@property`. Setting it would require a
  setter we deliberately don't define — writes always go through
  `duration_minutes`.
- `_normalize_duration` is the single source-of-truth for input
  reduction. Accepts (start, end_dt, duration_minutes), returns the
  canonical `duration_minutes`, raises `ValueError` for negative
  durations, end-before-start, or end/duration disagreement.
- `create_event` and `update_event` accept either `end_dt` or
  `duration_minutes` for ergonomic compat; both convert via
  `_normalize_duration`. Update validates the post-update state when
  the patch includes either.
- `list_events` filter is simpler now: a coarse SQL prefilter
  (`start_dt <= date_to`) plus Python-side refinement using the
  derived `end_dt`. Avoids Postgres-specific interval arithmetic in
  the WHERE clause; refinement runs over a per-user result set so
  there's no scan-cost concern at personal scale.
- Recurring-event expansion uses `event.duration_minutes` directly
  instead of computing `end - start`. No more negative-timedelta
  hazard.

## CalDAV sync (incoming + outgoing)

- `caldav_sync.py` (pull) and `calendar_sync.py` (Radicale upsert)
  both convert iCal `DTEND` → `duration_minutes` on the way in.
  Outbound iCal still emits `DTEND` as `start_dt + duration_minutes`
  via the model's derived property. iCal interop is unchanged.

## Behavioral upgrade for `update_event`

Pure end_dt model: moving start past the existing end_dt would either
silently corrupt or hard-reject. Duration model: the duration is
preserved by default, so moving start slides the effective end
forward — which is what users mean when they "move" an event.
Explicit clear is still possible via `end_dt=None`.

## Tests

`tests/test_events_service.py`:
- 6 new `_normalize_duration` unit tests (sugar conversion, zero
  duration valid as point event, end-before-start rejected, negative
  duration rejected, inconsistent end+duration rejected, none → None)
- New behavioral test: `update_event` preserves duration when only
  start_dt changes (sliding semantics)
- New: clearing `end_dt=None` on update collapses to point event
- New: list_events surfaces a point event in the upcoming window
- New: list_events excludes a timed event whose effective end has
  already passed
- Existing mock-event helper updated to use `duration_minutes`
  instead of stored `end_dt`.

44 event-related tests pass; ruff clean.

## Out of scope (separate task)

Fable #161 — `find_events_by_query` returning multiple matches and
silently picking matches[0]. The exact root cause of how event id=2
got mutated in the first place; orthogonal to the storage model.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Bryan Van Deusen
2026-04-29 14:19:44 -04:00
parent b7e7073425
commit 4c58603009
7 changed files with 517 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/fabledassistant/models/event.py
+22 -3
View File
@@ -1,4 +1,4 @@
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, Text
from sqlalchemy.orm import Mapped, mapped_column
@@ -20,7 +20,11 @@ class Event(Base):
uid: Mapped[str] = mapped_column(Text)
title: Mapped[str] = mapped_column(Text, default="")
start_dt: Mapped[datetime] = mapped_column(DateTime(timezone=True))
end_dt: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
# Duration in minutes; NULL = point event with no end specified.
# Replaces the prior `end_dt` column (Fable #160 / migration 0043).
# The DB has a CHECK constraint that this is NULL or >= 0, so an
# event whose end is before its start is structurally inexpressible.
duration_minutes: Mapped[int | None] = mapped_column(Integer, nullable=True)
all_day: Mapped[bool] = mapped_column(Boolean, default=False)
description: Mapped[str] = mapped_column(Text, default="")
location: Mapped[str] = mapped_column(Text, default="")
@@ -38,7 +42,21 @@ class Event(Base):
onupdate=lambda: datetime.now(timezone.utc),
)
@property
def end_dt(self) -> datetime | None:
"""Derived end datetime: ``start_dt + duration_minutes``.
Returns ``None`` for point events (``duration_minutes is None``).
Computed at access time rather than stored — a stored end was
the source of the "end before start" corruption that motivated
this redesign.
"""
if self.duration_minutes is None:
return None
return self.start_dt + timedelta(minutes=self.duration_minutes)
def to_dict(self) -> dict:
end_dt = self.end_dt
return {
"id": self.id,
"user_id": self.user_id,
@@ -47,7 +65,8 @@ class Event(Base):
"project_id": self.project_id,
"title": self.title,
"start_dt": self.start_dt.isoformat() if self.start_dt else None,
"end_dt": self.end_dt.isoformat() if self.end_dt else None,
"end_dt": end_dt.isoformat() if end_dt else None,
"duration_minutes": self.duration_minutes,
"all_day": self.all_day,
"description": self.description,
"location": self.location,
src/fabledassistant/routes/events.py
+26 -19
View File
@@ -54,19 +54,23 @@ async def create_event():
end_dt = _parse_dt(data["end_dt"]) if data.get("end_dt") else None
except ValueError:
return jsonify({"error": "Invalid datetime format"}), 400
event = await events_svc.create_event(
user_id=_get_current_user_id(),
title=data["title"],
start_dt=start_dt,
end_dt=end_dt,
all_day=data.get("all_day", False),
description=data.get("description", ""),
location=data.get("location", ""),
color=data.get("color", ""),
recurrence=data.get("recurrence"),
project_id=data.get("project_id"),
reminder_minutes=data.get("reminder_minutes"),
)
try:
event = await events_svc.create_event(
user_id=_get_current_user_id(),
title=data["title"],
start_dt=start_dt,
end_dt=end_dt,
duration_minutes=data.get("duration_minutes"),
all_day=data.get("all_day", False),
description=data.get("description", ""),
location=data.get("location", ""),
color=data.get("color", ""),
recurrence=data.get("recurrence"),
project_id=data.get("project_id"),
reminder_minutes=data.get("reminder_minutes"),
)
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
return jsonify(event.to_dict()), 201
@@ -93,7 +97,7 @@ async def update_event(event_id: int):
for bool_field in ("all_day",):
if bool_field in data:
fields[bool_field] = data[bool_field]
for int_field in ("project_id", "reminder_minutes"):
for int_field in ("project_id", "reminder_minutes", "duration_minutes"):
if int_field in data:
fields[int_field] = data[int_field]
for dt_field in ("start_dt", "end_dt"):
@@ -106,11 +110,14 @@ async def update_event(event_id: int):
fields[dt_field] = _parse_dt(data[dt_field])
except ValueError:
return jsonify({"error": f"Invalid datetime for {dt_field}"}), 400
event = await events_svc.update_event(
user_id=_get_current_user_id(),
event_id=event_id,
**fields,
)
try:
event = await events_svc.update_event(
user_id=_get_current_user_id(),
event_id=event_id,
**fields,
)
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
if event is None:
return jsonify({"error": "Event not found"}), 404
return jsonify(event.to_dict())
src/fabledassistant/services/caldav_sync.py
+14 -3
View File
@@ -130,6 +130,17 @@ async def sync_user_events(user_id: int) -> dict:
async with async_session() as session:
for ev in remote_events:
caldav_uid = ev["caldav_uid"]
# Storage uses duration, not end_dt. Convert here so the
# rest of this function can compare/upsert in one shape.
ev_start = ev["start_dt"]
ev_end = ev["end_dt"]
ev_duration = (
int((ev_end - ev_start).total_seconds() // 60)
if ev_end is not None and ev_start is not None and ev_end > ev_start
else None
)
ev["duration_minutes"] = ev_duration
result = await session.execute(
select(Event).where(
Event.user_id == user_id,
@@ -145,8 +156,8 @@ async def sync_user_events(user_id: int) -> dict:
uid=str(uuid.uuid4()),
caldav_uid=caldav_uid,
title=ev["title"],
start_dt=ev["start_dt"],
end_dt=ev["end_dt"],
start_dt=ev_start,
duration_minutes=ev_duration,
all_day=ev["all_day"],
description=ev["description"],
location=ev["location"],
@@ -157,7 +168,7 @@ async def sync_user_events(user_id: int) -> dict:
else:
# Update if anything changed
changed = False
for field in ("title", "start_dt", "end_dt", "all_day", "description", "location", "recurrence"):
for field in ("title", "start_dt", "duration_minutes", "all_day", "description", "location", "recurrence"):
if getattr(existing, field) != ev[field]:
setattr(existing, field, ev[field])
changed = True
src/fabledassistant/services/calendar_sync.py
+10 -2
View File
@@ -207,6 +207,14 @@ async def sync_event_to_db(user_id: int, ical_uid: str) -> Event | None:
d = dtend.dt
end_dt = datetime(d.year, d.month, d.day, tzinfo=timezone.utc)
# Storage uses duration, not end_dt. Convert iCal DTEND to a
# minute count anchored on DTSTART. Treat invalid (end <= start)
# incoming data as a point event rather than rejecting; we
# don't control external CalDAV writers.
duration_minutes = None
if end_dt is not None and end_dt > start_dt:
duration_minutes = int((end_dt - start_dt).total_seconds() // 60)
async with async_session() as session:
result = await session.execute(
select(Event).where(Event.user_id == user_id, Event.uid == ical_uid)
@@ -215,7 +223,7 @@ async def sync_event_to_db(user_id: int, ical_uid: str) -> Event | None:
if existing:
existing.title = title
existing.start_dt = start_dt
existing.end_dt = end_dt
existing.duration_minutes = duration_minutes
existing.all_day = all_day
existing.description = description
existing.location = location
@@ -230,7 +238,7 @@ async def sync_event_to_db(user_id: int, ical_uid: str) -> Event | None:
uid=ical_uid,
title=title,
start_dt=start_dt,
end_dt=end_dt,
duration_minutes=duration_minutes,
all_day=all_day,
description=description,
location=location,
src/fabledassistant/services/events.py
+171 -59
View File
@@ -1,4 +1,13 @@
"""Internal event store service with CalDAV push sync."""
"""Internal event store service with CalDAV push sync.
Storage model: an event is anchored at ``start_dt`` and has an optional
``duration_minutes``. The end of the event is *derived* via
``Event.end_dt`` (a Python property), never stored. Callers may still
pass ``end_dt`` on writes for ergonomic compatibility — the service
converts to ``duration_minutes`` internally. This rules out the entire
"end before start" bug class structurally (Fable #160 / migration
0043). Open-ended events use ``duration_minutes = None``.
"""
from __future__ import annotations
import asyncio
@@ -7,7 +16,7 @@ import uuid
from datetime import datetime, timedelta, timezone
from dateutil.rrule import rrulestr
from sqlalchemy import and_, or_, select
from sqlalchemy import or_, select
from fabledassistant.models import async_session
from fabledassistant.models.event import Event
@@ -15,11 +24,56 @@ from fabledassistant.models.event import Event
logger = logging.getLogger(__name__)
def _normalize_duration(
*,
start_dt: datetime,
end_dt: datetime | None,
duration_minutes: int | None,
) -> int | None:
"""Reduce (end_dt, duration_minutes) inputs to a single canonical
``duration_minutes`` value.
Resolution order:
1. If ``duration_minutes`` is explicit, use it (validate >= 0).
If ``end_dt`` is also given, validate the two agree.
2. Otherwise, derive from ``end_dt - start_dt``.
3. Otherwise None (point event with no end).
Raises ``ValueError`` for any invalid combination — duration < 0,
end_dt < start_dt, or end_dt and duration_minutes inconsistent.
"""
if duration_minutes is not None:
if duration_minutes < 0:
raise ValueError(
f"duration_minutes must be >= 0, got {duration_minutes}"
)
if end_dt is not None:
expected = int((end_dt - start_dt).total_seconds() // 60)
if expected != duration_minutes:
raise ValueError(
f"end_dt ({end_dt.isoformat()}) implies "
f"{expected} minutes but duration_minutes={duration_minutes} "
f"was passed; pass only one or make them agree."
)
return duration_minutes
if end_dt is not None:
delta_seconds = (end_dt - start_dt).total_seconds()
if delta_seconds < 0:
raise ValueError(
f"end_dt ({end_dt.isoformat()}) must be at or after "
f"start_dt ({start_dt.isoformat()}); pass end_dt=None "
f"or omit it for point events."
)
return int(delta_seconds // 60)
return None
async def create_event(
user_id: int,
title: str,
start_dt: datetime,
end_dt: datetime | None = None,
duration_minutes: int | None = None,
all_day: bool = False,
description: str = "",
location: str = "",
@@ -27,12 +81,25 @@ async def create_event(
recurrence: str | None = None,
project_id: int | None = None,
reminder_minutes: int | None = None,
# CalDAV-only fields (not stored in DB, forwarded to push)
# ``duration`` is a legacy alias kept for the calendar tool layer
# and CalDAV pass-through callers; promotes to duration_minutes
# when duration_minutes isn't otherwise specified.
duration: int | None = None,
attendees: list[str] | None = None,
calendar_name: str | None = None,
) -> Event:
"""Create an event in the DB, then fire a CalDAV push task."""
"""Create an event in the DB, then fire a CalDAV push task.
Either ``end_dt`` or ``duration_minutes`` may be supplied; the
service converts to ``duration_minutes`` internally. Raises
``ValueError`` on invalid combinations (negative duration, end
before start, end/duration disagreement).
"""
if duration is not None and duration_minutes is None:
duration_minutes = duration
duration_minutes = _normalize_duration(
start_dt=start_dt, end_dt=end_dt, duration_minutes=duration_minutes,
)
uid = str(uuid.uuid4())
async with async_session() as session:
event = Event(
@@ -40,7 +107,7 @@ async def create_event(
uid=uid,
title=title,
start_dt=start_dt,
end_dt=end_dt,
duration_minutes=duration_minutes,
all_day=all_day,
description=description,
location=location,
@@ -54,7 +121,7 @@ async def create_event(
await session.refresh(event)
extra_fields = {
"duration": duration,
"duration": duration_minutes,
"reminder_minutes": reminder_minutes,
"attendees": attendees,
"calendar_name": calendar_name,
@@ -80,66 +147,74 @@ async def list_events(
"""List events for user_id that overlap [date_from, date_to].
Recurring events (with an RRULE recurrence string) are expanded into
individual occurrences within the range. Non-recurring events are
returned as-is. All results are sorted by start time and returned as
dicts (same shape as Event.to_dict()).
individual occurrences within the range. Non-recurring events are
returned as-is. All results are sorted by start time and returned as
dicts (same shape as ``Event.to_dict()``).
Filtering strategy: a coarse SQL prefilter (events that start on or
before ``date_to``), then refine in Python using the event's derived
end (``start_dt + duration_minutes``). Doing the end-of-event math
in SQL would require Postgres-specific interval arithmetic; the
Python-side refinement is a few row-loops over a small per-user
result set, which is fine for personal-scale data and avoids
coupling the query to a specific dialect.
"""
async with async_session() as session:
# Match strategy:
# - Recurring events: fetch all, expand via rrule below.
# - Non-recurring with an end_dt: standard overlap — starts before
# date_to and ends after date_from.
# - Non-recurring with no end_dt: treat as a point event at
# start_dt, include only if start_dt falls within the window.
# (Previously this branch matched any event with a null end_dt,
# returning all past events as "happening today".)
result = await session.execute(
select(Event).where(
select(Event)
.where(
Event.user_id == user_id,
or_(
Event.recurrence.isnot(None),
and_(
Event.recurrence.is_(None),
Event.start_dt <= date_to,
or_(
Event.end_dt >= date_from,
and_(
Event.end_dt.is_(None),
Event.start_dt >= date_from,
),
),
),
Event.start_dt <= date_to,
),
).order_by(Event.start_dt)
)
.order_by(Event.start_dt)
)
events = list(result.scalars().all())
items: list[dict] = []
for event in events:
if not event.recurrence:
items.append(event.to_dict())
if event.recurrence:
duration = (
timedelta(minutes=event.duration_minutes)
if event.duration_minutes is not None
else None
)
try:
rule = rrulestr(event.recurrence, dtstart=event.start_dt, ignoretz=False)
occurrences = rule.between(date_from, date_to, inc=True)
except Exception:
logger.warning("Failed to expand RRULE for event %d: %r", event.id, event.recurrence)
# Fall back to canonical event row; still apply the
# window check so a far-future canonical row doesn't
# leak into today's list.
if date_from <= event.start_dt <= date_to:
items.append(event.to_dict())
continue
base = event.to_dict()
for occ in occurrences:
if occ.tzinfo is None:
occ = occ.replace(tzinfo=timezone.utc)
occurrence_dict = dict(base)
occurrence_dict["start_dt"] = occ.isoformat()
if duration is not None:
occurrence_dict["end_dt"] = (occ + duration).isoformat()
items.append(occurrence_dict)
continue
# Expand recurring event occurrences within [date_from, date_to]
duration = (event.end_dt - event.start_dt) if event.end_dt else None
try:
rule = rrulestr(event.recurrence, dtstart=event.start_dt, ignoretz=False)
occurrences = rule.between(date_from, date_to, inc=True)
except Exception:
logger.warning("Failed to expand RRULE for event %d: %r", event.id, event.recurrence)
items.append(event.to_dict())
continue
base = event.to_dict()
for occ in occurrences:
# Ensure occurrence is UTC-aware
if occ.tzinfo is None:
occ = occ.replace(tzinfo=timezone.utc)
occurrence_dict = dict(base)
occurrence_dict["start_dt"] = occ.isoformat()
if duration is not None:
occurrence_dict["end_dt"] = (occ + duration).isoformat()
items.append(occurrence_dict)
# Non-recurring: refine the coarse prefilter in Python using the
# derived end_dt. A point event (duration None) is included when
# its start is at or after date_from. A timed event is included
# when its end is at or after date_from.
derived_end = event.end_dt
if derived_end is None:
if event.start_dt >= date_from:
items.append(event.to_dict())
else:
if derived_end >= date_from:
items.append(event.to_dict())
items.sort(key=lambda x: x["start_dt"])
return items
@@ -173,7 +248,13 @@ async def search_events(
async def update_event(user_id: int, event_id: int, **fields) -> Event | None:
"""Partial update. Returns updated event or None if not found."""
"""Partial update. Returns updated event or None if not found.
Accepts ``end_dt`` or ``duration_minutes`` (or both, validated for
agreement). The service converts to ``duration_minutes`` before
persisting; ``end_dt`` is never stored. Raises ``ValueError`` for
invalid combinations against the post-update state.
"""
async with async_session() as session:
result = await session.execute(
select(Event).where(Event.id == event_id, Event.user_id == user_id)
@@ -182,10 +263,39 @@ async def update_event(user_id: int, event_id: int, **fields) -> Event | None:
if event is None:
return None
old_title = event.title # capture before mutation for CalDAV lookup
allowed = {"title", "start_dt", "end_dt", "all_day", "description",
"location", "color", "recurrence", "project_id", "reminder_minutes"}
# Nullable fields that callers can explicitly set to None to clear
nullable = {"end_dt", "recurrence", "project_id", "reminder_minutes"}
# Resolve any end_dt/duration_minutes inputs against the
# post-update start_dt. If neither is in the patch, leave the
# existing duration_minutes alone.
post_update_start = (
fields["start_dt"]
if fields.get("start_dt") is not None
else event.start_dt
)
if "end_dt" in fields or "duration_minutes" in fields:
new_end = fields.pop("end_dt", None)
new_duration = fields.pop("duration_minutes", None)
# If end_dt is in the patch but explicitly None, that's a
# clear → duration_minutes = None. Same shape duration_minutes=None.
if new_end is None and new_duration is None:
fields["duration_minutes"] = None
else:
fields["duration_minutes"] = _normalize_duration(
start_dt=post_update_start,
end_dt=new_end,
duration_minutes=new_duration,
)
allowed = {
"title", "start_dt", "duration_minutes", "all_day",
"description", "location", "color", "recurrence",
"project_id", "reminder_minutes",
}
# Nullable fields callers can explicitly clear by passing None
nullable = {
"duration_minutes", "recurrence", "project_id",
"reminder_minutes",
}
for key, value in fields.items():
if key in allowed and (value is not None or key in nullable):
setattr(event, key, value)
@@ -255,11 +365,12 @@ async def _push_create(event: Event, user_id: int, extra: dict) -> None:
)
if not await is_caldav_configured(user_id):
return
derived_end = event.end_dt # property: start + duration_minutes
await caldav_create(
user_id=user_id,
title=event.title,
start=event.start_dt.isoformat(),
end=event.end_dt.isoformat() if event.end_dt else None,
end=derived_end.isoformat() if derived_end else None,
description=event.description or None,
location=event.location or None,
all_day=event.all_day,
@@ -296,12 +407,13 @@ async def _push_update(event: Event, user_id: int, old_title: str = "") -> None:
return
# Use old_title so CalDAV can find the event even if the title was changed
query_title = old_title or event.title
derived_end = event.end_dt
await caldav_update(
user_id=user_id,
query=query_title,
title=event.title,
start=event.start_dt.isoformat(),
end=event.end_dt.isoformat() if event.end_dt else None,
end=derived_end.isoformat() if derived_end else None,
description=event.description or None,
location=event.location or None,
)