Skip to content

Roadmap 2.1: the CalDAV backend - #13

Open
tobixen wants to merge 16 commits into
mainfrom
roadmap-2.1-caldav
Open

tobixen wants to merge 16 commits into
mainfrom
roadmap-2.1-caldav

Conversation

@tobixen

@tobixen tobixen commented Oct 20, 2026

Copy link
Copy Markdown
Member

Disclaimer: Code and the text below the line are AI-generated, by Claude Opus 5.5. The prompts are recorded in each commit message.


Implements roadmap item 2.1: a CalDAV backend over the caldav library, installed as the optional extra calendaring[caldav]. It passes the conformance suite in both modes against embedded Xandikos and Radicale.

Requires caldav 3.6, which is not released yet. CI fails until it is on PyPI, and this PR waits for that release. 3.6 sends If-None-Match: * on save(no_overwrite=True) (python-caldav/caldav#802), and keeps and sends raw bytes (raw_data, python-caldav/caldav#763). Tested locally against caldav master.

  • The capability table comes from caldav's server hints (API_DESIGN §5.1). On a server caldav has no hints for, what the server must store is UNKNOWN.
  • changes() uses the server's sync-collection report where caldav's hints say it is reliable, and falls back to an ETag listing everywhere else. changes(known=...) makes one PROPFIND.
  • Event searches are narrowed on the server where the server matches recurrences correctly. Task and journal searches are filtered client-side.
  • Writes go through caldav's save(). item.wire is the server's bytes, and wire= is sent unchanged. A weak ETag is compared against a fresh read.
  • The sync shim is generated from the async one, like the rest of the sync code.
  • Some API-level changes came along: Backend.connect() is renamed open(); deleting one occurrence adds an EXDATE (Q12); deleting a missing object is NotFoundError on every backend; test_* keys in config sections are reserved for test tooling; verify=True now checks alarms.

Follow-ups filed against caldav: python-caldav/caldav#799 (typing). Two more are to be drafted: AuthorizationError should carry the HTTP status, so that a 401 can be told apart from a 403; and a UID conflict found by caldav's own search should raise a distinct exception.

🤖 Generated with Claude Code

RFC 5545 requires DTSTART on a VEVENT outside a scheduling message, and
Radicale refuses one without it (HTTP 400).  The suite made events from
a summary alone, so a strict server failed tests that were not about
validity.  minimal() builds the smallest valid item of each kind.

Prompt: read the tmp-handover* document and the ROADMAP and continue working on this project [tmp-handover*: an untracked session-handover note]
Followup-Prompt: (...) then proceed with 2.1. [roadmap item 2.1, the CalDAV backend]

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
The backend never calls caldav directly.  _caldav_io.py holds AsyncDAV
and its sync twin DAV, which resolve caldav's "X | Coroutine" returns
once each, so a missing await in the generated code is a type error.
Writes are raw PUT/DELETE with If-None-Match or If-Match, since
caldav's save() sends no If-None-Match and bumps SEQUENCE.  The
capability table comes from caldav's server hints, mapped as API_DESIGN
section 5.1 says; for a server caldav has no hints for, what the server
must store is UNKNOWN.  Search, changes and known-changes are still
emulated client-side.  The conformance suite passes in both modes
against embedded Xandikos and Radicale.

Prompt: read the tmp-handover* document and the ROADMAP and continue working on this project [tmp-handover*: an untracked session-handover note]
Followup-Prompt: (...) then proceed with 2.1. [roadmap item 2.1, the CalDAV backend]

Prompts (comments made inline in the document while reviewing it):
> (on the review finding that the capability table claimed FULL for server-dependent features on every server, option 1: UNKNOWN when unhinted) we'll go for #1

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
changes(known=...) now lists ETags with one Depth: 1 PROPFIND instead
of downloading every object, so changes.known is FULL.  changes(token)
uses the server's sync-collection report (RFC 6578) when caldav's hints
say sync-token and its deletions work and an unknown token is refused,
not answered with a full listing (Stalwart, OX).  A truncated report,
or one without a new token, falls back to the etag listing on the
first call and is a SyncTokenError later, as is an expired token.  A
known= token stays on the etag listing.

Prompt: read the tmp-handover* document and the ROADMAP and continue working on this project [tmp-handover*: an untracked session-handover note]
Followup-Prompt: (...) then proceed with 2.1. [roadmap item 2.1, the CalDAV backend]

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
A search now asks the server only for the component kinds it can
match.  Event searches also pass their time range, but only where
caldav's hints say the server handles open ranges and matches
recurring events by their occurrences.  Task and journal searches
fetch everything, since servers disagree most on RFC 4791's time-range
rules for VTODO, and an under-match loses data.  The generic layer
still filters every result, and new unit tests pin the hint mapping.
The range is not widened here: caldav already widens it for CalDAV's
UTC-only ranges, and a code comment says so.

Prompt: read the tmp-handover* document and the ROADMAP and continue working on this project [tmp-handover*: an untracked session-handover note]
Followup-Prompt: (...) then proceed with 2.1. [roadmap item 2.1, the CalDAV backend]

Prompts (comments made inline in the document while reviewing it):
> (on the review finding that a server-side time range could under-match floating, all-day or naive bounds) Performance-wise this seems silly - we'll end up asking the caldav server for +/- 2 days, filter it in the caldav library to be correct +/- 1 day, and then filter once more to make it correct.  "The +/- 1 day and then filter"-logic in the caldav libary (sic) is there due to a design flaw in the caldav protocol (search range has to be in UTC-time)  I think we should trust the backend logic to do the right thing.  (This should be recorded somewhere, perhaps as code comments in the calendaring code)

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
verify=True compared unfolded content lines of the whole component, so
an alarm's DESCRIPTION counted as its event's.  A dropped or changed
VALARM went unnoticed unless it happened to share a property name with
the event.  Subcomponents are now compared on their own, in any order.
CATEGORIES compare as a set however a server splits them, and on a
single event an aware date-time stored in UTC instead of with its TZID
compares as the same instant.  In a recurring series it does not, since
the occurrences would shift at a DST change, and a floating time
converted to UTC is a difference too.  As at the top level, only what
was sent is compared: properties or alarms a server adds are no
difference.

Prompt: read the tmp-handover* document and the ROADMAP and continue working on this project [tmp-handover*: an untracked session-handover note]
Followup-Prompt: (...) then proceed with 2.1. [roadmap item 2.1, the CalDAV backend]

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
caldav pulls in niquests and lxml, which a user of the files or tracker
backends does not need, so it is the extra calendaring[caldav].
Without it, kind="caldav" is a ConfigurationError naming the pip
command.  An http(s) URL without kind= now means CalDAV, until roadmap
2.3's feed-or-CalDAV probe replaces that.  The dev extra installs
caldav and the embedded Xandikos and Radicale servers, so CI runs the
CalDAV conformance suite.

Prompts (comments made inline in the document while reviewing it):
> (on the 2.1 plan, caldav as an optional extra or a hard dependency) We'll go for the soft dependency

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
The shim's sync class DAV was a hand-written copy of AsyncDAV.  AsyncDAV
now lives in _async/dav.py, and the generator writes DAV from it like
the rest of the sync code, with _awaited renamed to _returned and
AsyncDAVClient to DAVClient.  What both modes share stays in
_caldav_io.py.  The parity test now guards the generated copy.

Prompts (comments made inline in the document while reviewing it):
> (on the 2.1 plan, the hand-written caldav shim) This involves a lot of hand-written mapping code, I presume?  Any alternatives?
> (on the answer proposing to generate the sync half) go for it.

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
A stored object may hold overrides without their master (RFC 4791
section 4.1, e.g. an attendee invited to some occurrences only).
get(uid) returned whichever the stored object listed first.  It now
returns the one with the earliest RECURRENCE-ID, and API_DESIGN says
that get() then misses the others, which search() returns.

Prompts (comments made inline in the document while reviewing it):
> (on the 2.1 plan, what get(uid) returns for overrides stored without a master) This problem arises because an Item is defined to either be a full recurrence set including the master, or a single occurrence?
> (on the answer: earliest RECURRENCE-ID, or AmbiguousError) Let's give the first one, and document that this may be a problem.

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
The shared config file carries keys only test tooling reads, such as
caldav's test-server selection.  Workspace.from_config() passed every
key to connect(), so such a section failed to load.  Keys starting
with test_ are now reserved for test tooling and dropped.  Any other
unknown key is still a ConfigurationError, so typos are caught.
API_DESIGN section 2.2 records the decision, which was left open
for the CalDAV backend.

Prompts (comments made inline in the document while reviewing it):
> (on the 2.1 plan, which config keys a backend ignores) Are there other use-cases except for giving instructions to the test framework?  Perhaps we could have some standard like all attributes exclusively read by the test framework should be prepended with `test_` and all of them ignored?
> (on the answer recommending a reserved test_ prefix) We'll go for "`test_*`".  The caldav code needs to be made both forward- and backward compatible eventually.  We don't need to care about backward compatibility in the calendaring package (I may be wrong, but I believe there are very few except for me that have configured test servers in the config file).

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
delete() on an occurrence raised UnsupportedError while API_DESIGN Q12
was open.  It now adds an EXDATE to the master in the form of its
DTSTART, and removes the occurrence's override.  The known client bugs
with EXDATE came from mismatched forms.  A stale etag from a sibling's
edit is no conflict: only a change to this occurrence raises
ConflictError.  Q12 records the decision and why not STATUS:CANCELLED,
and the conformance suite tests it on every backend that recurs.

Prompts (comments made inline in the document while reviewing it):
> (on the 2.1 plan, delete() on an occurrence: EXDATE or STATUS:CANCELLED) I vaguely remember that there was some reason for not using EXDATE, like lack of client support.  Is there no arguments for/against EXDATE in the issue discussions?  Please do some research on client support
> (on the research answer) We'll go for EXDATE then.  Please draft up a comment to be posted to the 598-issue.

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
News fragments for the CalDAV backend, deleting one occurrence and the
reserved test_ keys.  CONTRIBUTING says that make dev installs the
embedded test servers, and how to wrap a dual-mode library such as
caldav.  SYNC_ASYNC_ARCHITECTURE marks the caldav bridge settled, and
ROADMAP 2.1 is ticked, linking the caldav issues it filed.

Prompt: read the tmp-handover* document and the ROADMAP and continue working on this project [tmp-handover*: an untracked session-handover note]
Followup-Prompt: (...) then proceed with 2.1. [roadmap item 2.1, the CalDAV backend]

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
For CalDAV, files and feeds the factory does no network I/O: caldav's
DAVClient() only builds an object, and opening a file connects to
nothing, so "connect" over-promised.  It stays a classmethod and a
coroutine in async mode, since JMAP fetches a session resource and a
tracker may check its token.  The private hook _connect becomes _open.
Nothing is released, so there is no deprecated alias.

Prompts (comments made inline in the document while reviewing it):
> (on the review finding that connect(token=...) raised a bare TypeError) Should we even have a `.connect`-method in the calendaring library?  The `get_davclient()` and `DAVClient`-constructor in the caldav library does not (unless it has been changed recently) connect to any server, it just builds up an object that can be used for communicating.  When opening a file there is also no "connect" going on.

Prompt: Another subagent can work on renaming connect to open. [= rename Backend.connect() in calendaring to Backend.open()]

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
The backend wrote with raw PUT and DELETE, because caldav's save() sent
no If-None-Match, and read caldav's private response body to recognise
a UID conflict.  caldav 3.6 sends If-None-Match: * for
save(no_overwrite=True), so creates, updates and deletes now go through
caldav's own objects and exceptions, and the raw-write helpers are gone.
save() re-serialises what it sends, so write.wire is UNSUPPORTED on
CalDAV until python-caldav/caldav#763 is
fixed.  A weak ETag, which If-Match cannot match, is compared with a
fresh read instead, and deleting a missing object is NotFoundError on
every backend.  The caldav extra now needs caldav 3.6, not released
yet.

Prompt: I will close the tmp-push-review-gate.md file - because when this fix is in caldav, we should get rid of the _body-logic in calendaring_client.  The rename from connect to open should also go in through the same pull request.
Followup-Prompt: (answer to a clarifying question: how the 2.1 PR gets the unreleased caldav) Wait for caldav 3.6
Followup-Prompt: (answer to a clarifying question: keep a raw PUT for wire= only?) Drop raw PUT entirely, and caldav #763 will land soon [python-caldav/caldav#763]
Followup-Prompt: meanwhile, rewrite the calendaring writes against the caldav branch

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
caldav 3.6 keeps the server's bytes as raw_data and sends given bytes
unchanged with save(raw_data=...)
(python-caldav/caldav#763).  item.wire is
now raw_data rather than caldav's normalised copy, wire= goes out
byte for byte, and write.wire is FULL on CalDAV.  Xandikos refolds
what it stores, so the new test checks the PUT body.

Prompt: Continue working here.  Check the handover-document.  caldav 3.6 may not be released today or tomorrow, but do testing locally with the local caldav version [the handover's next item: use caldav's raw-bytes API for wire, as promised on PyCalendar/calendaring-sync#19]

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
Roadmap 1.6 asked for a review from plann's author before the
aggregation design freezes.  What matters is how the design affects
plann, whoever reviews it, so the item says that.

Prompt: fix those two too [two leftover items: the test_ prefix in calendaring-config, already done, and this stashed ROADMAP 1.6 edit]

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
caldav 3.6.0 gives every DAVError a status and preconditions, and a
RateLimitError its retry_after_seconds, so _translate reads them
directly instead of probing with getattr.  A UID that caldav's own
search finds now raises UidConflictConsistencyError, which creating()
already reports as taken="uid".  Tests pin both: 401 vs 403
(python-caldav/caldav#805) and the UID clash
(python-caldav/caldav#806).

Prompt: IIRC, we had some PRs here that was waiting for the caldav 3.6.0-release?  It has been released now
Followup-Prompt: followup PR13 [pick up the caldav 3.6 fixes for issues 805 and 806 in PR 13's error translation]

Assisted-By: Claude Opus 5.5 (1M context)
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant