Pimalaya <> PyCal

[soywod]

Hi,
I'm Clément, the owner of the Pimalaya project. We met at NLnet meeting (@niccokunzmann).

Himalaya is our main project, targeting mails.
We included contacts few months ago to the project, and developed a vcard lib for Rust.
We recently started to include calendars, and we developed a ical lib for Rust. It's still in development, not as mature as vcard, but quite promising.
Since libs are written in Rust, it could export Python bindings.

It could be interested to see where to put common efforts :)
Have a great day!

[niccokunzmann]

Hi!

Nice to meet you again. We are in the process of writing the grant proposal.
I wonder where to allocate time for that.

Generally, we work with icalendar to be a really good library with documentation for developers. So, I do not know yet how to fit that in, if we can replace parts of that - I do not think so.

My suggestion is to explore creating a Rust Python package with your code and host it under github.com/pycalendar. This way, the Pythonistas will find it along with other projects in the eco system. I do not know how long it takes to do that but it can be worth the try.

What do you think about this?

What is interesting to cooperate on and actually beneficial for us both is this:
(1) python-recurring-ical-events does the rrule expansion and event calculations.
The problem we face: we use dateutil.rrule and a liable to people submitting malicious events with a lot of occurrences that could potentially DoS calendar clients who want to know e.g. what happens in a calendar on a specific day.
So, with NLnet, we will get a security review to see how that can be exploited.
Then, we might need to re-write the rrule/rdate/exdate expansion code to be DoS safe. I propose that this should go into another package that replaces dateutil.rrule easily. The calendar specific code lives in recurring-ical-events.

(2) since the last grant, I wanted to do this RRULE with different calendars. https://www.rfc-editor.org/info/rfc7529/ But it is a huge job, especially without anyone in the affected communities to review or drive a change. It appears Google Calendar supports HEBREW calendars.
This is where the cooperation can be useful again.

I am happy to help with publishing the Python binding of your calendar parser. I would learn something new there.
Do you do event time calculations in your code? In that case, we have a clear functional overlap: DoS proof rrule & event expansion.

What are your thoughts? Where do you see potential mutual benefit?

[niccokunzmann]

@SashankBhamidi For your interest in jmap, have a look at thier project. If you create a Python binding, this might be really good. https://github.com/pimalaya/io-jmap

[soywod]

I will better reply next week, just to tell that I'm also in the process of writing NLnet planing, it's definitely the right time to have a proper discussion. So far I like the idea of python bindings, and make rrule safer!

[niccokunzmann]

Nice! If you like, lets have a call this week to coordinate that. You can write me an email or join us on Matrix:
https://matrix.to/#/#pycal:chat.pycal.org

[niccokunzmann]

@WhyNotHugo you are also welcome to join our chat there. Let me know when you are in!

[stevepiercy]

I'm interested. I'm the documentation lead for icalendar and the Python Calendaring Ecosystem.

Other than making it possible for Rust to talk to Python, what would bindings do and what would be the benefits?

What would be any disadvantages?

[soywod]

Today we had a call with Nicco and Sashank, we seem to agree about exporting Python bindings from Pimalaya. Starting with a small scoped task like calendar's RRULE seems a great and safe direction.

Other than making it possible for Rust to talk to Python, what would bindings do and what would be the benefits? What would be any disadvantages?

I would say that the main benefits are to gather efforts than splitting them. Pimalaya has a iCalendar lib (for Rust), you also have one (for Python). Since it's a complex subject with a lot of edge cases, it's appealing to have one single source of truth. So the idea is to merge both projects Rust side, then expose a Python API via C-bindings.

Python side will benefit from Rust strong typing and memory safety, at the cost of being tied up to a lib (less portable). Unless the Rust lib is packaged with your Python API (static build?).

---

Here the list of potential candidates, sorted by scope-size (smaller first):

• At Pimalaya we have a solid vCard parser, validator, editor and builder. The lib is released, and started to be used in production. Still early. We also have a iCal lib, not yet in production, but taking the same direction as vCard. This is where we can definitely do sth. Starting by the RRULE is a good thing, we can extend over time to the parser, validator, editor, types, builder etc.

• We also have a WebDAV I/O-free lib, not sure where it can go, but sth to look at. Still early, but base solid and tested.

• Additionnally, we have a JMAP I/O-free lib, that handles mails, contacts and calendars. Both last domains are still experimental for now, but base solid as well.

• Finally, we heavily work on Pimdir, a standard for storing text-based PIM items like mails, contacts, calendars. The aim is to be able to store dedup items into a single base, mixing filesystem and sqlite, and with soft delete by design. See it as a Maildir (or Vdir) that scales a bit better and handles all PIM domains. We have a sync engine (in dev, don't use it yet) and a mail synchronizer that will turn as a generic PIM synchronizer as soon as the pimdir integration is over.

I may edit this post if other things come to my mind afterwards.

[WhyNotHugo]

Hi there all! I'm also working via NLNet, currently on pimsync, which is rather mature/stable at this point.

Pimsync does synchronisation, so it only partially parses iCalendar/vCard. Pimsync needs to handle elements which might be semantically invalid — if users can't fetch an invalid element from their server, they have no way of fixing it. So the underlying parser, vparser is much lower level: it basically tokenizes and returns content lines, splitting only name parameter and value. It is heavily tested and optimised though, and quite suitable for higher level blocks.

Regarding RRULEs, I maintain the Rust crate recurrence. The main limitation right now is properly supporting VTIMEZONE elements: we need to implement a TimeZone type which uses data parsed from a VTIMEZONE, and this requires further work on refactoring this library. recurrence it itself a fork on rust-rrule, with some work in this general direction (the original/upstream is kind of stagnant in terms of merging PRs). Substantial more work is still required. I'd love if we could better coordinate around this, since it seems that it's something blocking a lot of the ecosystem. Basically anything that needs to parse non-UTC RRULES in Rust is currently blocked on this. Exposing this to Python would require substantial glue, in order to convert chrono's datetime and timezones into Python objects and viceversa.

Pimsync also relies on libdav for WebDAV, CalDAV and CardDAV. The core of the library is sans-IO, and it also includes a glue layer to use the typical Rust stack: tokio/tower/hyper. The sans-IO approach is quite unlike io-webdav: libdav simply has types which represent a query (e.g.: list events in given time range), produce an http request and parse the http response into idiomatic types. It's a bit more of a "naïve" design, but likely quite easy to interface with other languages.

Pimsync stores filesystem storage in the same format as vdirsyncer: vdir. It's compatible with khal, todoman, and a few other tools from various developers. The storage format is merely a filesystem convention of standard iCalendar and vCard files. I've had interest in pushing this as an RFC, but I haven't heard much back form the IEEE on the subject.

BTF, @soywod, I see a huge overlap in the work we're doing, and we aligned a couple of years back on scopes. I'm curious why none of pimsync's/vstorage's stack worked out and why you felt the need to implement them differently. I'd love if we were able to converge at some point.

[WhyNotHugo]

Sorry for the somewhat long comment. Honestly, I couldn't find more bits to trim down 😅

[soywod]

Happy to see you here @WhyNotHugo. It's been a while :)

There is a lot to say, I will try to structure my answer the best I can.

BTF, @soywod, I see a huge overlap in the work we're doing, and we aligned a couple of years back on scopes. I'm curious why none of pimsync's/vstorage's stack worked out and why you felt the need to implement them differently. I'd love if we were able to converge at some point.

I forgot when was our last sync, but here what I remember:

• You tried to bring mails inside your sync engine and just did not fit, due to mutability and meta/flags. You told me that you abandonned the idea and decided to move forward. I think we discussed that I may try from my side when I reach contacts and calendars, which I did with Pimdir (I will detail a bit further).
• I was experimenting about the Sans I/O approach, and I had a bit of working stuff related to WebDAV. Your lib was tied up to async with tokio, and we agreed that if one day the io-webdav is ready from my side we may integrate it into your lib. The day is almost there, io-webdav is partially in production from my side.
• You were banging you head on timezone and rrule, which I understand why now since I stepped up into the domain.
• You were polishing vparser, which I remember was a low-level parser (more a tokenizer in fact), and we agree that I may use it if I plan to implement a higher vcard/ical parser.
• You were also working with Vdir as main standard used to store text-based contacts and calendars, close to what Maildir does for mails.

Please complete and/or correct me if needed.

---

The current state of Pimalaya is the following:

Sans I/O

The Sans I/O pattern is mature enough (I called it I/O-free, since it sits at higher level), and is used in production.

JMAP

We have io-jmap, which is a I/O-free JMAP client lib. It support mails, contacts and calendars for now.

Mail

Himalaya reached v2 (with I/O-free libs), the domain is considered stable at this point. I will not expand to much as it is out of PyCal scope.

Contact

The domain is considered complete and tested, but still beta for now.

We have vcard-rs, which is a vCard parser, validator, editor and builder. It's not based on vparser, which I will explain later why.

We have io-vdir, which is the I/O-free lib for our vdir standard.

We have io-webdav, which is a I/O-free lib for managing WebDAV-based protocols, including CardDAV and CalDAV for now. It's based on io-http (same core as io-jmap).

We have Cardamum, a CLI to manage contacts that reached v0.2 (with I/O-free libs).

Calendar

The domain is in development, unstable for now.

We have ical-rs, which is a iCalendar parser, validator, editor and builder. It's not based on vparser, which I will explain later why.

We have io-vdir, which is the I/O-free lib for our vdir standard.

We have io-webdav, which is a I/O-free lib for managing WebDAV-based protocols, which includes CardDAV and CalDAV for now. It's based on io-http (same core as io-jmap).

We have Calendula, a CLI to manage calendar components not yet released (with I/O-free libs).

Discovery

We have io-pim-discovery, which is a PIM discovery client library (and a CLI), used by all other apps to properly discover mail, contact and calendar configs.

Sync

We have Neverest, which was a mail synchronizer. Since mail domain did not fit pimsync, we agreed to keep both doing their own job. Neverest algorithm was simpler, due to immutability nature of mails, yet mutable envelope flags were not fitting your model.

I said was a mail synchronizer, because it's now a generic PIM synchronizer. Few words on that in the storage section below.

GUI

Now that we reached solid I/O-free level, working well within basic terminal applications, we plan to develop a Pimalaya GUI which would gather all the domains, all the libs into one single place. A Thunderbird-like application. This application reuse literally every single piece: protocol libs, discovery, sync for offline usage, but it misses one piece: the storage.

Mail has Maildir. Flags management is a pain, new standard like m2dir sounds better but few adoption for now.
Contacts and calendars have Vdir.
Both try to do the same thing: storing text-based contents using filesystem. They share both the same issues: it badly scales on large amount of items, it is poorly searchable (that's why most of power users install notmuch on the top of their Maildir to properly search), it behaves differently between OS (mostly due to windows special name casing and forbidden chars).

For a cross-platform GUI we needed something more reliable:

Storage

We initiated a standard called Pimdir. It's in heavy development at the moment, borders are not yet well defined, but:

• It aims to be a storage for portable offline usage. Client use it directly to build interfaces for users.
• It uses sqlite (portable, works the same on every OS) and filesystem (for storing blobs only).
• It stores text-based contents, so it works for mails, contacts, calendars, files etc.
• It can store only meta, or full content depending on the need; this unlocks partial sync, especially useful for mobile apps. Database can lazily be hydrated.
• It deduplicates items, which is really a must for mails. On mail can be in multiple mailboxes, which is impossible to achieve with pure IMAP. JMAP (and notmuch) already do the same.
• It soft deletes, so you never loose contents. Unused items are garbadge-collected with custom strategy.
• It merge and resolve conflicts with remotes, when it can (based on stategy)
• It enqueues actions: it means any client can modify safely the database (modifying a local flag on a mail, or a contact); it can even queue items for submission (mostly a mail feature).

Neverest is now using Pimdir directly instead of Maildir, which make it a generic PIM synchronizer.

---

Knowing that, I will try to reply point by point:

RRULE

I did not even know about the existence of your recurrence crate. It looks just perfect, and should be used by PyCal team to define C-bindings for Python. I will have a deeper look in the nearest days, because I will definitely be interested to use inside ical-rs.

My apologies, I should have subscribed to your blog and follow a bit more what you did. Although SourceHut does not help at all. Pimalaya used to be on SourceHut but we moved back to GitHub mostly for that reason: people were just lost inside the project, not knowing where to go. SourceHut was a nightmare experience for us. Yet I don't support GitHub's direction either, but not at the cost of visibility and well being of Pimalaya's users.

Anyway this is another topic, I just wanted to say that I'm sad to have missed your recurrence crate.

vparser

We did not use your lib mostly because it assumes UTF-8 strings. Why not raw bytes? There are some obscure areas that involve non UTF-8 data (thinking about CHARSET in vCard 2.1).

We also partially forgot about existence of your lib, for the same reason we did not find recurrence.

libdav

Pimsync also relies on libdav for WebDAV, CalDAV and CardDAV. The core of the library is sans-IO, and it also includes a glue layer to use the typical Rust stack: tokio/tower/hyper.

I did not know about libdav based on Sans I/O pattern. As I said earlier, in my memories it used not to be the case.

Let me dive into your lib and compare it with our actual io-webdav. My intuition is that io-webdav sits at even lower level that libdav, but I'm afraid they are incompatible (for now). io-webdav coroutines emit WantsRead and WantsWrite "events" (expects a raw stream to read/write), while libdav expects a "HTTP sendable" client. I'm pretty sure there is room for improvement.

libjmap

I first thought we overlap, but in fact we don't. io-jmap sits just below your libjmap. So I think it makes sense for libjmap to use io-jmap under the hood, and expose the same async interface.

disco

We overlap a bit on discovery (yours seem to be spread a bit everywhere, by domain, where Pimalaya gathers it in io-pim-discovery), I think it makes sense to also gather our efforts.

pimsync

This is were we probably overlap the most, but for a "good" reason. Our aim and vision are slightly different. You have the vision of contacts and calendars, at lower level, Pimalaya has more a globalist vision about PIM (and higher level). So the sync and storage we develop don't really match. Yet we must have plenty of small things to put together, we should definitely spend more time on it. I'm in the process of writing down the new planing for 2027, I can definitely allocate tasks for you if you are interested in.

[niccokunzmann]

Hi @WhyNotHugo Thanks for the comment! And hi @soywod,

This grant, I will put time in collaborating on having a joint rrule implementation in Python and rust. It would be cool if we can have a call next week to get to know each other and map that out a little.

Also, there seem to be some cool conventions we have in Python and that you have in Rust that are quite important to explore:
E.g. datetime and timezone is standardized in Python and we will need to look at how to cooperate there.
I am interested in the way you implement I/O-free libraries.

This grant, also I want to explore binding competition to icalendar into Python, the vCard library. It somewhat existst in icalendar but this allows us to explore both a bit more.

The recurrence rule stuff is super complex. The choice of whether to use timezones or not is a relatively small issue compared to how to make this work in general, especially across languages. So, I would like to have some exchange on that and map out a proper plan and entry points.

I think it is good to get to know each other and the projects first and then slide into a conversation of how to do things together.

[soywod]

@WhyNotHugo I read the sources of your recurrence crate, and as far as I understand it's quite "incompatible" with Pimalaya's ical-rs recurrence code. They seem to diverge too much at some critical paths, like for example:

• recurrence models dates as chrono::DateTime<Tz>, which forces a timezone resolution of DTSTART and at each expansion step. Pimalaya ical uses civil time (equivalent of chrono NaiveDateTime), and moves away timezone resolution, which in my understanding, makes the expansion way simpler and somehow closer to the RFC.
• Pimalaya ical core is dependency free and no-std compatible (thanks to the point above)
• There is some missing RFC implems from both sides

So I'm not sure how to genuinely proceed. I would not be against developing together a merged lib that we both agree on and that we can use independently from our both sides. And this lib could be used for the python bindings. But it's a lot of work.

I'm open to any kind of proposition.

[niccokunzmann]

We will have a call now. I will update you after this!

[niccokunzmann]

So I'm not sure how to genuinely proceed. I would not be against developing together a merged lib that we both agree on and that we can use independently from our both sides. And this lib could be used for the python bindings. But it's a lot of work.

Basically, I maintain recurring-ical-events. It has a lot of downloads. In a major release, if that lib is working, we can switch over from dateutil. Then, there is an instant massive amount of users that can give feedback.

[niccokunzmann]

We already have a grant approved and we can cooperate there on vCard and RRULE.

There is another grant in the making. I would appreciate if you two @soywod and @WhyNotHugo could name 2-3 packages that would benefit from having a Python binding. We would like to incorporate them. That is a good start. Soft deadline for this is basically tomorrow. Hard deadline on Monday morning.

Basically: Which other parts of software can be cut out and offered as a self-contained package with a small interface that we can bring into the Python community to improve collaboration and feedback across these ecosystems? As well as an estimation of how long it would take you to work on the refactoring. Let me know on Martix, please.

[soywod]

From my side I can propose https://github.com/pimalaya/vcard for vCard and https://github.com/pimalaya/ical for a RRULE extraction (this requires to sync with @WhyNotHugo). Are you interested in WebDAV and vdir libs? We also have at Pimalaya, but it requires sync with Hugo again.