Read the config file as caldav 3.6 does

CHANGELOG.md

@@ -19,6 +19,8 @@ and I do try to adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.h
 * `completed!` in `set-task-attribs` moves a recurring task on to its next recurrence, like `edit --complete` does, instead of completing the whole series.
 * Shell tab completion scripts (bash, zsh, fish) now ship with the package.  They work without shell configuration for system-wide installs (and for bash user/venv installs with bash-completion 2.12+); see the README for other setups.  The `make install-completion` targets are gone.
 * icalendar 6.2.0 or newer is required.
+* caldav 3.6 or newer is required.  The config file is now read as caldav 3.6 reads it, so it may also use `kind: caldav` sections with plain keys, `password_command` (a program printing the password) and `password_keyring`.
+* The config file is found on the search path shared with the caldav library and other tools: `~/.config/pycal/calendar.yaml` first, then caldav's paths, `~/.config/calendar.conf` and `/etc`, or the file `CALENDARING_CONFIG_FILE` / `CALDAV_CONFIG_FILE` names.  Before, plann read `~/.config/calendar.conf` only, so with several of these files a different one may be used now.  `plann configure` creates `~/.config/pycal/calendar.yaml` when there is none, or when the file it found cannot be written (one under `/etc`), and makes the file readable by its owner only.
 
 ### Fixed
 

README.md

@@ -85,7 +85,7 @@ not be up-to-date and may contain features not implemented yet.
 
 * --caldav-url, --caldav-user, --caldav-pass: how to connect to the CalDAV server.  (Fits better into a configuration file)
 * --calendar-url: url to the calendar one wants to use.  A relative URL (path) or a calendar-id is also accepted.
-* --config-file: use a specific configuration file (default: $HOME/.config/calendar.conf)
+* --config-file: use a specific configuration file (default: the first that exists on the search path shared with the caldav library, see below)
 * --config-section: use a specific section from the config file (i.e. to select a different caldav-server to connect to)
 * --icalendar: Write or read icalendar to/from stdout/stdin
 * --nocaldav: don't connect to a caldav server
@@ -137,7 +137,7 @@ The file TASK_MANAGEMENT.md contains some thoughts on how to organize tasks.
 
 ## Configuration file
 
-Configuration file is by default located in $HOME/.config/calendar.conf.  `plann` supports both json and yaml.
+`plann` shares its configuration file with the caldav library and other calendar tools.  Unless `--config-file` names one, it uses the first that exists of `$XDG_CONFIG_HOME/pycal/calendar.yaml` (by default `~/.config/pycal/calendar.yaml`), `~/.config/caldav/calendar.conf`, `~/.config/caldav/calendar.yaml`, `~/.config/caldav/calendar.json`, `~/.config/calendar.conf`, `/etc/calendar.conf` and `/etc/caldav/calendar.conf`; `CALENDARING_CONFIG_FILE` or `CALDAV_CONFIG_FILE` may name another.  `plann configure` creates `~/.config/pycal/calendar.yaml` when none exists.  `plann` supports both json and yaml.
 
 (I considered .ini, but I was told that it's actually not a standard.  I'd like any calendar application to be able to access the file, hence calendar.conf and not plann.conf)
 
@@ -165,7 +165,16 @@ The connection settings (`caldav_url`, `caldav_user`, `caldav_pass`, `caldav_pro
 }
 ```
 
-Only the connection settings are expanded; other keys, such as `calendar_url` and `calendar_name`, are used as written.
+Instead of a password in the file or the environment, `caldav_password_command` may name a program (as a list of the program and its arguments) whose output is the password, and `caldav_password_keyring` a service name to look up in the system keyring:
+
+```yaml
+default:
+  caldav_url: http://foo.bar.example.com/caldav/
+  caldav_user: luser
+  caldav_password_command: [pass, show, dav/work]
+```
+
+The command is only run when the config file is writable by nobody but its owner (`chmod 600`); `plann configure` writes the file with your umask, so check it after configuring.  A section may also be written as `kind: caldav` with plain keys (`url`, `username`, `password_command`, ...).  The file format is shared with the caldav library and other tools, and is specified in [calendaring-config](https://github.com/pycalendar/calendaring-config/blob/main/docs/format.md).
 
 A `features` key may be given to enable server-specific compatibility workarounds, referring to a server profile in the caldav library's `compatibility_hints` module (e.g. `"features": "ecloud"`).  For servers with a known profile, `caldav_url` may even be omitted - the caldav library will derive the URL from the profile.
 

USER_GUIDE.md

@@ -48,7 +48,7 @@ It's recommended to rather use a config file.  Those options can be used for spe
 * `--config-file`
 * `--config-section`
 
-The default is to utilize the `default` section under `$HOME/.config/calendar.conf`  Here is an example configuration file:
+The default is to utilize the `default` section of the configuration file, found on the search path shared with the caldav library (`~/.config/pycal/calendar.yaml` first; see the README).  Here is an example configuration file:
 
 ```yaml
 ---

examples/copy-calendar.md

@@ -28,7 +28,7 @@ Take the events from the work calendar and mark up in your default calendar that
 
 ## Sub-task: take out all data in raw ical from the calendar
 
-`plann` without any options will select the default calendar from `.config/calendar.conf`.
+`plann` without any options will select the default calendar from the configuration file (`~/.config/pycal/calendar.yaml` or another file on the shared search path, see the README).
 
 `plann select` without any options will select everything.
 

plann/cli.py

@@ -39,7 +39,14 @@
     _split_high_pri_tasks,
     _split_huge_tasks,
 )
-from plann.config import config_section, expand_config_section, interactive_config, read_config
+from plann.config import (
+    config_section,
+    expand_config_section,
+    interactive_config,
+    read_config,
+    resolve_config_file,
+    writable_config_file,
+)
 from plann.interactive import _abort
 from plann.lib import (
     COMMA_LIST_ATTRS,
@@ -115,7 +122,7 @@ def extend(self, more):
 @click.option('--show-native-timezone/--show-local-timezone', help="Show timestamps as they are in the calendar (default is to convert to local timezone)")
 @click.option('--implicit-timezone', help="Timestamps entered without timezone is assumed to be in this timezone (default: local/system tz)")
 @click.option('--store-timezone', help="Timestamps saved to the calendar should be converted to this timezone (default: same as --implicit-timezone)")
-@click.option('-c', '--config-file', default=f"{os.environ.get('HOME')}/.config/calendar.conf")
+@click.option('-c', '--config-file', help="Configuration file (default: the first that exists of ~/.config/pycal/calendar.yaml, ~/.config/caldav/calendar.conf and the other paths shared with the caldav library)")
 @click.option('--skip-config/--read-config', help="Skip reading the config file")
 @click.option('--config-section', default=["default"], multiple=True)
 @click.option('--caldav-url', help="Full URL to the caldav server", metavar='URL')
@@ -158,25 +165,37 @@ def _discover_calendars(kwargs):
     --skip-config is given, from every selected config-file section."""
     calendars = find_calendars(kwargs, kwargs['raise_errors'])
     if not kwargs['skip_config']:
-        config = read_config(kwargs['config_file'])
+        config_file = _config_file(kwargs['config_file'])
+        config = read_config(config_file)
         if config:
             for meta_section in kwargs['config_section']:
-                calendars.extend(_config_calendars(config, meta_section, kwargs['raise_errors']))
+                calendars.extend(_config_calendars(config, meta_section, kwargs['raise_errors'], config_file))
     return calendars
 
-def _config_calendars(config, meta_section, raise_errors):
+def _config_file(given):
+    """The config file to read, looked up only when it is needed, so that
+    --help and --skip-config work whatever CALDAV_CONFIG_FILE says.  caldav
+    checks this file before running a password_command from it."""
+    try:
+        return resolve_config_file(given)
+    except ValueError as e:  ## CALDAV_CONFIG_FILE naming a file that is not there
+        raise click.UsageError(str(e)) from e
+
+def _config_calendars(config, meta_section, raise_errors, config_file):
     calendars = []
     for section in expand_config_section(config, meta_section):
-        calendars.extend(find_calendars(config_section(config, section), raise_errors=raise_errors))
+        calendars.extend(find_calendars(config_section(config, section), raise_errors=raise_errors,
+                                        config_file=config_file, section_name=section))
     return calendars
 
 def _section_calendars(ctx, meta_section):
     """Find the calendars of one config-file section, independent of the
     sections selected with --config-section.  Errors are raised."""
-    config = read_config(ctx.obj['config_file'])
+    config_file = _config_file(ctx.obj['config_file'])
+    config = read_config(config_file)
     if not config:
         return []
-    return _config_calendars(config, meta_section, raise_errors=True)
+    return _config_calendars(config, meta_section, raise_errors=True, config_file=config_file)
 
 @cli.command()
 @click.pass_context
@@ -201,11 +220,11 @@ def configure(ctx):
 
     Prompts for connection parameters and writes them to the config file.
     """
-    config_file = ctx.obj['config_file']
+    config_file = _config_file(ctx.obj['config_file'])
     sections = ctx.obj['config_section']
     section = sections[0] if sections else 'default'
     config = read_config(config_file, interactive_error=True) or {}
-    interactive_config(config, config_file, config_section=section)
+    interactive_config(config, writable_config_file(ctx.obj['config_file'], config_file), config_section=section)
 
 def _set_attr_options_(func, verb, desc=""):
     """

plann/config.py

@@ -12,6 +12,8 @@
 from caldav.config import expand_config_section as expand_config_section
 from caldav.config import read_config as _read_config
 
+import calendaring_config
+
 ## Connection parameters are written caldav_-prefixed; caldav's reader
 ## (extract_conn_params_from_section) maps caldav_user/caldav_pass to
 ## username/password.  The list is hand-written, and tests/test_config.py
@@ -132,10 +134,7 @@ def interactive_config(config, config_file, config_section='default', allow_use=
                         del config[section]
                     section = new_section
                 try:
-                    if os.path.isfile(config_file):
-                        os.rename(config_file, f"{config_file}.{int(time.time())}.bak")
-                    with open(config_file, 'w') as outfile:
-                        json.dump(config, outfile, indent=4)
+                    write_config_file(config, config_file)
                 except Exception as e:
                     print(e)
                 else:
@@ -146,6 +145,41 @@ def interactive_config(config, config_file, config_section='default', allow_use=
         config['default'] = config[section]
     return config
 
+def write_config_file(config, config_file):
+    """Write `config` to `config_file`, keeping a backup of the old one.  The
+    file may hold a password, so it is created readable by its owner only -
+    and caldav will only run a password_command from a file nobody else can
+    write to."""
+    if os.path.isfile(config_file):
+        os.rename(config_file, f"{config_file}.{int(time.time())}.bak")
+    os.makedirs(os.path.dirname(config_file) or '.', mode=0o700, exist_ok=True)
+    fd = os.open(config_file, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+    with os.fdopen(fd, 'w') as outfile:
+        json.dump(config, outfile, indent=4)
+
+def writable_config_file(given, found):
+    """Where `plann configure` saves: the file it read (`found`), unless
+    that was picked from the search path and cannot be written - such as
+    /etc/calendar.conf - in which case a new file goes first on the path."""
+    if given:
+        return found
+    ## saving renames the old file to a backup, so the directory must be writable
+    directory = os.path.dirname(found) or '.'
+    if not os.path.exists(directory) or os.access(directory, os.W_OK):
+        return found
+    return str(calendaring_config.search_path()[0])
+
+def resolve_config_file(config_file):
+    """The config file to use: the one given (with ~ expanded), else the
+    first that exists on the search path shared with the caldav library and
+    other tools (calendaring-config decides it, also from
+    CALENDARING_CONFIG_FILE / CALDAV_CONFIG_FILE), else the first path on
+    it, where `plann configure` creates a new one."""
+    if config_file:
+        return os.path.expanduser(config_file)
+    found = calendaring_config.find_config_file()
+    return str(found if found else calendaring_config.search_path()[0])
+
 def read_config(fn, interactive_error=False):
     """
     Thin wrapper around the caldav library's read_config.  The caldav

plann/lib.py

@@ -175,18 +175,29 @@ def _warn_orphaned_overrides(ical):
                 "master, otherwise they are stored as detached occurrence(s).",
                 uid, [str(x.get('SUMMARY', '(no summary)')) for x in components])
 
-def find_calendars(args, raise_errors):
+def find_calendars(args, raise_errors, config_file=None, section_name="section"):
     """
     Find calendars from a dict of connection parameters - typically a config
     file section or the command line arguments.  The connection keys are
-    caldav_-prefixed (caldav_url, caldav_user, caldav_pass, ...), optionally
-    accompanied by `features` and the calendar_url/calendar_name filters.
+    caldav_-prefixed (caldav_url, caldav_user, caldav_pass, ...), or plain
+    in a `kind: caldav` section, optionally accompanied by `features` and the
+    calendar_url/calendar_name filters.
 
     Connection parameter extraction (including resolving the URL from a
-    `features` server profile when no caldav_url is given) and the calendar
-    lookup itself are delegated to the caldav library.
+    `features` server profile when no caldav_url is given, and running a
+    password_command) and the calendar lookup itself are delegated to the
+    caldav library.  `config_file` is the file a section came from; caldav
+    checks it before running a password_command from it, and
+    `section_name` names the section in error messages.
     """
-    conn_params = extract_conn_params_from_section(args)
+    try:
+        conn_params = extract_conn_params_from_section(args, config_file, section_name)
+    except ValueError as e:
+        ## a password_command that failed, an unsafe config file, ...
+        if raise_errors:
+            raise
+        logging.error(f"Skipping a config section: {e}")
+        return []
     if not conn_params:
         return []
     calendars = caldav.get_calendars(

pyproject.toml

@@ -24,7 +24,8 @@ classifiers = [
 ]
 keywords = ["caldav", "calendar", "cli", "todo", "planning"]
 dependencies = [
-    "caldav>=3.3.0",
+    "caldav>=3.6.0",
+    "calendaring-config>=0.4",
     "Click",
     "PyYAML",
     "sortedcontainers",

tests/conftest.py

@@ -12,3 +12,19 @@ def _reset_tz():
     tz.__dict__.clear()
     yield
     tz.__dict__.clear()
+
+
+@pytest.fixture(autouse=True)
+def _no_real_config(tmp_path, monkeypatch):
+    """Keep the developer's own config files and CALDAV_*/CALENDARING_*
+    variables out of every test: plann looks the config file up on the
+    shared search path when no -c is given."""
+    import os
+
+    import calendaring_config.discovery
+    for name in list(os.environ):
+        if name.startswith(("CALDAV_", "CALENDARING_")):
+            monkeypatch.delenv(name)
+    monkeypatch.setenv("HOME", str(tmp_path))
+    monkeypatch.delenv("XDG_CONFIG_HOME", raising=False)
+    monkeypatch.setattr(calendaring_config.discovery, "SYSTEM_PATHS", (tmp_path / "etc.conf",))

tests/test_lib.py

@@ -74,6 +74,37 @@ def test_features_without_url(self):
             'features': self.features})
         assert calendars == [cal]
 
+    def test_password_command(self, tmp_path):
+        """caldav runs a password_command only with the config file it came
+        from, so plann has to pass that file along."""
+        import sys
+        config_file = tmp_path / "calendar.conf"
+        config_file.write_text("")
+        config_file.chmod(0o600)
+        section = {'caldav_url': 'https://calendar.example.com/dav',
+                   'caldav_password_command': [sys.executable, '-c', "print('s3cr3t')"]}
+        from plann.lib import find_calendars
+        with patch('caldav.get_calendars', return_value=[]) as get_calendars:
+            find_calendars(section, raise_errors=True, config_file=str(config_file))
+        assert get_calendars.call_args.kwargs['password'] == 's3cr3t'
+
+    def test_failing_password_command(self, tmp_path, caplog):
+        """A failing command is reported, and only stops discovery under
+        --raise-errors."""
+        import sys
+        import pytest
+        config_file = tmp_path / "calendar.conf"
+        config_file.write_text("")
+        config_file.chmod(0o600)
+        section = {'caldav_url': 'https://calendar.example.com/dav',
+                   'caldav_password_command': [sys.executable, '-c', 'raise SystemExit(3)']}
+        from plann.lib import find_calendars
+        assert find_calendars(section, raise_errors=False, config_file=str(config_file),
+                              section_name='work') == []
+        assert 'section work exited with status 3' in caplog.text
+        with pytest.raises(ValueError, match='status 3'):
+            find_calendars(section, raise_errors=True, config_file=str(config_file))
+
     def test_no_connection_params(self):
         calendars, cal = self._find_calendars({})
         assert calendars == []