Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 38 additions & 12 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@ jobs:
fail-fast: false
matrix:
python: ["3.9", "3.10", "3.11", "3.12", "3.13"]
os: ["ubuntu-latest"]
os: ["ubuntu-24.04"]
pytest_opts: ["--workers 4 --tests-per-worker 1"]
requirements: [""]
include:
- os: "ubuntu-latest"
- os: "ubuntu-24.04"
python: "3.9"
requirements: "requirements/minimum.txt"
- os: "macos-latest"
Expand All @@ -38,8 +38,8 @@ jobs:
# ignore doctests
pytest_opts: "-k pins/tests"
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v4
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: ${{ matrix.python }}
- name: Install dependencies
Expand Down Expand Up @@ -79,13 +79,39 @@ jobs:
# https://github.com/browsertron/pytest-parallel/issues/93
no_proxy: "*"

test-data-dict:
name: "Test data dictionary CLI"
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-24.04, macos-latest, windows-latest]
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e '.[data-dict]' pytest
- name: Run data dictionary tests and offline bundle example
shell: bash
run: |
pytest pins/tests/test_data_dict.py pins/tests/test_connect_preview.py pins/tests/test_data_dict_security.py
python examples/data-dict/create_bundle.py \
"$RUNNER_TEMP/pins-python-data-dict-bundle" \
--dictionary examples/data-dict/data-dict.yaml
env:
PINS_REQUIRE_DATA_DICT_CLI: "1"

test-connect:
name: "Test Posit Connect"
runs-on: ubuntu-latest
if: ${{ !github.event.pull_request.head.repo.fork }}
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v4
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.10"
- name: Install dependencies
Expand All @@ -112,8 +138,8 @@ jobs:
runs-on: ubuntu-latest
if: ${{ github.event.pull_request.head.repo.fork }}
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v4
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.10"
- name: Install dependencies
Expand All @@ -131,8 +157,8 @@ jobs:
name: "Build Docs"
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v4
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.10"
- name: Install dependencies
Expand Down Expand Up @@ -234,8 +260,8 @@ jobs:
if: github.event_name == 'release'
needs: [build-docs, tests]
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v4
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.10"
- name: "Build Package"
Expand Down
50 changes: 50 additions & 0 deletions examples/data-dict/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Deploy a data dictionary pin

This example prepares a pin bundle from synthetic data for Posit Connect.
The publisher runs the `data-dict` CLI while preparing the bundle and stores the rendered dictionary with the pin.
The combined entrypoint keeps the standard pin preview available alongside the dictionary view.
The `create_bundle.py` script uses `CONNECT_SERVER`, which defaults to `https://connect.example.com`, so it can build the bundle offline.
The rsconnect-python package publishes the bundle through the saved account named by `CONNECT_ACCOUNT`.

Run from the repository root:

```{.bash filename="Terminal"}
uv run --no-project --with-editable '.[data-dict]' python examples/data-dict/create_bundle.py /tmp/pins-python-data-dict-bundle --dictionary examples/data-dict/data-dict.yaml
uv tool run --from rsconnect-python rsconnect list
env -u CONNECT_SERVER uv tool run --from rsconnect-python rsconnect deploy manifest /tmp/pins-python-data-dict-bundle/manifest.json --name "$CONNECT_ACCOUNT" --new --title 'Pins Python data dictionary' --no-metadata
```

The output directory must be empty.
Set `CONNECT_ACCOUNT` to a saved account name before publishing, and register the account with `rsconnect login` if needed.
The deployment uses the server's default private content access.

To build a regular pin preview without a dictionary, run:

```{.bash filename="Terminal"}
uv run --no-project --with-editable . --with pyarrow python examples/data-dict/create_bundle.py /tmp/pins-python-preview-bundle --no-dictionary
```

Supply `--dictionary /path/to/data-dict.yaml` to attach an authored dictionary.
Omit `--dictionary` to generate a draft.
The dictionary must omit table `source` fields so rendering does not depend on files on the publisher's machine.
Attaching an authored dictionary to a folder board does not require the CLI.
Drafting and Connect rendering require the optional dependency or a `data-dict` executable on `PATH`.
For a pin with one Parquet file and one dictionary table, rendering profiles the pinned data and embeds column statistics in the dictionary view.
The stored dictionary omits source paths; the renderer adds the bundle-relative Parquet path to a temporary copy.
You can also supply YAML text in `metadata["data_dict"]`, using one dictionary input per call.

For an authenticated Connect board, use the public API:

```{.python filename="example.py"}
board.pin_write(data, name="username/pin-name", type="parquet", data_dict=True)
board.pin_write(data, name="username/pin-name", type="parquet", data_dict="data-dict.yaml", force_identical_write=True)
```

Dictionary-only changes need `force_identical_write=True` because pins compares data hashes when deciding whether to write.
Generated dictionaries contain `todo` notes for human decisions; review the draft before relying on it.

The CLI is optional and the supported version is `0.0.3`.
Ordinary pins do not require it.
Generated dictionary pages block executable links and remove links with unsafe URL schemes.
Dictionary-only edits require the force option; skipped writes do not draft or render a dictionary.
Install the optional extra with `python -m pip install "pins[data-dict]"`, or provide an executable on `PATH`.
88 changes: 88 additions & 0 deletions examples/data-dict/create_bundle.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
"""Build a pin bundle from this checkout for deployment with rsconnect-python."""

import argparse
import json
import os
import tempfile
from pathlib import Path

import pandas as pd
import yaml

from pins import board_connect, board_folder
from pins.rsconnect.fs import PinBundleManifest


def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("output", type=Path)
dictionary_options = parser.add_mutually_exclusive_group()
dictionary_options.add_argument("--dictionary", type=Path)
dictionary_options.add_argument("--no-dictionary", action="store_true")
args = parser.parse_args()
if args.no_dictionary:
data_dict = None
elif args.dictionary is not None:
data_dict = args.dictionary
else:
data_dict = True
output = args.output.resolve()
if output.exists() and any(output.iterdir()):
parser.error("The output directory must be empty.")

data = pd.DataFrame(
{"id": [1, 2, 3], "value": [10.0, 20.0, 30.0], "group": ["a", "b", "a"]}
)
with tempfile.TemporaryDirectory() as temporary:
local = board_folder(temporary)
meta = local.pin_write(
data,
"dictionary-example",
type="parquet",
metadata={"example_language": "Python"},
data_dict=data_dict,
)
pd.testing.assert_frame_equal(local.pin_read("dictionary-example"), data)
assert meta.user["example_language"] == "Python"
dictionary = meta.user.get("data_dict")
if dictionary is not None:
assert all(
"source" not in table for table in yaml.safe_load(dictionary)["tables"]
)

# Prepare the normal Connect bundle offline; the deployment CLI handles OAuth.
connect = board_connect(
server_url=os.environ.get("CONNECT_SERVER", "https://connect.example.com"),
api_key="unused-offline",
cache=None,
)
output.mkdir(parents=True, exist_ok=True)
prepared = connect.prepare_pin_version(
output,
data,
"example/data-dict-python",
type="parquet",
title="Pins Python data dictionary",
metadata=meta.user,
)
PinBundleManifest.add_manifest_to_directory(output)
uploaded = yaml.safe_load((output / "data.txt").read_text(encoding="utf-8"))
if dictionary is None:
assert "data_dict" not in uploaded["user"]
else:
assert uploaded["user"]["data_dict"] == dictionary
pd.testing.assert_frame_equal(pd.read_parquet(output / prepared.file), data)
manifest = json.loads((output / "manifest.json").read_text(encoding="utf-8"))
assert manifest["metadata"]["primary_html"] == "index.html"
if dictionary is not None:
assert "Data dictionary" in (output / "index.html").read_text(
encoding="utf-8"
)
assert "unused-offline" not in "".join(
p.read_text(encoding="utf-8") for p in output.glob("*.txt")
)
print(output / "manifest.json")


if __name__ == "__main__":
main()
25 changes: 25 additions & 0 deletions examples/data-dict/data-dict.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
$version: 0.1.0
$learn_more: https://data-dict.tidyverse.org/
label: Pins sample data
description: >-
Three synthetic observations used to demonstrate publishing a data dictionary
alongside a Parquet pin from R and Python.
tables:
- name: data
columns:
- name: id
type: number(id)
description: Unique identifier for a synthetic observation.
constraints: [primary_key]
examples: [1, 2, 3]
- name: value
type: number
description: Synthetic numeric value used to demonstrate a numeric column.
constraints: [required]
range: [10, 30]
examples: [10, 20, 30]
- name: group
type: string
description: Synthetic grouping label; a and b identify the two example groups.
constraints: [required]
examples: [a, b]
Loading
Loading