diff --git a/CLAUDE.md b/CLAUDE.md index 8f4abdd..f558a40 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -34,7 +34,7 @@ maintains it. the service routes by `dev` and `sid` and never reads. - **Live readings stay in memory.** The versioned project statistics scope covers rounded closed-hour energy and configured capacities on - disk for 13 months. Only group cells pass `/v1/stats`; no staff route reads + disk for 48 hours. Only group cells pass `/v1/stats`; no staff route reads a home. Withdrawal and home purge delete its unpublished statistics. Memory holds live readings, and only while the roster's grant serves a copy; SQLite holds only the home id, the box key, the signed diff --git a/README.md b/README.md index 1e5404b..2a1ac71 100644 --- a/README.md +++ b/README.md @@ -384,7 +384,7 @@ The sender refuses redirects, proxies and local destinations. ## Project usage statistics `GET /v1/stats` publishes only group cells under -`ftw-project-stats-2026-10-v2`. Collection stays off until the signed C8 DPIA +`ftw-project-stats-2026-10-v3`. Collection stays off until the signed C8 DPIA and the legal basis for both flows have been reviewed. Empty `FTW_CLOUD_STATISTICS_POLICY` keeps it off; an exact approved policy is needed to advertise collection. While paused, `/v1/stats` returns 503 with @@ -393,7 +393,7 @@ The unsigned [review draft](privacy/project-statistics-dpia.md) records the open decisions. Clear terms alone do not establish a lawful basis. Statistics are included for every FTW Cloud user under -`ftw-cloud-2026-10-v2`. Owners accept the current scope through +`ftw-cloud-2026-10-v3`. Owners accept the current scope through `POST /api/cloud/grant`. There is no separate statistics opt-out within Cloud; `DELETE /api/cloud` ends Cloud and requests deletion. Old terms do not accept the new scope. Deploy compatible Core and clients, show the new terms, and @@ -408,7 +408,7 @@ no phone keys, pairing codes, Noise key, live cache or remote access. `DELETE /api/cloud/stats` stops this local contribution. A current Cloud grant refuses that separate stop with 409. Never answer for another household. -Cloud stores the approved hourly fields for 13 months and deletes unpublished +Cloud stores the approved hourly fields for 48 hours and deletes unpublished household rows on withdrawal or purge. Only closed hours starting after the new choice contribute; no earlier ledger history is uploaded. Group values need 20 homes and no home above 30 percent of a sum. The stats Worker receives diff --git a/deploy/README.md b/deploy/README.md index e29f956..756db7f 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,9 +1,9 @@ # Running FTW Cloud FTW Cloud runs on one `t4g.small` in AWS eu-north-1 (Stockholm) and answers -at `https://cloud.ftw.energy`. No box sends to it yet: release builds of Core -name no service, and a home can turn FTW Cloud on only from a Core built with -the uplink (srcfl/ftw#1595), after its owner decides to. +at `https://cloud.ftw.energy`. Current Core and native clients can use it after +the owner enables Cloud. Local control stays on the box. Hourly project +statistics remain off until the C8 review and policy approval are complete. ## The host @@ -25,7 +25,7 @@ Every AWS resource carries the tag `Project=ftw-cloud`. ## Shape -- One instance, one container (`compose.yml`), no standby. A rebuilt host +- One instance, `ftwcloudd` and the separate keys service (`compose.yml`), no standby. A rebuilt host starts empty and boxes fill it again when they reconnect. - `ftwcloudd` ends TLS 1.3 itself and gets its certificate from Let's Encrypt by TLS-ALPN-01 on 443, so port 80 stays shut. **No TLS-terminating proxy in @@ -43,7 +43,9 @@ Every AWS resource carries the tag `Project=ftw-cloud`. supplies `compose.yml`. - The Docker volume `ftw-cloud_ftw-cloud-state` holds the SQLite registry (home ids, box keys, signed rosters, tombstones) and the ACME cache, on the - encrypted root disk. No readings. No backup. + encrypted root disk. With the reviewed statistics policy enabled, it also + holds rounded closed-hour reports for 48 hours. Live readings stay in + memory. No statistics backup. ## The service key @@ -70,7 +72,8 @@ secret the same way. - Swap off (`swapoff -a`, no swap in fstab) and core dumps off: the kernel pipes cores to `/bin/false`, systemd-coredump stores nothing, `compose.yml` sets `ulimits: core: 0`, and `ftwcloudd` marks itself not dumpable. - Household readings exist only in this process's memory. + Live readings stay in process memory. The separate project-statistics + scope stores only its approved closed-hour fields on disk for 48 hours. - Nothing comes in but 443, TCP and UDP. There is no SSH and no key pair; administration is through SSM. Session logs to a separate account are not set up yet. - Outbound is open. The host now needs only GitHub and `ghcr.io` (the image @@ -373,6 +376,9 @@ Cloud data; the images on `ghcr.io` hold only code. The APNs key ([apns.md](apns.md)) and the Noise key phones dial ([noise.md](noise.md)) come from Parameter Store. HTTP/3 needs UDP 443 in the security group ([http3.md](http3.md)). Widget tokens need no -secret: the registry holds their hashes. Statistics (C7–C8) add secrets, -egress and storage when they land. The erase test, canary homes -and the external heartbeat that pages Sourceful belong with the first homes. +secret: the registry holds their hashes. Statistics (C8) have their own +[review](../privacy/project-statistics-dpia.md) and +[rights and retention procedure](../privacy/statistics-operations.md). The +policy flag stays empty until that review is signed off. Signed deletion, +offline rights handling and startup retention have tests. Canary homes and +the external heartbeat that pages Sourceful still need production evidence. diff --git a/deploy/compose.yml b/deploy/compose.yml index f1ef77e..e936d46 100644 --- a/deploy/compose.yml +++ b/deploy/compose.yml @@ -40,7 +40,8 @@ services: # "Deploys"). stop_grace_period: 15s volumes: - # The registry and certificates. No readings are ever written here. + # Registry, certificates and, only after C8 approval, closed-hour + # project statistics with 48-hour retention. Live readings stay in memory. - ftw-cloud-state:/state # Apple's push key, read from Parameter Store into tmpfs at boot by # deploy/apns.sh. Empty means no alerts, and the service runs. diff --git a/go.mod b/go.mod index aeffb7c..a0d658d 100644 --- a/go.mod +++ b/go.mod @@ -9,7 +9,7 @@ require ( github.com/fxamacker/cbor/v2 v2.9.4 github.com/gorilla/websocket v1.5.3 github.com/quic-go/quic-go v0.63.0 - github.com/srcfl/ftw/go/protocol v0.0.0-20261017145623-2b1cc2563972 + github.com/srcfl/ftw/go/protocol v0.0.0-20261017163816-75a8b2c910ad golang.org/x/crypto v0.57.0 golang.org/x/net v0.58.0 golang.org/x/sys v0.48.0 diff --git a/go.sum b/go.sum index 153446d..d3572ca 100644 --- a/go.sum +++ b/go.sum @@ -52,8 +52,8 @@ github.com/quic-go/quic-go v0.63.0 h1:LIFGHI4PFUhhw2dDD1ARHdCff143ffMHwZtbnbuJ78 github.com/quic-go/quic-go v0.63.0/go.mod h1:RAro2j2yN9a9EiPACLHT9IB2NXCvGQmmo/alT0yYI0w= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= -github.com/srcfl/ftw/go/protocol v0.0.0-20261017145623-2b1cc2563972 h1:+YZbUeomGm9bDdpVICXMJS8AbgaTaeWrRA+QZjcuNAc= -github.com/srcfl/ftw/go/protocol v0.0.0-20261017145623-2b1cc2563972/go.mod h1:VEjr/o0yZijJ1nApuJK+nak6FxJ/fubAUBNOunqNrUU= +github.com/srcfl/ftw/go/protocol v0.0.0-20261017163816-75a8b2c910ad h1:DFoeYtGWX2iVb6QdZMY4rv7HRrhUG649sKs2Yjd6X0s= +github.com/srcfl/ftw/go/protocol v0.0.0-20261017163816-75a8b2c910ad/go.mod h1:VEjr/o0yZijJ1nApuJK+nak6FxJ/fubAUBNOunqNrUU= github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE= github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg= github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM= diff --git a/internal/registry/statistics.go b/internal/registry/statistics.go index 2e6bece..56e5d56 100644 --- a/internal/registry/statistics.go +++ b/internal/registry/statistics.go @@ -15,14 +15,41 @@ import ( // Statistics tables contain only versioned, owner-approved closed-hour values. // They have no staff read route, and leave with the home's purge. func (r *Registry) StatisticsInit() error { - _, err := r.db.Exec(`CREATE TABLE IF NOT EXISTS statistics_grants ( + tx, err := r.db.Begin() + if err != nil { + return err + } + defer tx.Rollback() + _, err = tx.Exec(`CREATE TABLE IF NOT EXISTS statistics_grants ( home TEXT PRIMARY KEY REFERENCES homes(home) ON DELETE CASCADE, - stats_id BLOB NOT NULL, since_ms INTEGER NOT NULL); + stats_id BLOB NOT NULL, since_ms INTEGER NOT NULL, policy TEXT NOT NULL DEFAULT ''); CREATE TABLE IF NOT EXISTS statistics_hours ( home TEXT NOT NULL REFERENCES homes(home) ON DELETE CASCADE, hour_ms INTEGER NOT NULL, metrics BLOB NOT NULL, PRIMARY KEY(home,hour_ms)); + CREATE TABLE IF NOT EXISTS statistics_restrictions ( + home TEXT PRIMARY KEY REFERENCES homes(home) ON DELETE CASCADE); CREATE INDEX IF NOT EXISTS statistics_hours_time ON statistics_hours(hour_ms);`) - return err + if err != nil { + return err + } + var column int + if err := tx.QueryRow(`SELECT COUNT(*) FROM pragma_table_info('statistics_grants') WHERE name='policy'`).Scan(&column); err != nil { + return err + } + if column == 0 { + if _, err := tx.Exec(`ALTER TABLE statistics_grants ADD COLUMN policy TEXT NOT NULL DEFAULT ''`); err != nil { + return err + } + } + // A stored grant from an older binary proves no acceptance of this policy. + if _, err := tx.Exec(`DELETE FROM statistics_hours WHERE NOT EXISTS + (SELECT 1 FROM statistics_grants g WHERE g.home=statistics_hours.home AND g.policy=?)`, cloudwire.StatsPolicy); err != nil { + return err + } + if _, err := tx.Exec(`DELETE FROM statistics_grants WHERE policy<>?`, cloudwire.StatsPolicy); err != nil { + return err + } + return tx.Commit() } // StatisticsGrant clears old consent rows before a changed grant may send. @@ -33,7 +60,8 @@ func (r *Registry) StatisticsGrant(home string, g cloudwire.Grant) error { } defer tx.Rollback() var old []byte - err = tx.QueryRow(`SELECT stats_id FROM statistics_grants WHERE home=?`, home).Scan(&old) + var oldPolicy string + err = tx.QueryRow(`SELECT stats_id,policy FROM statistics_grants WHERE home=?`, home).Scan(&old, &oldPolicy) if err != nil && !errors.Is(err, sql.ErrNoRows) { return err } @@ -41,7 +69,16 @@ func (r *Registry) StatisticsGrant(home string, g cloudwire.Grant) error { (g.TermsPolicy == cloudwire.StatsPolicy && len(g.Scopes) == 0 && len(g.Purposes) == 1 && g.Purposes[0] == protocol.CloudPurposeStats) on := terms && g.Controller == cloudwire.ControllerSourceful && g.StatsPolicy == cloudwire.StatsPolicy && len(g.StatsID) == 16 && g.StatsSinceMs > 0 && slices.Contains(g.Purposes, protocol.CloudPurposeStats) - if !on || !bytes.Equal(old, g.StatsID) { + if on { + var restricted int + if err := tx.QueryRow(`SELECT COUNT(*) FROM statistics_restrictions WHERE home=?`, home).Scan(&restricted); err != nil { + return err + } + if restricted != 0 { + return errors.New("statistics: subject request restricts this home") + } + } + if !on || !bytes.Equal(old, g.StatsID) || oldPolicy != cloudwire.StatsPolicy { if _, err := tx.Exec(`DELETE FROM statistics_hours WHERE home=?`, home); err != nil { return err } @@ -50,8 +87,8 @@ func (r *Registry) StatisticsGrant(home string, g cloudwire.Grant) error { } } if on { - if _, err := tx.Exec(`INSERT INTO statistics_grants(home,stats_id,since_ms) VALUES(?,?,?) - ON CONFLICT(home) DO UPDATE SET stats_id=excluded.stats_id,since_ms=excluded.since_ms`, home, g.StatsID, g.StatsSinceMs); err != nil { + if _, err := tx.Exec(`INSERT INTO statistics_grants(home,stats_id,since_ms,policy) VALUES(?,?,?,?) + ON CONFLICT(home) DO UPDATE SET stats_id=excluded.stats_id,since_ms=excluded.since_ms,policy=excluded.policy`, home, g.StatsID, g.StatsSinceMs, g.StatsPolicy); err != nil { return err } } @@ -69,14 +106,51 @@ func (r *Registry) StatisticsPut(home string, report cloudwire.Stats, now time.T // Consent and the write are one SQL statement: a withdrawn grant cannot // race a pending report back into storage. Reconnects replace one hour. _, err = r.db.Exec(`INSERT INTO statistics_hours(home,hour_ms,metrics) - SELECT home,?,? FROM statistics_grants WHERE home=? AND stats_id=? AND since_ms<=? - ON CONFLICT(home,hour_ms) DO UPDATE SET metrics=excluded.metrics`, report.HourMS, data, home, report.StatsID, report.HourMS) + SELECT home,?,? FROM statistics_grants WHERE home=? AND stats_id=? AND since_ms<=? AND policy=? + AND NOT EXISTS (SELECT 1 FROM statistics_restrictions x WHERE x.home=statistics_grants.home) + ON CONFLICT(home,hour_ms) DO UPDATE SET metrics=excluded.metrics`, report.HourMS, data, home, report.StatsID, report.HourMS, cloudwire.StatsPolicy) return err } +// StatisticsRestrictions is read once before restoring the in-memory homes. +func (r *Registry) StatisticsRestrictions() (map[string]bool, error) { + rows, err := r.rd.Query(`SELECT home FROM statistics_restrictions`) + if err != nil { + return nil, err + } + defer rows.Close() + homes := map[string]bool{} + for rows.Next() { + var home string + if err := rows.Scan(&home); err != nil { + return nil, err + } + homes[home] = true + } + return homes, rows.Err() +} + +// StatisticsRestricted is an operator-imposed rights hold, never an owner +// setting. Only the offline rights procedure writes it with Cloud stopped. +func (r *Registry) StatisticsRestricted(home string) (bool, error) { + var n int + err := r.rd.QueryRow(`SELECT COUNT(*) FROM statistics_restrictions WHERE home=?`, home).Scan(&n) + return n != 0, err +} + func (r *Registry) StatisticsPrune(now time.Time) error { - _, err := r.db.Exec(`DELETE FROM statistics_hours WHERE hour_ms=? AND h.hour_ms=? AND h.hour_ms","home":"","identity_checked":true,"verified_by":"","verified_at":""} + ``` + +3. Planera ett kort Cloud-avbrott. Dränera och stoppa `ftwcloudd`; stoppa inte + lokal styrning på boxarna. Kontrollera att processen har slutat. Databasens + exklusiva lås ger ytterligare skydd; verktyget ska vägra när Cloud kör. +4. Kör från den granskade källversionen med värdens Python 3. Databasen ligger + i den befintliga state-volymen; använd den faktiska mount-sökvägen från + Compose, inte en sökväg som du gissar. + + ```sh + python3 scripts/statistics-subject-request.py --database "$FTW_REGISTRY" --request "$FTW_CASE" --output "$FTW_EXPORT" + ``` + + Utdraget innehåller endast det angivna hemmets statistik-id, policy, starttid och + timrader. Det läser inte nycklar, roster, tokens eller andra hem. Utfilen + skapas med 0600; verktyget vägrar skriva över en fil eller följa en utfilslänk. +5. För verifierad radering eller tillfällig begränsning av statistik: + + ```sh + python3 scripts/statistics-subject-request.py --database "$FTW_REGISTRY" --request "$FTW_CASE" --erase + ``` + + Åtgärden raderar timrader och statistikgrant och sparar en spärr för hemmet. + Den använder SQLite `secure_delete` och tömmer WAL. Det bevisar inte + radering av varje fysisk kopia på underliggande SSD. En gammal box eller + ny policy får inte börja ladda upp igen. Cloud-åtkomst väntar också medan + ärendet är spärrat, eftersom Cloud omfattar statistik. Detta är ingen + vanlig inställning för att välja bort statistik inom Cloud. Informera den + berörda personen om följden före åtgärden. Lokal drift fortsätter. + + Om verktyget avslutas med fel: utgå inte från att inget har ändrats. Behåll + Cloud stoppat, kontrollera ärendet och databasens tillstånd och försök igen + med samma verifierade ärende. Radering är upprepningsbar. En tom databas får + aldrig ersätta originalet för att dölja ett fel. + +6. Starta Cloud med insamlingens tidigare aktiveringsläge. Kontrollera hälsa, + att andra hem kan återansluta och att det spärrade hemmet inte återställs. + Spara verktygets ärende-id och radantal, inte hushållsvärden i driftloggen. +7. Lämna utdraget bara till verifierad mottagare via en överenskommen skyddad + kanal. Lägg det inte i en oskyddad e-postbilaga. Radera arbetsfiler efter + bekräftad leverans, senast efter sju dagar enligt denna föreslagna rutin. + +En spärr gäller tills ärendet har avgjorts. Den följer inte med till en annan +hemnyckel. Ingen självbetjäning tar bort den. Ett separat, dokumenterat beslut +krävs innan en operatör häver en spärr med Cloud stoppat. Radera inte spärren +bara för att en box försöker ansluta igen. Full hemradering via originalnyckeln +fungerar även under spärren och tar bort hemmets övriga Cloud-uppgifter. + +För rättelse: begränsa statistiken enligt ovan medan ansvarig utreder. Ändra +inte signerade mätvärden som om boxen hade skickat dem. Radera de felaktiga +serverraderna, rätta källan på boxen och avgör ärendet innan ny rapportering. +Redan publicerade gruppsummor kan kräva separat rättelse på stats-sidan; +radering av hushållsrader ändrar dem inte automatiskt. + +Proven `TestStatisticsSubjectExportIsPrivateScopedAndOffline` och +`TestOfflineRightsErasureSurvivesCloudRestartAndBoxReconnect` använder verklig +Go-skapad SQLite, två testhem, filskydd, omstart och riktiga TLS-anslutningar. +De provar inte en faktisk persons identitet eller leveranskanal. + +## Lagring, backup och incident + +- Hushållstimmar: gallring vid 48 timmars ålder från timstart. Starten gallrar + innan tjänsten blir tillgänglig; därefter kör underhåll varje minut, även + med insamlingen av. Uppgraderingen tar bort äldre policyers grants och + timmar. Ett driftstopp skjuter upp gallring tills processen startar igen. + Publik vy använder bara senaste dygnet och aktuell policy. +- `TestStatisticsMigratesLegacyGrantsAndFiltersEarlierPolicies` provar den + äldre tabellformen och policybytet. `TestStartupPrunesOldHoursEvenWithCollectionDisabled` + provar startgallringen utan aktivering. `TestRestoreReturnsRestrictionStorageFailure` + visar att databasfel i rättighetsspärrar stoppar starten. +- Ingen backup av statistikdatabasen ingår. En trasig värd kan förlora + statistik; boxen behåller sin egen historik. Inför ingen snapshot eller + databasdump utan ny bedömning av lagring och radering. +- Verifierad drift den 17 oktober: inga snapshots för FTW-volymen eller + skyddade AWS Backup-resurser i Stockholm; ingen upptäckt schemalagd + backupväg på värden. Det bevisar inte att ingen någonsin gjort en annan kopia. +- Dockerloggar: tre filer om 10 MiB per container, ingen åldersgräns. Kodens + loggar räknar händelser utan hem-id, adresser eller statistikvärden. + CloudTrail Event History visar SSM-API-anrop; den är ingen inspelning av en + terminals alla kommandon. Ingen beständig trail eller sessionsinspelning + har verifierats för FTW. Bevara relevant incidentbevis i ett separat ärende. +- Vid misstänkt läcka: pausa timinsamlingen, begränsa drifttillträde, bevara + nödvändigt bevis och kontakta Sourcefuls ansvariga samma dag. Ansvarig bedömer + anmälan till IMY inom 72 timmar från vetskap och information till berörda. + Att bara stänga flaggan raderar inte tidigare insamlade rader. + +Källor: [IMY:s rättigheter](https://www.imy.se/verksamhet/dataskydd/det-har-galler-enligt-gdpr/de-registrerades-rattigheter/) +och [förordningen, artiklarna 11–21 och 33–34](https://www.imy.se/verksamhet/dataskydd/det-har-galler-enligt-gdpr/introduktion-till-gdpr/dataskyddsforordningen-i-fulltext/). diff --git a/scripts/statistics-subject-request.py b/scripts/statistics-subject-request.py new file mode 100644 index 0000000..190ab72 --- /dev/null +++ b/scripts/statistics-subject-request.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 +"""Handle one home's verified statistics request, with Cloud stopped.""" +import argparse +import datetime as dt +import json +import os +from pathlib import Path +import re +import sqlite3 +import stat +import sys +import uuid + + +def unique_object(pairs): + value = dict(pairs) + if len(value) != len(pairs): + raise ValueError("duplicate request fields") + return value + + +def handle_request(database, request, output, erase=False): + info = request.lstat() + if not stat.S_ISREG(info.st_mode) or info.st_mode & 0o077 or info.st_size > 4096: + raise ValueError("request file must be private and at most 4096 bytes") + case = json.loads(request.read_text(), object_pairs_hook=unique_object) + if set(case) != {"case_id", "home", "identity_checked", "verified_by", "verified_at"}: + raise ValueError("unexpected request fields") + if any(not isinstance(case[key], str) for key in ("case_id", "home", "verified_by", "verified_at")): + raise ValueError("request identifiers and timestamp must be text") + uuid.UUID(case["case_id"]) + if not re.fullmatch(r"[a-z2-7]{26}", case["home"]) or case["identity_checked"] is not True: + raise ValueError("a verified request for one home is required") + if not re.fullmatch(r"[A-Za-z0-9_.@-]{1,80}", case["verified_by"]): + raise ValueError("a staff identifier is required") + checked = dt.datetime.fromisoformat(case["verified_at"].replace("Z", "+00:00")) + age = dt.datetime.now(dt.timezone.utc) - checked + if not dt.timedelta(minutes=-5) <= age <= dt.timedelta(days=7): + raise ValueError("identity check must be recent") + + db = sqlite3.connect(database.resolve().as_uri() + "?mode=rw", uri=True, timeout=0) + try: + # EXCLUSIVE locking mode also excludes existing WAL readers. Stop and + # drain Cloud first; a normal WAL write transaction alone is not enough. + db.execute("PRAGMA synchronous=FULL") + db.execute("PRAGMA locking_mode=EXCLUSIVE") + db.execute("BEGIN EXCLUSIVE") + home = case["home"] + if not db.execute("SELECT 1 FROM homes WHERE home=?", (home,)).fetchone(): + raise ValueError("home not found; do not guess another identifier") + if erase: + # Keep the hold across reconnects and policy changes. Only a + # resolved rights case may remove it; an old box cannot do so. + db.execute("PRAGMA secure_delete=ON") + db.execute("INSERT OR IGNORE INTO statistics_restrictions(home) VALUES(?)", (home,)) + count = db.execute("DELETE FROM statistics_hours WHERE home=?", (home,)).rowcount + db.execute("DELETE FROM statistics_grants WHERE home=?", (home,)) + db.commit() + if db.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()[0] != 0: + raise sqlite3.OperationalError("WAL checkpoint incomplete") + return {"case_id": case["case_id"], "erased_rows": count, "restricted": True} + grant = db.execute("SELECT stats_id,since_ms,policy FROM statistics_grants WHERE home=?", (home,)).fetchone() + hours = db.execute("SELECT hour_ms,metrics FROM statistics_hours WHERE home=? ORDER BY hour_ms", (home,)).fetchall() + body = {"schema": "ftw.statistics-subject-export/1", "case_id": case["case_id"], + "home": home, "generated_at": dt.datetime.now(dt.timezone.utc).isoformat(), + "grant": {"statistics_id": grant[0].hex(), "since_ms": grant[1], "policy": grant[2]} if grant else None, + "hours": [{"hour_ms": hour, "metrics": json.loads(metrics)} for hour, metrics in hours]} + fd = os.open(output, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + try: + with os.fdopen(fd, "w") as dest: + json.dump(body, dest, ensure_ascii=False, indent=2) + dest.write("\n") + dest.flush() + os.fsync(dest.fileno()) + except BaseException: + os.unlink(output) + raise + return {"case_id": case["case_id"], "rows": len(hours), "output": str(output)} + finally: + db.rollback() + db.close() + + +if __name__ == "__main__": + p = argparse.ArgumentParser(description=__doc__) + p.add_argument("--database", type=Path, required=True) + p.add_argument("--request", type=Path, required=True) + action = p.add_mutually_exclusive_group(required=True) + action.add_argument("--output", type=Path) + action.add_argument("--erase", action="store_true", help="erase statistics and restrict this home's Cloud pending the rights case") + a = p.parse_args() + try: + print(json.dumps(handle_request(a.database, a.request, a.output, a.erase))) + except (OSError, ValueError, TypeError, sqlite3.Error) as error: + print("Request incomplete: " + type(error).__name__ + ". Check the request and database state before retrying; keep Cloud stopped.", file=sys.stderr) + sys.exit(1)