diff --git a/Dockerfile b/Dockerfile index 4db85e3..43127b2 100644 --- a/Dockerfile +++ b/Dockerfile @@ -14,6 +14,9 @@ WORKDIR /app COPY --from=build /work/build/server ./server COPY LICENSE /app/LICENSE COPY licenses/ /app/licenses/ +# Mount point for STORE_DIR (/notes storage), owned by the runtime user so a +# fresh named volume inherits it. Unused unless STORE_DIR is set. +RUN install -d -o 65532 -g 65532 /data ENV PORT=8080 EXPOSE 8080 USER 65532:65532 diff --git a/README.md b/README.md index a0947c9..d52360e 100644 --- a/README.md +++ b/README.md @@ -6,8 +6,10 @@ One small JSON API, one shared Almide implementation, three execution routes: - **Wasm + compiler-generated JavaScript host**: Cloudflare Workers - **Wasm + Node function adapters**: AWS Lambda, Google Cloud Run functions, Azure Functions -The first example proves portable application logic. It is not a cloud SDK, -production framework, storage abstraction, or claim that all Almide I/O works on +The example proves portable application logic, including logic that reads and +writes storage: `/notes` keeps a small list in Workers KV, Cloud Storage or a +file, and the same Almide code decides every read and write on every route. It +is not a cloud SDK, production framework, or claim that all Almide I/O works on every provider. ## Verified scope @@ -38,6 +40,14 @@ See [verification notes](docs/verification.md#compiler-pin-moved-to-the-v0660-re | Google Cloud Run functions, Node 24 | Source package, local Functions Framework, and managed source build via `deploy.sh --execute` | 18 direct adapter cases; in the cloud, 18 octet-stream cases passed and JSON showed the same 3 framework rejections as locally | Deployed temporarily with internal ingress + IAM, verified, deleted | | Azure Functions v4, Node 24 | Source package, adapter/config tests and actual SDK request objects tested | 18 common cases; Functions host/key enforcement not run | Not deployed | +The `/notes` storage scenario ([tests/notes.mjs](tests/notes.mjs), 10 requests in +order against an empty store) passed locally on every route (native files and a +restart, Wasm, Node adapters, local workerd KV, Compose with a named volume and a +container restart) and live on Cloudflare Workers with KV, on the Cloud Run +container with Almide reading and writing Cloud Storage itself, and on Cloud Run +functions with Cloud Storage; each store held exactly the two saved notes +afterwards. The ConoHa VPS run predates `/notes`. + GitHub Actions (`ubuntu-24.04`, compiler install and every reproduction step) has passed. See [verification notes](docs/verification.md) for exact commands and limits. @@ -124,21 +134,52 @@ authentication emulators. ## What is shared [src/api.almd](src/api.almd) owns path routing, input validation, response status, -JSON encoding, and method errors. Its public boundary is three strings in and -one JSON envelope out: `handle(method, target, body) -> String`. +JSON encoding, method errors and what to store. Its public boundary is strings in +and one JSON envelope out: `step(method, target, body, reads) -> String` +(`handle(method, target, body)` is the same without storage). -- [src/native.almd](src/native.almd) maps Almide HTTP requests/responses -- [src/wasm.almd](src/wasm.almd) exposes the same handler to generated JS -- [worker.js](providers/cloudflare-workers/worker.js) maps Workers Request/Response +- [src/native.almd](src/native.almd) maps Almide HTTP requests/responses, and performs storage itself +- [src/wasm.almd](src/wasm.almd) exposes the same functions to generated JS +- [worker.js](providers/cloudflare-workers/worker.js) maps Workers Request/Response, with KV as the store - [adapters/node-wasm.mjs](adapters/node-wasm.mjs) shares one lazy Wasm initializer across Node function adapters -- [tests/cases.mjs](tests/cases.mjs) is the shared expected-behavior fixture +- [adapters/step.mjs](adapters/step.mjs) runs `step` for the JS hosts; [adapters/gcs-store.mjs](adapters/gcs-store.mjs) is Cloud Storage for Node +- [tests/cases.mjs](tests/cases.mjs) and [tests/notes.mjs](tests/notes.mjs) are the shared expected-behavior fixtures The HTTP response contains the envelope's `body`; `status` and optional `allow` -become HTTP metadata. The host adapters do not duplicate the greeting logic. +become HTTP metadata. The host adapters do not duplicate the application logic. + +### Storage: Almide decides, the host performs + +The generated JS host has no asynchronous I/O, and a stock Wasm build refuses +Almide's HTTP client (`http.get` is E081 on `--target wasm`), while Workers KV, +`fetch` and Cloud Storage are asynchronous. So the shared code describes storage +instead of doing it: + +1. The host calls `step(method, target, body, "{}")`. +2. If the envelope has `"read": ["notes"]`, the host reads those keys and calls + `step` again with `reads` = `{"notes": "" | null}`. +3. The envelope with `"status"` is final. If it has `"write": {"key", "value"}`, + the host stores it before answering; a failed read or write answers 503 + `storage_unavailable`. + +| Route | Store | Who performs it | +| --- | --- | --- | +| Native (ConoHa, Cloud Run container) | `STORE_DIR` files, or `GCS_BUCKET` objects | Almide: `fs`, or `http.request` with a metadata-server token | +| Cloudflare Workers | KV binding `NOTES` | `worker.js` | +| Cloud Run functions | `GCS_BUCKET` objects | `adapters/gcs-store.mjs` (`fetch`, no SDK) | +| Lambda, Azure Functions | none configured | `/notes` answers 503 | + +`/notes` keeps one JSON list under the key `notes`: `POST {"text": ...}` (1–280 +code points) adds `{n, text}` and answers 201; `GET` lists the newest first. Only +the newest 50 are kept. A request is a read-modify-write of that one key: native +`http.serve` handles one request at a time, so one native process never +interleaves two, but concurrent instances (Workers isolates, Cloud Run instances, +several replicas) can lose a write. There is no locking, versioning or +conditional write; this is a demonstration of the boundary, not a database. ## Deliberately small contract -- `/health`: GET; `/greet`: POST; query strings are ignored +- `/health`: GET; `/greet`: POST; `/notes`: GET and POST; query strings are ignored - `name` must be a string of 1–100 Unicode code points; it is not trimmed - Request body limit in shared logic: 8,192 Unicode code points - Application errors are JSON: 400, 404, 405 with `Allow`, and 413 @@ -149,8 +190,8 @@ become HTTP metadata. The host adapters do not duplicate the greeting logic. ceiling and 30-second read/response limits. Rejections before the shared handler need not have the same JSON shape as application errors - Host-specific request normalization, streaming, transport limits, HEAD behavior, - concurrency/load, TLS, storage, retries, Secrets and outbound async I/O are - outside the portable contract. Provider authentication configuration is + concurrency/load, TLS, storage consistency, retries and Secrets are outside the + portable contract. Provider authentication configuration is included, but cloud enforcement has not been verified The JS host supports scalar/String boundaries, not arbitrary records and lists. diff --git a/adapters/gcs-store.mjs b/adapters/gcs-store.mjs new file mode 100644 index 0000000..c2210ad --- /dev/null +++ b/adapters/gcs-store.mjs @@ -0,0 +1,32 @@ +// Cloud Storage as a key -> string store for the Node adapters, with the runtime +// identity's token from the metadata server. Uses only fetch: no SDK dependency. +const METADATA_TOKEN = 'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token'; + +async function token() { + const response = await fetch(METADATA_TOKEN, { headers: { 'Metadata-Flavor': 'Google' } }); + if (!response.ok) throw new Error(`metadata token: HTTP ${response.status}`); + return (await response.json()).access_token; +} + +export function gcsStore(bucket) { + const base = `https://storage.googleapis.com/storage/v1/b/${encodeURIComponent(bucket)}/o`; + const upload = `https://storage.googleapis.com/upload/storage/v1/b/${encodeURIComponent(bucket)}/o`; + return { + async get(key) { + const response = await fetch(`${base}/${encodeURIComponent(key)}?alt=media`, { + headers: { Authorization: `Bearer ${await token()}` }, + }); + if (response.status === 404) return null; + if (!response.ok) throw new Error(`GCS read: HTTP ${response.status}`); + return response.text(); + }, + async put(key, value) { + const response = await fetch(`${upload}?uploadType=media&name=${encodeURIComponent(key)}`, { + method: 'POST', + headers: { Authorization: `Bearer ${await token()}`, 'Content-Type': 'application/json' }, + body: value, + }); + if (!response.ok) throw new Error(`GCS write: HTTP ${response.status}`); + }, + }; +} diff --git a/adapters/node-wasm.mjs b/adapters/node-wasm.mjs index 10273a7..72c7085 100644 --- a/adapters/node-wasm.mjs +++ b/adapters/node-wasm.mjs @@ -1,16 +1,21 @@ // Common Node host for the three function adapters. The compiler-generated JS // owns the Wasm ABI; every request uses the same synchronous Almide function. import { readFile } from 'node:fs/promises'; -import { init, handle } from '../build/app.js'; +import { init, step } from '../build/app.js'; +import { runStep } from './step.mjs'; +import { gcsStore } from './gcs-store.mjs'; let initialization; -export async function callApi(method, target, body = '') { +// GCS_BUCKET selects Cloud Storage; without it, requests that need storage get 503. +const defaultStore = process.env.GCS_BUCKET ? gcsStore(process.env.GCS_BUCKET) : null; + +export async function callApi(method, target, body = '', { store = defaultStore } = {}) { if (![method, target, body].every(value => typeof value === 'string')) { throw new TypeError('callApi expects method, target and body strings'); } initialization ??= readFile(new URL('../build/app.wasm', import.meta.url)) .then(bytes => init(bytes)); await initialization; - return JSON.parse(handle(method, target, body)); + return runStep(step, method, target, body, store); } diff --git a/adapters/step.mjs b/adapters/step.mjs new file mode 100644 index 0000000..1cd3fc9 --- /dev/null +++ b/adapters/step.mjs @@ -0,0 +1,38 @@ +// Runs the shared Almide `step` to a final envelope. Almide decides what to read +// and write; the host performs it. `store` is { get(key) -> string | null, +// put(key, value) } (sync or async), or null when the deployment has none. +// No Node or provider APIs here, so Workers and the Node adapters share it. +export const UNAVAILABLE = Object.freeze({ status: 503, body: { error: 'storage_unavailable' } }); + +export async function runStep(step, method, target, body, store) { + const reads = {}; + for (let round = 0; ; round++) { + const envelope = JSON.parse(step(method, target, body, JSON.stringify(reads))); + if (Array.isArray(envelope.read)) { + if (!store || round >= 2) return UNAVAILABLE; + try { + for (const key of envelope.read) reads[key] = (await store.get(key)) ?? null; + } catch (error) { + console.error('storage read failed:', error?.message ?? error); + return UNAVAILABLE; + } + continue; + } + const { write, ...result } = envelope; + if (write) { + if (!store) return UNAVAILABLE; + try { + await store.put(write.key, write.value); + } catch (error) { + console.error('storage write failed:', error?.message ?? error); + return UNAVAILABLE; + } + } + return result; + } +} + +export function memoryStore(initial = {}) { + const data = new Map(Object.entries(initial)); + return { data, get: key => data.get(key) ?? null, put: (key, value) => { data.set(key, value); } }; +} diff --git a/docs/verification.md b/docs/verification.md index 9b38a98..37b35f5 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -260,9 +260,52 @@ locally and used through `dev_overrides`), region c3j1. `CapDrop=[ALL]`; `docker compose down` took 0.5 s 6. `terraform destroy` removed all 5 resources; the VPS existed about 23 minutes +## /notes: storage decided by Almide, performed by the host + +Date: 2026-10-04, Almide `v0.66.0` release binary. The probe that settled the +design: `http.get` builds natively but a `--target wasm --host js` build refuses +it (E081), while a synchronous `@extern(wasm, "js", ...)` builds. JS storage APIs +are asynchronous, so `api.step` returns the reads it needs and the write to make, +and each host performs them (root README, "Storage: Almide decides, the host +performs"). + +Local (macOS arm64, Node 24.21.0): + +1. `npm test` 156/156: the 10-request scenario through generated Wasm with an + in-memory store, the stored document, 503 without a store, 500 on a corrupt + document, the 50-note cap; native with `STORE_DIR` (and notes surviving a + server restart) and native without storage (503); the Node function host with + an injected store, and Lambda without one (503) +2. `test:workers` 29/29 with a local KV (`--persist-to` a fresh directory); + `test:google-framework` 39/39; `test:staged-functions` 38/38 +3. Compose with the named volume: 18 cases + the scenario, then `down` / `up` + kept both notes; root filesystem still read-only, UID 65532 + +Live: + +1. Cloudflare Workers: `wrangler deploy` provisioned the KV namespace + `almide-cloud-example-notes` from the id-less binding. 18 cases + the scenario + passed (29/29); `wrangler kv key get notes --remote` returned exactly the two + saved notes. `wrangler delete` left the namespace; it was deleted separately +2. Google, a new disposable project (deleted afterwards), one bucket per service + with public access prevention, `roles/storage.objectUser` for the runtime + account on each bucket only: + - Cloud Run container with `GCS_BUCKET`: the Almide server took the + metadata-server token and read/wrote Cloud Storage with `http.request`. + From the in-VPC VM with an ID token: 18/18 and the scenario 11/11 + - Cloud Run functions with `GCS_BUCKET` via `deploy.sh`: the Node host read + and wrote with `fetch`. 18/18 and the scenario 11/11 (octet-stream bodies) + - Each bucket's `notes` object held exactly the two saved notes + (`application/json`, 53 bytes); unauthenticated requests got 403 +3. The ConoHa VPS was not rerun with `/notes` (its run predates it) + +Not shown: behavior under concurrent writers. The single-key read-modify-write +has no conditional write, so concurrent instances can lose a note. + ## Not established - Native x86_64 Docker build outside the ConoHa VPS +- `/notes` on the ConoHa VPS, and `/notes` under concurrent writers on any route - ConoHa TLS/reverse proxy, restart behavior, production load, or plans smaller than `g2l-t-c4m4` - Azure Container Apps or ECS Fargate provider validation/deployment diff --git a/providers/aws-lambda/README.md b/providers/aws-lambda/README.md index f25ead6..794afa1 100644 --- a/providers/aws-lambda/README.md +++ b/providers/aws-lambda/README.md @@ -45,6 +45,11 @@ cleanup. Reserved concurrency is not a spending cap. ## Configuration, secrets and logs +`/notes` has no store on this route: the adapter passes none, so `/notes` +answers 503 `storage_unavailable` (tested). A store would be a host-side +`{ get, put }` passed to `callApi`, as Cloud Run functions does with Cloud Storage. + + Environment variables configured on Lambda are visible to the Node host through `process.env`. The current shared `callApi` boundary passes only method, target and body; it does not import host environment values into Wasm. There are no diff --git a/providers/azure-functions/README.md b/providers/azure-functions/README.md index bf23af1..851bbf3 100644 --- a/providers/azure-functions/README.md +++ b/providers/azure-functions/README.md @@ -50,6 +50,12 @@ small package into an app containing unrelated functions. ## Configuration, secrets, logs and cleanup +`/notes` has no store on this route: the adapter passes none, so `/notes` +answers 503 `storage_unavailable` (the shared `callApi` default, tested +through the Lambda adapter). A store would be a host-side +`{ get, put }` passed to `callApi`, as Cloud Run functions does with Cloud Storage. + + Azure application settings appear in the Node host environment. `callApi` does not pass those settings into Wasm; it currently sends only method, target and body. No application secret retrieval is implemented. A later Key Vault diff --git a/providers/cloudflare-workers/README.md b/providers/cloudflare-workers/README.md index cb96dd8..bf9e39a 100644 --- a/providers/cloudflare-workers/README.md +++ b/providers/cloudflare-workers/README.md @@ -32,9 +32,30 @@ after a first deploy, the workers.dev route can briefly answer quotas and behavior under real load were not tested. See the root support matrix for the exact evidence. -Bindings, Secrets, outbound asynchronous `fetch`, D1, R2 and KV are intentionally -outside this first example. They would belong in host adapters. The generated JS -host's environment is not a transparent bridge to Workers bindings. +## /notes and KV + +`/notes` is stored in the KV binding `NOTES`. The shared Almide `step` says which +key to read and what to write; `worker.js` performs it with `env.NOTES` (see the +root README, "Storage: Almide decides, the host performs"). The generated JS +host's environment is not a transparent bridge to Workers bindings, so the +adapter, not Almide, touches KV. + +`wrangler.jsonc` names the binding without an `id`, so `wrangler deploy` +provisions a namespace called `almide-cloud-example-notes`, and `wrangler dev +--local` simulates one (the tests use `--persist-to` with a fresh directory). +**`wrangler delete` does not delete that namespace**; remove it separately: + +```sh +npx wrangler kv namespace list +npx wrangler kv namespace delete --namespace-id +``` + +On 2026-10-04 the scenario in `tests/notes.mjs` passed against the deployed +Worker, the namespace then held exactly the two saved notes +(`wrangler kv key get notes --remote`), and the Worker and namespace were deleted. +KV is eventually consistent across locations and has no conditional write, so +concurrent POSTs from different isolates can lose one. Secrets, D1, R2 and +outbound `fetch` remain outside the example. ## Official references diff --git a/providers/cloudflare-workers/worker.js b/providers/cloudflare-workers/worker.js index 8ebf2e5..d6d8b1e 100644 --- a/providers/cloudflare-workers/worker.js +++ b/providers/cloudflare-workers/worker.js @@ -1,16 +1,20 @@ import module from '../../build/app.wasm'; -import { init, handle } from '../../build/app.js'; +import { init, step } from '../../build/app.js'; +import { runStep } from '../../adapters/step.mjs'; // One init per isolate. No filesystem/URL fallback and no handwritten Wasm ABI. const ready = init(module); +// The NOTES KV namespace is the store; without the binding, /notes answers 503. +const kvStore = kv => kv && { get: key => kv.get(key), put: (key, value) => kv.put(key, value) }; + export default { - async fetch(request) { + async fetch(request, env) { await ready; const url = new URL(request.url); const body = request.method === 'GET' || request.method === 'HEAD' ? '' : await request.text(); - const response = JSON.parse(handle(request.method, url.pathname, body)); + const response = await runStep(step, request.method, url.pathname, body, kvStore(env.NOTES)); return Response.json(response.body, { status: response.status, ...(response.allow ? { headers: { Allow: response.allow } } : {}), diff --git a/providers/cloudflare-workers/wrangler.jsonc b/providers/cloudflare-workers/wrangler.jsonc index dfb81e4..6b3f531 100644 --- a/providers/cloudflare-workers/wrangler.jsonc +++ b/providers/cloudflare-workers/wrangler.jsonc @@ -6,5 +6,8 @@ // Almide's generated glue has a top-level import.meta.url and an unused // node:fs/promises fallback. Keep both requirements explicit and tested. "compatibility_flags": ["nodejs_compat", "new_module_registry"], - "workers_dev": true + "workers_dev": true, + // /notes storage. Without an id, `wrangler deploy` provisions a namespace; + // `wrangler dev --local` simulates it. Remove the binding and /notes answers 503. + "kv_namespaces": [{ "binding": "NOTES" }] } diff --git a/providers/conoha/README.md b/providers/conoha/README.md index 430ba48..4154e6c 100644 --- a/providers/conoha/README.md +++ b/providers/conoha/README.md @@ -67,6 +67,13 @@ curl --fail http://127.0.0.1:8080/greet \ docker compose -f providers/conoha/compose.yaml down ``` +`/notes` is stored by the Almide server itself as files under `STORE_DIR=/data`, +a named volume (`notes`) that outlives the container; the image creates `/data` +owned by the runtime user, so the read-only root filesystem stays read-only. +`docker compose down` keeps the volume; `docker compose down -v` deletes the notes. +Locally (macOS arm64) the notes scenario passed and the notes survived +`down` / `up`; the ConoHa VPS run above predates `/notes`. + Compose binds the app only to the host loopback address. For public access, place a TLS reverse proxy in front and configure the intended ConoHa security group and guest firewall yourself. Docker-published ports can bypass UFW; UFW alone is diff --git a/providers/conoha/compose.yaml b/providers/conoha/compose.yaml index cdbf33e..3a1630e 100644 --- a/providers/conoha/compose.yaml +++ b/providers/conoha/compose.yaml @@ -5,6 +5,7 @@ services: context: ../.. environment: PORT: "8080" + STORE_DIR: /data # /notes storage, in the named volume below # Keep the sample private on the VPS. Add a TLS reverse proxy deliberately. ports: - "127.0.0.1:8080:8080" @@ -12,3 +13,8 @@ services: read_only: true cap_drop: [ALL] security_opt: [no-new-privileges:true] + volumes: + - notes:/data + +volumes: + notes: diff --git a/providers/google-cloud-functions/README.md b/providers/google-cloud-functions/README.md index 0bb2bcd..1aa699a 100644 --- a/providers/google-cloud-functions/README.md +++ b/providers/google-cloud-functions/README.md @@ -38,8 +38,16 @@ export GCP_REGION=YOUR_REGION export GCP_SERVICE=YOUR_NEW_SERVICE export GCP_SERVICE_ACCOUNT=RUNTIME_ACCOUNT@YOUR_PROJECT.iam.gserviceaccount.com export GCP_BUILD_SERVICE_ACCOUNT=projects/YOUR_PROJECT/serviceAccounts/BUILD_ACCOUNT@YOUR_PROJECT.iam.gserviceaccount.com +# Optional: /notes storage. Without it, /notes answers 503. +export GCS_BUCKET=YOUR_EXISTING_BUCKET ``` +With `GCS_BUCKET`, `deploy.sh` adds `--set-env-vars GCS_BUCKET=...` and +[adapters/gcs-store.mjs](../../adapters/gcs-store.mjs) stores `/notes` as the object +`notes` with `fetch` and the metadata-server token (no SDK dependency). Grant the +runtime account `roles/storage.objectUser` on that bucket only. The shared Almide +`step` decides the read and the write; the Node host performs them. + From the repository root, `bash providers/google-cloud-functions/deploy.sh` only prints a reviewed, shell-escaped command. Adding `--execute` runs it and uploads source, builds/stores an image, and deploys a billable service. Neither diff --git a/providers/google-cloud-functions/deploy.sh b/providers/google-cloud-functions/deploy.sh index c876d81..945b5b9 100755 --- a/providers/google-cloud-functions/deploy.sh +++ b/providers/google-cloud-functions/deploy.sh @@ -19,5 +19,7 @@ command=(gcloud run deploy "$GCP_SERVICE" --project "$GCP_PROJECT" --region "$GC --service-account "$GCP_SERVICE_ACCOUNT" --build-service-account "$GCP_BUILD_SERVICE_ACCOUNT" --no-allow-unauthenticated --invoker-iam-check --ingress internal --concurrency 1 --min-instances 0 --max-instances 3 --memory 256Mi --cpu 1 --timeout 30s) +# Optional: the bucket /notes is stored in. Without it, /notes answers 503. +if [[ -n "${GCS_BUCKET:-}" ]]; then command+=(--set-env-vars "GCS_BUCKET=$GCS_BUCKET"); fi printf '%q ' "${command[@]}"; printf '\n' if [[ ${1:-} == '--execute' ]]; then exec "${command[@]}"; fi diff --git a/providers/google-cloud-run/README.md b/providers/google-cloud-run/README.md index 60036d7..d901819 100644 --- a/providers/google-cloud-run/README.md +++ b/providers/google-cloud-run/README.md @@ -113,10 +113,21 @@ docker push "$IMAGE_TAG" docker buildx imagetools inspect "$IMAGE_TAG" # Copy the returned full Digest (64 hexadecimal characters), not the mutable tag. export IMAGE_DIGEST="${IMAGE_PATH}@sha256:REPLACE_WITH_64_HEX_DIGEST" +# Optional: /notes storage. Without it, /notes answers 503. +export GCS_BUCKET='your-existing-bucket' mkdir -p build node providers/google-cloud-run/render.mjs > build/cloud-run.service.yaml ``` +With `GCS_BUCKET`, the renderer adds it to the container environment and the +Almide server itself stores `/notes` as the object `notes` in that bucket: it +takes the runtime service account's token from the metadata server and calls the +Cloud Storage JSON API with `http.request`. Grant the runtime account +`roles/storage.objectUser` on that bucket only, for example +`gcloud storage buckets add-iam-policy-binding gs://$GCS_BUCKET --member=serviceAccount:$RUNTIME_SERVICE_ACCOUNT --role=roles/storage.objectUser`. +Keep the bucket's public access prevention on. Each Cloud Run instance does its +own read-modify-write, so concurrent POSTs on different instances can lose one. + [buildx](https://docs.docker.com/reference/cli/docker/buildx/build/) builds the shared image; [imagetools inspect](https://docs.docker.com/reference/cli/docker/buildx/imagetools/inspect/) shows the registry digest. The renderer is local-only, rejects missing values and diff --git a/providers/google-cloud-run/render.mjs b/providers/google-cloud-run/render.mjs index a3ab9f7..e76a308 100644 --- a/providers/google-cloud-run/render.mjs +++ b/providers/google-cloud-run/render.mjs @@ -26,6 +26,12 @@ const service = JSON.parse(readFileSync(new URL('./service.template.yaml', impor service.metadata.name = serviceName; service.spec.template.spec.serviceAccountName = runtimeServiceAccount; service.spec.template.spec.containers[0].image = imageDigest; +// Optional: the bucket /notes is stored in. Without it, /notes answers 503. +const bucket = process.env.GCS_BUCKET; +if (bucket) { + if (!/^[a-z0-9][a-z0-9._-]{1,61}[a-z0-9]$/.test(bucket)) throw new Error('GCS_BUCKET must be a bucket name'); + service.spec.template.spec.containers[0].env = [{ name: 'GCS_BUCKET', value: bucket }]; +} const output = JSON.stringify(service, null, 2); if (/__[A-Z_]+__/.test(output)) throw new Error('Unresolved template placeholder'); process.stdout.write(`${output}\n`); diff --git a/scripts/package-faas.mjs b/scripts/package-faas.mjs index fd5c7d3..806c6b6 100644 --- a/scripts/package-faas.mjs +++ b/scripts/package-faas.mjs @@ -16,7 +16,7 @@ const metadata = JSON.stringify({ provider, source: 'almide-cloud-examples' }) + // Check all required inputs before touching a previous generated package. for (const path of [ - 'adapters/node-wasm.mjs', 'build/app.js', 'build/app.wasm', + 'adapters/node-wasm.mjs', 'adapters/step.mjs', 'adapters/gcs-store.mjs', 'build/app.js', 'build/app.wasm', `providers/${provider}/package.json`, `providers/${provider}/package-lock.json`, ]) await stat(join(root, path)); @@ -37,7 +37,7 @@ try { await mkdir(output, { recursive: true }); await writeFile(join(output, marker), metadata); -for (const path of ['adapters/node-wasm.mjs', 'build/app.js', 'build/app.wasm', 'licenses', 'LICENSE']) { +for (const path of ['adapters/node-wasm.mjs', 'adapters/step.mjs', 'adapters/gcs-store.mjs', 'build/app.js', 'build/app.wasm', 'licenses', 'LICENSE']) { const dest = join(output, path); await mkdir(resolve(dest, '..'), { recursive: true }); await cp(join(root, path), dest, { recursive: true }); diff --git a/src/api.almd b/src/api.almd index 5508076..7008879 100644 --- a/src/api.almd +++ b/src/api.almd @@ -1,4 +1,6 @@ // Shared, synchronous application logic. No sockets, environment, or provider SDK. +// Storage is described, not performed: `step` asks the host for the keys it needs +// to read and returns the write the host must make before answering. import json fn reply(status: Int, body: Value) -> String = json.stringify(value.object([ @@ -37,3 +39,83 @@ pub fn handle(method: String, target: String, body: String) -> String = { else method_error("POST") } else error(404, "not_found") } + +// --- /notes: a small list kept by the host under one key --------------------- + +fn read_request(key: String) -> String = json.stringify(value.object([ + ("read", value.array([value.str(key)])), +])) + +// none: the host has not read `key` yet; some(none): it did, and nothing is stored. +fn lookup(reads: Value, key: String) -> Option[Option[String]] = + match json.get_path(reads, json.field(json.root(), key)) { + none => none, + some(v) => match value.as_string(v) { + ok(text) => some(some(text)), + err(_) => some(none), + }, + } + +fn stored_notes(doc: Option[String]) -> Result[List[Value], String] = match doc { + none => ok([]), + some(text) => match json.parse(text) { + err(e) => err(e), + ok(v) => value.as_array(v), + }, +} + +fn note_text(body: String) -> Result[String, String] = match json.parse(body) { + err(_) => err("invalid_json"), + ok(data) => match json.get_string(data, "text") { + none => err("text_required"), + some(text) => if string.len(text) == 0 or string.len(text) > 280 then err("text_length") + else ok(text), + }, +} + +fn list_notes(notes: List[Value]) -> String = + reply(200, value.object([("notes", value.array(list.reverse(notes)))])) + +fn add_note(notes: List[Value], text: String) -> String = { + let n = match list.last(notes) { + some(last) => (json.get_int(last, "n") ?? 0) + 1, + none => 1, + } + let note = value.object([("n", value.int(n)), ("text", value.str(text))]) + let kept = list.take_end(list.flatten([notes, [note]]), 50) + json.stringify(value.object([ + ("status", value.int(201)), + ("body", value.object([("note", note)])), + ("write", value.object([ + ("key", value.str("notes")), + ("value", value.str(json.stringify(value.array(kept)))), + ])), + ])) +} + +fn with_notes(reads: Value, f: (List[Value]) -> String) -> String = match lookup(reads, "notes") { + none => read_request("notes"), + some(doc) => match stored_notes(doc) { + ok(notes) => f(notes), + err(_) => error(500, "store_corrupt"), + }, +} + +fn notes(method: String, body: String, reads: Value) -> String = + if method == "GET" then with_notes(reads, (notes) => list_notes(notes)) + else if method == "POST" then match note_text(body) { + err(code) => error(400, code), + ok(text) => with_notes(reads, (notes) => add_note(notes, text)), + } + else method_error("GET, POST") + +// The host calls step with reads = "{}", performs any "read" it gets back (a +// JSON object of key -> stored string or null), and calls again; a reply with +// "status" is final, and its optional "write" is made before the reply is sent. +pub fn step(method: String, target: String, body: String, reads: String) -> String = { + let path = string.split(target, "?") |> list.get(0) ?? target + if path == "/notes" and string.len(body) <= 8192 then match json.parse(reads) { + ok(r) => notes(method, body, r), + err(_) => error(500, "invalid_reads"), + } else handle(method, target, body) +} diff --git a/src/native.almd b/src/native.almd index 1a28c83..956703a 100644 --- a/src/native.almd +++ b/src/native.almd @@ -1,10 +1,116 @@ import api import env +import fs import http import json +import url + +// --- Storage the native host performs for api.step ------------------------- +// STORE_DIR keeps each key in a file; GCS_BUCKET keeps it as a Cloud Storage +// object, authenticated with the runtime identity from the metadata server. +// With neither, a request that needs storage is answered 503. + +// An empty variable counts as unset. +fn nonempty(o: Option[String]) -> Option[String] = match o { + some(v) => if v == "" then none else some(v), + none => none, +} + +effect fn gcs_token() -> Result[String, String] = { + let (status, body) = http.request_status("GET", + "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token", + "", ["Metadata-Flavor": "Google"])! + if status != 200 then err("metadata token: HTTP ${status}") + else match json.get_string(json.parse(body)!, "access_token") { + some(token) => ok(token), + none => err("metadata token: no access_token"), + } +} + +effect fn read_key(key: String) -> Result[Option[String], String] = + match nonempty(env.get("STORE_DIR")) { + some(dir) => { + let path = "${dir}/${key}.json" + if fs.exists(path) then ok(some(fs.read_text(path)!)) else ok(none) + }, + none => match nonempty(env.get("GCS_BUCKET")) { + some(bucket) => { + let object = url.encode_component(key) + let (status, body) = http.request_status("GET", + "https://storage.googleapis.com/storage/v1/b/${bucket}/o/${object}?alt=media", + "", ["Authorization": "Bearer " + gcs_token()!])! + if status == 200 then ok(some(body)) + else if status == 404 then ok(none) + else err("GCS read: HTTP ${status}") + }, + none => err("no storage configured"), + }, + } + +effect fn write_key(key: String, content: String) -> Result[Unit, String] = + match nonempty(env.get("STORE_DIR")) { + some(dir) => { + // Write then rename, so a reader never sees a half-written file. + let path = "${dir}/${key}.json" + fs.write(path + ".tmp", content)! + fs.rename(path + ".tmp", path) + }, + none => match nonempty(env.get("GCS_BUCKET")) { + some(bucket) => { + let object = url.encode_component(key) + let (status, _) = http.request_status("POST", + "https://storage.googleapis.com/upload/storage/v1/b/${bucket}/o?uploadType=media&name=${object}", + content, ["Authorization": "Bearer " + gcs_token()!, "Content-Type": "application/json"])! + if status == 200 then ok(()) else err("GCS write: HTTP ${status}") + }, + none => err("no storage configured"), + }, + } + +effect fn read_all(keys: List[String], acc: List[(String, Value)]) -> Result[List[(String, Value)], String] = + match list.first(keys) { + none => ok(acc), + some(key) => { + let stored = match read_key(key)! { + some(text) => value.str(text), + none => value.null(), + } + read_all(list.drop(keys, 1), list.flatten([acc, [(key, stored)]])) + }, + } + +fn unavailable() -> Value = json.parse( + "{\"status\":503,\"body\":{\"error\":\"storage_unavailable\"}}") ?? value.null() + +// Runs api.step to a final envelope, reading what it asks for (at most twice) +// and making its write. Native http.serve is sequential, so the read-modify-write +// of one request cannot interleave with another's in this process. +effect fn run(method: String, target: String, body: String, reads: List[(String, Value)], round: Int) -> Value = { + let envelope = json.parse(api.step(method, target, body, json.stringify(value.object(reads))))! + match json.get_array(envelope, "read") { + some(keys) => if round >= 2 then unavailable() else { + let names = list.filter_map(keys, (k) => match value.as_string(k) { ok(s) => some(s), err(_) => none }) + match read_all(names, reads) { + ok(more) => run(method, target, body, more, round + 1)!, + err(e) => { eprintln("storage read failed: ${e}"); unavailable() }, + } + }, + none => match json.get_path(envelope, json.field(json.root(), "write")) { + none => envelope, + some(w) => { + let key = json.get_string(w, "key") ?? "" + let content = json.get_string(w, "value") ?? "" + match write_key(key, content) { + ok(_) => envelope, + err(e) => { eprintln("storage write failed: ${e}"); unavailable() }, + } + }, + }, + } +} effect fn handle(req: HttpRequest) -> HttpResponse = { - let envelope = json.parse(api.handle(http.req_method(req), http.req_path(req), http.req_body(req)))! + let envelope = run(http.req_method(req), http.req_path(req), http.req_body(req), [], 0)! let status = json.get_int(envelope, "status") ?? 500 let body = json.get_path(envelope, json.field(json.root(), "body")) ?? value.null() let response = http.json(status, json.stringify(body)) @@ -17,5 +123,9 @@ effect fn handle(req: HttpRequest) -> HttpResponse = { effect fn main() -> Unit = { let port_text = env.get("PORT") ?? "8080" let port = int.parse(port_text)! + match nonempty(env.get("STORE_DIR")) { + some(dir) => fs.mkdir_p(dir)!, + none => (), + } http.serve(port, (req) => handle(req)!)! } diff --git a/src/wasm.almd b/src/wasm.almd index 768b0db..642eb93 100644 --- a/src/wasm.almd +++ b/src/wasm.almd @@ -3,4 +3,7 @@ import api pub fn handle(method: String, target: String, body: String) -> String = api.handle(method, target, body) +pub fn step(method: String, target: String, body: String, reads: String) -> String = + api.step(method, target, body, reads) + fn main() -> Unit = {} diff --git a/tests/contract.test.mjs b/tests/contract.test.mjs index 2a3df82..36a7c6b 100644 --- a/tests/contract.test.mjs +++ b/tests/contract.test.mjs @@ -1,9 +1,12 @@ import assert from 'node:assert/strict'; import { test } from 'node:test'; -import { readFile } from 'node:fs/promises'; +import { readFile, mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; import { init, handle } from '../build/app.js'; import { cases } from './cases.mjs'; import { freePort, startServer, verifyHttp } from './http-harness.mjs'; +import { verifyNotes, httpCall } from './notes.mjs'; test('generated Wasm JS contract (compiled-module entry)', async t => { const module = await WebAssembly.compile(await readFile(new URL('../build/app.wasm', import.meta.url))); @@ -28,3 +31,51 @@ test('native HTTP contract', async t => { t.after(server.stop); await verifyHttp(t, server.base); }); + +test('generated Wasm notes: the host performs the reads and writes step asks for', async t => { + const { step } = await import('../build/app.js'); + const { runStep, memoryStore } = await import('../adapters/step.mjs'); + const store = memoryStore(); + await verifyNotes(t, async (method, path, body) => { + const r = await runStep(step, method, path, body, store); + return { status: r.status, allow: r.allow ?? null, json: r.body }; + }); + await t.test('stored document is the JSON list of notes', () => { + assert.deepEqual(JSON.parse(store.data.get('notes')), [{ n: 1, text: 'hello' }, { n: 2, text: 'δΈ–η•Œ 🌏' }]); + }); + await t.test('no store answers 503, and a corrupt document 500', async () => { + assert.deepEqual(await runStep(step, 'GET', '/notes', '', null), { status: 503, body: { error: 'storage_unavailable' } }); + const corrupt = await runStep(step, 'GET', '/notes', '', memoryStore({ notes: 'not json' })); + assert.deepEqual(corrupt, { status: 500, body: { error: 'store_corrupt' } }); + }); + await t.test('only the newest 50 notes are kept', async () => { + const many = memoryStore(); + for (let i = 1; i <= 55; i++) await runStep(step, 'POST', '/notes', JSON.stringify({ text: `n${i}` }), many); + const kept = JSON.parse(many.data.get('notes')); + assert.equal(kept.length, 50); + assert.deepEqual([kept[0], kept.at(-1)], [{ n: 6, text: 'n6' }, { n: 55, text: 'n55' }]); + }); +}); + +test('native notes: Almide reads and writes files under STORE_DIR, across restarts', async t => { + const dir = await mkdtemp(join(tmpdir(), 'almide-notes-')); + t.after(() => rm(dir, { recursive: true, force: true })); + const port = await freePort(); + let server = await startServer('./build/server', [], port, { PORT: String(port), STORE_DIR: dir }); + await verifyNotes(t, httpCall(server.base)); + await server.stop(); + server = await startServer('./build/server', [], port, { PORT: String(port), STORE_DIR: dir }); + t.after(server.stop); + await t.test('notes survive a restart', async () => { + const r = await httpCall(server.base)('GET', '/notes', ''); + assert.deepEqual(r.json.notes.map(n => n.n), [2, 1]); + }); +}); + +test('native notes without storage answer 503', async t => { + const port = await freePort(); + const server = await startServer('./build/server', [], port, { PORT: String(port), STORE_DIR: '', GCS_BUCKET: '' }); + t.after(server.stop); + const r = await httpCall(server.base)('GET', '/notes', ''); + assert.deepEqual([r.status, r.json], [503, { error: 'storage_unavailable' }]); +}); diff --git a/tests/faas.test.mjs b/tests/faas.test.mjs index d305ae7..e28a2ea 100644 --- a/tests/faas.test.mjs +++ b/tests/faas.test.mjs @@ -112,3 +112,21 @@ test('warm and overlapping invocations remain isolated', async () => { assert.deepEqual(JSON.parse(result.body), { message: `Hello, ${i}!` }); } }); + +test('Node function host runs the notes scenario against an injected store', async t => { + const { callApi } = await import('../adapters/node-wasm.mjs'); + const { memoryStore } = await import('../adapters/step.mjs'); + const store = memoryStore(); + const { verifyNotes } = await import('./notes.mjs'); + await verifyNotes(t, async (method, path, body) => { + const r = await callApi(method, path, body, { store }); + return { status: r.status, allow: r.allow ?? null, json: r.body }; + }); +}); + +test('function adapters without GCS_BUCKET answer /notes with 503', async () => { + const c = { method: 'GET', path: '/notes' }; + const result = await lambda(eventFor(c)); + assert.equal(result.statusCode, 503); + assert.deepEqual(JSON.parse(result.body), { error: 'storage_unavailable' }); +}); diff --git a/tests/notes.mjs b/tests/notes.mjs new file mode 100644 index 0000000..bf89ac6 --- /dev/null +++ b/tests/notes.mjs @@ -0,0 +1,38 @@ +// The /notes scenario: the same requests, in order, against one empty store. +// `call(method, path, body)` returns { status, json, allow } for any route. +import assert from 'node:assert/strict'; + +export const notesScenario = [ + { name: 'empty list', method: 'GET', path: '/notes', status: 200, json: { notes: [] } }, + { name: 'first note', method: 'POST', path: '/notes', body: '{"text":"hello"}', status: 201, json: { note: { n: 1, text: 'hello' } } }, + { name: 'Unicode note', method: 'POST', path: '/notes', body: JSON.stringify({ text: 'δΈ–η•Œ 🌏' }), status: 201, json: { note: { n: 2, text: 'δΈ–η•Œ 🌏' } } }, + { name: 'newest first', method: 'GET', path: '/notes?x=1', status: 200, json: { notes: [{ n: 2, text: 'δΈ–η•Œ 🌏' }, { n: 1, text: 'hello' }] } }, + { name: 'invalid JSON', method: 'POST', path: '/notes', body: '{', status: 400, json: { error: 'invalid_json' } }, + { name: 'missing text', method: 'POST', path: '/notes', body: '{}', status: 400, json: { error: 'text_required' } }, + { name: 'empty text', method: 'POST', path: '/notes', body: '{"text":""}', status: 400, json: { error: 'text_length' } }, + { name: '281 code points', method: 'POST', path: '/notes', body: JSON.stringify({ text: '🌏'.repeat(281) }), status: 400, json: { error: 'text_length' } }, + { name: 'wrong method', method: 'DELETE', path: '/notes', status: 405, allow: 'GET, POST', json: { error: 'method_not_allowed' } }, + { name: 'rejected writes stored nothing', method: 'GET', path: '/notes', status: 200, json: { notes: [{ n: 2, text: 'δΈ–η•Œ 🌏' }, { n: 1, text: 'hello' }] } }, +]; + +export async function verifyNotes(t, call) { + for (const c of notesScenario) { + await t.test(`notes: ${c.name}`, async () => { + const result = await call(c.method, c.path, c.body ?? ''); + assert.equal(result.status, c.status); + if (c.allow) assert.equal(result.allow, c.allow); + assert.deepEqual(result.json, c.json); + }); + } +} + +// HTTP form of `call`, for servers, local runtimes and deployed services. +export const httpCall = (base, { headers = {} } = {}) => async (method, path, body) => { + const response = await fetch(base + path, { + method, + headers: { 'content-type': 'application/json', ...headers }, + ...(method === 'GET' || method === 'HEAD' ? {} : { body }), + signal: AbortSignal.timeout(10000), + }); + return { status: response.status, allow: response.headers.get('allow'), json: await response.json() }; +}; diff --git a/tests/package-faas.test.mjs b/tests/package-faas.test.mjs index 70e6994..1f5717b 100644 --- a/tests/package-faas.test.mjs +++ b/tests/package-faas.test.mjs @@ -10,7 +10,8 @@ async function fixture(t) { t.after(() => rm(root, { recursive: true, force: true })); const provider = 'aws-lambda'; const files = { - 'adapters/node-wasm.mjs': '// adapter\n', 'build/app.js': '// generated\n', + 'adapters/node-wasm.mjs': '// adapter\n', 'adapters/step.mjs': '// step\n', + 'adapters/gcs-store.mjs': '// gcs\n', 'build/app.js': '// generated\n', 'build/app.wasm': 'fixture bytes', 'LICENSE': 'fixture license', 'licenses/Almide-MIT.txt': 'third-party notice', [`providers/${provider}/handler.mjs`]: 'export const handler = () => {};', diff --git a/tests/workers.test.mjs b/tests/workers.test.mjs index f3cec07..2d3a1c5 100644 --- a/tests/workers.test.mjs +++ b/tests/workers.test.mjs @@ -1,14 +1,22 @@ import { test } from 'node:test'; +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; import { freePort, startServer, verifyHttp } from './http-harness.mjs'; +import { verifyNotes, httpCall } from './notes.mjs'; test('Workers local workerd contract', async t => { const port = await freePort(); const inspectorPort = await freePort(); + // A fresh local KV state, so the notes scenario starts from an empty store. + const state = await mkdtemp(join(tmpdir(), 'almide-wrangler-')); + t.after(() => rm(state, { recursive: true, force: true })); const server = await startServer('./node_modules/.bin/wrangler', [ 'dev', '--local', '--ip', '127.0.0.1', '--port', String(port), '--config', 'providers/cloudflare-workers/wrangler.jsonc', - '--inspector-port', String(inspectorPort), + '--inspector-port', String(inspectorPort), '--persist-to', state, ], port, { WRANGLER_SEND_METRICS: 'false', CI: '1' }); t.after(server.stop); await verifyHttp(t, server.base); + await verifyNotes(t, httpCall(server.base)); });