Experimental auditable asynchronous inference. Submit batches, retrieve results and verify usage/settlement receipts.
Latest high-end follow-up: H100 SXM smoke report. 10/10 real requests and receipts passed, zero drift; larger stages were skipped by the time gate after cold startup. The pod is terminated. Posted RunPod GPU/disk cost $0.67913; displayed credit immediately after that run was $7.46 (historical). This does not complete the sustained hardware comparison or demonstrate H100 savings.
Latest complete multi-stage GPU validation: complete hosted A40 report and aggregate measurements. 1,010/1,010 real requests succeeded, including a controlled worker interruption, with 1,010 verified receipts, 1,010 unique usage/capture records and zero wallet drift. The independent watchdog terminated the pod; no paid resources remain. Displayed credit after that A40 run: $7.94 (historical).
Matched 100-request arms averaged 46.80 seconds baseline concurrent versus 53.31 seconds adaptive from API receipt through completion: adaptive was 13.9% slower. Pooled baseline output throughput was 2.67x sequential. Posted full validation GPU/disk cost is $0.21493; the initial quote estimate was $0.21523. These results support supervised execution/audit demonstrations, not a cheapest-inference or adaptive-savings claim. Public beta, live Stripe and automatic provisioning remain disabled.
Auditable execution receipts preserve model identity, prompt/payload/output hashes, token usage and request settlement checks through the API and Python SDK. They are unsigned operator records, not independent attestation or proof of zero retention. Wallet algorithms are unchanged. The hosted experiment verified all 1,010 real receipts against results and a separate SQL audit. This is a defensibility foundation; customer value and competitive economics remain to be established.
The code passed 254 tests (one real-GPU pytest skipped), 31 scheduler-v2 regressions, lint/type checks, the dashboard production build and a 40/40 mock hosted rehearsal with 40 verified receipts before the real run. Real GPU evidence is in the separate experiment report above.
Latest DeepInfer compatibility and budget readiness: source/CPU checks passed, native CUDA build and GPU execution unverified. No four/eight-B200 stock at the check; even five minutes on four B200s exceeds funds above the $5 floor. Current displayed credit: $7.26. No paid resources created.
The evidence-driven throughput controller provides a working offline profiler, bounded benchmark client, hash-bound evidence review and token-aware dispatch replay. Synthetic data cannot qualify a profile; live scheduling is unchanged. Latest DeepSeek has a pinned proposal manifest, with runtime/hardware validation still blocked. No throughput improvement or GPU savings is claimed.
| Experiment | Outcome |
|---|---|
| A40 REST log-contract failure | Model initialized; zero customer admissions. Invalid source query fixed before the completed run. |
| A40 HTTP startup retry | Zero admissions; independent cleanup confirmed. |
| A40 networking readback failure | Zero admissions; provider response handling fixed. |
| Partial H100 run | 110/110 succeeded; comparison stages incomplete. Posted GPU/disk cost $1.36398. |
| Earlier A40 smoke | 10/10 succeeded with four active sequences; later stages not admitted. |
| Second L4 run | 410/410 succeeded with interruption recovery and no ledger drift; adaptive slower. Posted provider cost $0.21400138374883682. |
| First L4 run | Initial real baseline evidence; subsequent recovery fixes documented separately. |
Hosted stage execution keeps the driver, telemetry, policy transitions and audits on GitHub. Provider read authority and independent cleanup credentials stay separate. GitHub protected environments require owner review. The owner has given standing financial approval while preserving a $5 RunPod credit floor, with fresh quote/balance/pending-cost checks, one pod at a time and an authenticated armed watchdog before each manual creation. The provider does not enforce that floor. Hard-deadline, provider-outage and all possible failure paths are not claimed live-proven.
Licensed under Apache License 2.0; model weights, images and dependencies retain their own licenses. Three extraction customer hypotheses and interview guide remain research only. Download v0.3.0 prerelease and release notes.
An isolated Docker backend and temporary HTTPS connection passed 14 mock checks, including 10 completed requests, outbound worker authentication, idempotency and ledger reconciliation. The complete backend suite passed 128 tests with 1 real-GPU test skipped; the targeted safety/readiness suite passed 23 tests. The reusable local rehearsal generates private credentials only under ignored .runtime/, blocks development login/payment routes, and supervises automatic shutdown. See the dated report. A subsequent genuine staging OIDC login passed with owner-restricted access, a protected wallet read and zero reconciliation drift. Those were the preparation blockers at that date; the October 25 reports above record real-mode staging and successful live teardown. This rehearsal used mock inference; no GPU inference performance or cost advantage is claimed.
NOT READY — BLOCKERS REMAIN. Final local validation: 113 tests passed, 1 real-GPU test skipped. See the complete live-site, scheduling, provider and GPU approval report, website screenshots and issue register, provider architecture, first-GPU economics and future private-alpha checklist. At that date, website fixes awaited publication and real GPU inference, savings, remote staging and independent cleanup were unverified; no paid GPU had been created. Current evidence is linked at the top of this README.
Private experimental real mode now has a separate REVIEWED_GPU_EXPERIMENT_ENABLED gate; public beta, Stripe and auto provisioning remain off. Worker metrics include cumulative engine tokens, overlap/idle and hardware memory. Provider selection remains simulation-only; optional providers cannot create/terminate or execute. Customer wallet/settlement algorithms are unchanged.
Submit inference batches. Track execution. Pay for completed tokens.
A prepaid asynchronous text-inference platform built with FastAPI, PostgreSQL, a durable scheduler, outbound GPU workers, vLLM, and Next.js.
Quickstart · Architecture · API and SDK · Testing · GPU deployment · Release notes
MVP prerelease: mock execution, staging OIDC, live L4 baseline inference/accounting and independent provider teardown have passed validation. GPU reconnect/recovery, matched policy comparisons, 1,000-request completion, external payments and sustainable economics remain unvalidated. Limited measured performance is in the dated real-inference report; no pricing advantage or production-readiness claim is made.
The multi-provider and batching readiness report distinguishes implemented/tested behavior and provides the manual comparison procedure. Worker overlap traces, engine metrics, four-slot limits, read-only billing audits and optional-provider discovery contracts are included. Runpod remains the only selectable execution provider; optional paid mutations, automatic failover and provisioning remain disabled. The October 25 L4 test measured four active sequences and sub-hour completion for two baseline stages only; adaptive benefits, larger-load reliability and cost advantage remain unproven.
V0.3 adds an opt-in 60-minute best-effort objective, explicit admission predictions, adaptive concurrency, recommendation-only capacity planning, and internal GPU economics. It preserves the existing queue, worker fences, idempotency and wallet ledger. PR #3, the engineering blog and readiness pull requests are merged into main. No paid capacity or public deployment is activated. The v0.3.0 prerelease identifies its original source snapshot; subsequent merged fixes do not move that release tag.
- Scheduler architecture and gap analysis · editable Mermaid diagram
- Simulation comparison · raw seeded results · real-GPU experiment plan
- GPU unit economics · product objective and limits
- Why we're building InferenceFlow · v0.3 release notes
Apply migration 0006 and restart API/scheduler/worker together. To try v2 locally, add the following to .env before starting Compose or the native services:
SCHEDULER_V2_ENABLED=true
ADAPTIVE_DISPATCH_ENABLED=true
DEFAULT_COMPLETION_TARGET_SECONDS=3600
ESTIMATOR_CALIBRATED=false
PROVISIONING_ENABLED=false
STRIPE_ENABLED=falseBoth new flags default false. Disabling them restores prior scheduling/default-target behavior; existing accepted targets and rates remain intact. Missing mock GPU telemetry conservatively limits adaptive dispatch to two slots. vLLM retains token-level continuous batching.
New submissions return admission with best-effort outcome, estimated duration (or unknown), calibration status, source and reasons. Set admission_policy="reject_if_at_risk" to receive an actionable 503 before credit reservation if the requested target cannot be confidently supported. Uncalibrated estimates cannot satisfy strict admission. This is not an SLA.
The dashboard shows tenant queue/risk/latency/attainment measures. Its separate operator panel requires the metrics credential for capacity, GPU activity, cost and margin reports; tenant credentials cannot read other customers' data. The same report is available to trusted operators via python -m packages.common.cli scheduling-report. Quotes, invoices, sampled activity, mock data and missing coverage are distinguished.
Run reproducible offline experiments without a GPU or database:
python -m scripts.benchmark_scheduler --seed 42 --count 240 --output .runtime/benchmark-v03.jsonThe implementation report and file manifest records validation and remaining work. View the unmodified scheduling dashboard capture.
The six-scenario comparison is synthetic: some changes regress some metrics. It is not GPU evidence; use the dated real-GPU reports above for measured results. General reliability and competitive cost advantage remain unproven. Follow the benchmark plan before real-GPU calibration or spending.
See the October 25 results, evidence, failure analysis and cleanup proof. Preparation history below is not a current quote or permission to create another pod.
For the current six-run baseline/adaptive 10/100/1,000 comparison and optional worker-interruption gate, follow the readiness report. Its policy-verified instructions supersede the earlier single-policy sequence. Customer wallet/rate/settlement behavior is unchanged; physical inference may retry while customer usage/capture must remain unique. All new real-mode commands require explicit approval, valid private credentials and independent provider teardown within the $10 ceiling.
Original selection: RTX 4090 Community had no available capacity at its historical $0.34/hour quote. Subsequent explicit approval used an L4 Secure pod at $0.59/hour GPU plus $0.011/hour temporary disk. The pinned model initialized with context 10,240 and four resident sequences. Historical proposals are superseded by the October 25 results. Gated tools remain in scripts/gpu_experiment.py; no wallet/billing implementation changed.
The first-GPU runbook records the repository review, dated RunPod stock/price snapshot, pinned model/container checks, worker connection instructions, staged 10/100/1,000-request test plan, measurement definitions, budget controls and complete termination procedure.
The October 23, 2026 snapshot recommended one Secure Cloud L40S 48 GB at $1.09/hour compute because the final reads showed no L4 or RTX 4090 stock. This is a historical quote, not reserved capacity or a verified performance-per-dollar advantage. Price, stock and placement must be checked again before any approved creation.
Review the proposed RunPod v2 payload, in-Pod launcher, and control-plane environment overlay. They request one GPU, 80 GB ephemeral disk, no persistent volumes and no exposed inbound ports. Keep PROVISIONING_ENABLED=false and STRIPE_ENABLED=false. Replace connection/credential placeholders securely only after approval; never commit a rendered credential-bearing payload.
The ceiling remains $10 all-in and 120 minutes maximum; RunPod does not guarantee this experiment cap. Earlier approved pods were removed; the second inference rental lasted about 21.5 minutes. Worker recovery passed in that run; durable staging supervision is the remaining execution blocker. Live payments remain disabled and untested.
Previous full-suite validation reached 162 passing tests with one gated GPU pytest skipped. The real GPU loader and read-only audits were executed separately on October 25; their results do not imply that every GPU or recovery scenario passed.
Customers submit independent chat-generation requests, receive a job ID immediately, and retrieve ordered results while the platform owns queueing, execution, recovery, metering, and prepaid accounting. Workloads include document extraction, classification, enrichment, summaries, and synthetic-data batches that tolerate asynchronous completion.
| Area | Included |
|---|---|
| API | 1–100 requests/job, idempotency, partial results, pagination, JSONL, cancellation |
| Identity | OIDC code + PKCE/state/nonce, local sign-in, memberships, one-time API keys |
| Scheduling | Model affinity, weighted fairness/aging, deadlines, concurrency and KV-cache pressure caps |
| Workers | Outbound async agent, deterministic mock, private vLLM, canary, lease recovery |
| Accounting | Integer micro-USD, immutable ledger/rates, atomic capture/release, audited reconciliation |
| Payments | Gated Stripe test Checkout, verified webhooks, deduplication, refunds, dispute freeze |
| Callbacks | Registered HTTPS, HMAC, IP pinning, SSRF/rebinding protection, durable retries |
| Operations | Provider kill switch/caps, watchdog, Prometheus, optional OTLP, content retention |
| Tools | Next.js dashboard, Python SDK, OpenAPI, Compose, CI, tests and runbooks |
Open the deployed InferenceFlow Site — currently owner-private. Making this repository public does not grant access to the hosted Site.
The responsive showcase includes an interactive execution replay, seven actual dashboard captures, Python/cURL integration examples, and downloadable evidence. Its complete static source is included here. The replay uses recorded mock data; the Site does not host the API, database, GPU workers, or payment services.
Preview the Site locally from the repository root:
python -m http.server 8093 --bind 127.0.0.1 --directory sites/inferenceflow/distOpen http://127.0.0.1:8093. Hosting identity is preserved in sites/inferenceflow/.openai/hosting.json; pushing to GitHub does not automatically redeploy Sites.
The recorded walkthrough follows sign-in → prepaid reservation → queued job → worker execution → results → metering and billing, with seven real browser captures, the actual SDK/JSONL evidence, integration diagrams, and commands to reproduce it locally. An interactive screenshot player is included (serve docs/ locally to play it).
This local recording completed four requests and canceled one: 124 microUSD total synthetic charge, zero remaining reservations/debt, and no reconciliation drift. It used seeded credits and mock inference; no GPU capacity or Stripe payments were used. The guide separately explains real GPU, OIDC, Stripe and callback integration.
Requires Git, a running Docker Engine with Compose v2, and free local ports 3000, 8080, and 55432. Mock mode requires no GPU or external service account.
git clone https://github.com/manishklach/inferenceflow.git
cd inferenceflow
cp .env.example .env
docker compose up --build -d
docker compose exec api python -m packages.common.cli seedPowerShell users can copy the environment template with:
Copy-Item .env.example .envThe seed command creates developer@example.local, adds $10 in test funds, and prints a new API key once. Funding is idempotent for its default reference: rerunning creates another key without duplicating that funding entry. Store the displayed key securely.
| Service | URL |
|---|---|
| Dashboard | http://localhost:3000 |
| Interactive API docs | http://localhost:8080/docs |
| OpenAPI | http://localhost:8080/openapi.json |
| Readiness | http://localhost:8080/readyz |
| Metrics | http://localhost:8080/metrics (requires METRICS_TOKEN bearer credential) |
Sign in as developer@example.local, submit a sample, and inspect tokens, charges, results, and ledger. Local outputs start with MOCK; rates are test fixtures.
docker compose exec api python scripts/docker_smoke.pyThis submits 100 requests over HTTP, waits for the separate scheduler/worker, retrieves results, and reconciles the ledger. Recorded result:
{
"completed": 100,
"captured_micro_usd": 2500,
"usage": {"input_tokens": 3690, "output_tokens": 6900},
"ledger_drift": 0
}The charge is $0.002500 at synthetic test rates, not a public service price or GPU cost measurement.
docker compose logs --tail=100 api scheduler callbacks worker
docker compose down
docker compose up -dStopping preserves the PostgreSQL named volume. Retain it to keep accounts, jobs, and accounting history.
The audit remediation guide describes all ten fixes, migration 0005, worker protocol changes, secret rotation, and remaining infrastructure validation. Existing deployments must stop old API/scheduler/worker processes, back up PostgreSQL, apply migrations, and restart all services together. Old worker enrollment credentials must be replaced for production. Run the new callback process as well as the scheduler.
Customer sessions and API keys now share a PostgreSQL-backed allowance of 120 requests per tenant per UTC minute across API replicas. Refunds/disputes freeze the account and cancel outstanding work without further customer capture; consumed reversed funds become journaled debt. Authenticated wallet responses include debt_micro_usd.
flowchart LR
C[SDK / dashboard] --> A[FastAPI]
A --> P[(PostgreSQL)]
S[Scheduler] --> P
W[Outbound worker] -->|poll / heartbeat / finalize| A
W --> M{Execution mode}
M --> L[Local mock]
M --> V[Private vLLM / NVIDIA]
ST[Stripe test Checkout] -->|verified webhook| A
P --> H[Callback delivery worker]
H --> CB[Signed customer callbacks]
G[Independent watchdog] --> R[GPU provider]
PostgreSQL provides durable work and transactional SKIP LOCKED leases. The worker keeps one model warm and calls vLLM concurrently so the engine performs continuous batching.
- Validate the batch, count tokens, snapshot rates, and atomically reserve the maximum charge.
- Assign a lease with an increasing fencing epoch; perform inference outside DB transactions.
- Accept finalization only from the current worker/epoch with a valid lease.
- Atomically persist output/usage, capture once, and release unused funds.
- Recover abandoned work with bounded retries/backoff and reject stale completion.
Execution is at least once; accounting is exactly once. Crashes can repeat physical inference, but fencing and unique accounting references prevent repeated capture.
stateDiagram-v2
[*] --> queued
queued --> leased
leased --> running
leased --> queued: lease recovery
running --> queued: retryable failure
queued --> canceled
leased --> canceled
queued --> expired
leased --> expired
running --> succeeded
running --> failed
running --> billing_pending: inconsistent usage
billing_pending --> succeeded: audited capture
billing_pending --> failed: audited release
Targets are best effort. Unstarted work past target expires without capture. Running work can finish late, including after cancellation, and can be billed if its output is returned.
Customer calls use a Bearer API key. Submissions also require an Idempotency-Key header. Dashboard and worker authentication are separate.
| Method | Endpoint | Purpose |
|---|---|---|
| GET | /v1/models | Revision, limits, rates, mode |
| POST / GET | /v1/jobs | Submit batch / list history |
| GET | /v1/jobs/{id} | Progress, usage and accounting |
| GET | /v1/jobs/{id}/results | Ordered paginated partial results |
| GET | /v1/jobs/{id}/results.jsonl | Authenticated download |
| POST | /v1/jobs/{id}/cancel | Best-effort cancellation |
| GET | /v1/wallet, /v1/wallet/transactions | Balances and ledger |
| POST / GET / DELETE | /v1/keys and /v1/keys/{id} | Dashboard-only key management |
| POST | /v1/webhook-destination | Register callbacks |
| POST | /v1/billing/checkout-session | Gated test top-up |
Sample payload: examples/request.json.
export INFERENCEFLOW_API_KEY='<key from seed or dashboard>'
curl -X POST http://localhost:8080/v1/jobs \
-H "Authorization: Bearer $INFERENCEFLOW_API_KEY" \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: example-job-001' \
--data-binary @examples/request.jsonHTTP 202 returns a job ID and reservation. A repeated key/body returns the same job; a changed body under that key returns 409. See API contract and OpenAPI export.
The SDK is included here; no PyPI publication is claimed. Create/activate a virtual environment, then install:
python -m pip install -e '.[dev]'import os
from packages.python_sdk import InferenceFlowClient
with InferenceFlowClient(os.environ["INFERENCEFLOW_API_KEY"]) as client:
job = client.create_job(requests=[{
"custom_id": "doc-001",
"messages": [{"role": "user", "content": "Summarize this document."}],
"max_output_tokens": 128,
}], tier="economy")
print(client.wait_for_job(job["id"], timeout=300))
print(client.get_results(job["id"]))
print(client.get_wallet())The client generates idempotency keys and retries transient failures. It also supports catalog/status/cancellation. The batch example submits 100 requests:
python examples/batch.pyMoney uses integer micro-USD: USD 1 = 1,000,000 units. Rates are per million tokens and snapshotted at acceptance.
reserve = ceil(prompt_tokens × input_rate / 1,000,000)
+ ceil(max_output_tokens × output_rate / 1,000,000)
capture = ceil(engine_input_tokens × input_rate / 1,000,000)
+ ceil(engine_output_tokens × output_rate / 1,000,000)
release = reservation − capture
Missing/inconsistent usage enters billing_pending, holds its reservation, and requires audited reconciliation; no customer charge is guessed. Failed attempts cost the operator without a customer token charge. Refunds/disputes freeze affected accounts and audit shortfalls.
| Default | Limit |
|---|---|
| Requests/job / body | 1–100 / 5 MiB |
| Prompt / output tokens | 8,192 / 2,048 |
| Pending / in-flight per tenant | 2,000 / 16 |
| Lease TTL / attempts | 60 seconds / 3 |
| Standard / economy target | Legacy: 30 minutes / 24 hours; opt-in v2: configurable 60 minutes, best effort |
| Content retention / GPUs | 7 days / 1 active GPU |
Read billing invariants and reconciliation before enabling payments.
Copy .env.example; keep .env out of Git.
| Setting | Purpose / safe default |
|---|---|
| DATABASE_URL | PostgreSQL; Compose overrides its internal host |
| SECRET_KEY | Dashboard/PKCE signing only; replace development value before deployment |
| EXECUTION_MODE | mock locally, vllm for attached capacity |
| MODEL_REVISION | mock-v1 locally, exact reviewed SHA in real mode |
| WORKER_BOOTSTRAP_TOKEN | Separate worker credential; rotate before deployment |
| PROVISIONING_ENABLED | false |
| MAX_ACTIVE_GPUS | 1, including creation intents |
| MAX_DAILY_GPU_SPEND_USD | Quote-based ceiling, initially 25 |
| STRIPE_ENABLED / PUBLIC_BETA_ENABLED | false / false |
| OIDC_* | Provider/client, issuer/audience, redirect settings |
| OTEL_EXPORTER_OTLP_ENDPOINT | Optional collector; install the observability extra first |
Follow GPU deployment for pins, private vLLM, outbound worker, TLS/network isolation, smoke/restart testing, and shutdown. See model/image lock.
# Downloads only the pinned tokenizer; no weights or rental.
python -m pip install -e '.[gpu]'
python scripts/check_tokenizer.pyReal GPU testing requires approved spending and attached infrastructure. RUN_GPU_E2E=1 opts in after configuration. Stripe stays test-only and grants credits only from verified paid webhooks. Managed OIDC validates PKCE/state/nonce/signature/issuer/audience/expiry. Development sign-in and seeding stay local-only.
| Check | Recorded result |
|---|---|
| v0.3 native PostgreSQL suite | 78 passed; 1 GPU test skipped |
| Existing platform suite with v2 enabled | 31 passed |
| Pre-v0.3 audit baseline | 56 passed; 1 GPU test skipped |
| Earlier v0.1.0 Python 3.12 Docker suite | 32 passed; 1 GPU test skipped (before audit fixes) |
| Native + Docker 100-request HTTP smoke | Succeeded; zero ledger drift |
| Ruff / mypy | Passed |
| Dashboard build / TypeScript | Passed |
| npm production audit | Zero reported vulnerabilities at validation time |
| Exact real tokenizer | Loaded; checked prompt counts 18 tokens |
| GPU / external service integration | Pending |
Coverage includes auth, isolation, idempotency, ordering, concurrent reserves/submissions/finalizers, rounding, immutable ledger, crash recovery, stale epochs, retries, partial success, cancellation, deadlines, signed payments, OIDC, SSRF/rebinding/redirects, callbacks, logging, limits, retention, migrations, and KV-cache pressure.
# Create once; skip if this dedicated test database exists.
docker compose exec postgres createdb -U inferenceflow inferenceflow_test
docker compose exec api python -m pytest -q
docker compose exec api python -m ruff check .
docker compose exec api python -m mypy apps packages
docker compose exec api python -m packages.common.cli reconcileTests apply migrations and truncate only a separate database ending _test. TEST_DATABASE_URL customizes its connection. PostgreSQL is required; SQLite is not used to approximate locking.
For native checks, install the dev extra, then run make test lint typecheck or the matching python -m commands. Build the dashboard separately:
cd apps/dashboard
npm ci
npm run buildRequires PostgreSQL 16, Python 3.12 (3.11 development compatibility), Node.js 22 and npm. Create/activate .venv, copy the environment template, and configure DATABASE_URL.
python -m pip install -e '.[dev,gpu]'
python -m alembic upgrade head
python -m packages.common.cli seedRun in separate terminals from the root:
python -m uvicorn apps.api.main:app --host 127.0.0.1 --port 8080
python -m apps.scheduler.main
# In another terminal:
python -m apps.callbacks.main
# Start after API readiness succeeds:
python -m apps.worker.mainIn another terminal run npm ci and npm run dev from apps/dashboard. The temporary native smoke harness starts its own API/scheduler/worker and cleans them up; run it with ports free:
python scripts/local_demo.py- Docker connection: start your engine/Desktop and verify docker info.
- Occupied ports: stop native previews before normal Docker deployment.
- No credits: use the exact seeded email or seed the intended development account.
- Queued work: inspect logs, readiness, revisions, heartbeats and KV-cache pressure.
- Billing pending: follow audited procedures; never edit ledger rows.
- Monitoring: enable the observability Compose profile; Prometheus is at http://localhost:9090. See alerts.
- Traces: install the observability extra, configure an OTLP HTTP collector, restart API.
- GPU leaks: use an independent watchdog, dedicated account and provider-side limits/alerts. Control-plane ceilings cannot guarantee billing stops during outages.
apps/api/ FastAPI and OIDC
apps/scheduler/ Dispatch, recovery, callbacks, watchdog
apps/worker/ Agent, mock/vLLM engines, telemetry
apps/dashboard/ Next.js + TypeScript interface
packages/common/ DB, auth, ledger, providers, metrics
packages/python_sdk/ InferenceFlowClient
migrations/ Alembic revisions
infra/ Dockerfile, model pins, Prometheus
scripts/ Smoke tests and tokenizer check
tests/ API, accounting, concurrency, security, migrations
examples/ SDK example and API payload
docs/ Contracts, deployment, runbooks, release notes
- Implementation evidence and boundaries
- Architecture, API, OpenAPI
- Billing, GPU deployment, security, operations
- Internal pricing experiment
- GPU Hourly Rates — full RunPod hardware catalog, recorded hourly quotes and daily summaries; data and methodology.
- CPU pricing and history — six CPU Pod families, per-vCPU rates and instance estimates; units and availability caveats.
- Hourly GitHub-hosted pricing collector — independent of the local computer; GET-only catalog access with a separate read-only key, preserved history and automatic website data updates.
- Second real GPU report: 410/410 completed, controlled worker recovery and billing audits passed; adaptive was slower than baseline, and a staging-supervision outage blocked the 1,000-request stage. Pod termination is verified; commercial readiness remains NO-GO.
- Changelog and release notes
- Product specification and implementation brief
One model is enabled. The optional second model, learned throughput prediction, organization UI, and automatic invoice ingestion are outside this implementation. Reservations/idempotency/leases are consolidated into durable rows; encrypted PostgreSQL columns replace object storage.
Before limited prepaid beta, validate GPU inference/recovery, actual OIDC/Stripe/provider flows, callback TLS/egress, backups, measured margins, model licensing, and security/legal/accounting requirements. Mock results and synthetic rates do not establish GPU performance or savings.