Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/actionlint.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# The Namespace runner profile every workflow runs on (runs-on: namespace-profile-execupack).
self-hosted-runner:
labels:
- namespace-profile-execupack
2 changes: 1 addition & 1 deletion .github/workflows/audit-published.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
audit:
# x86: the gate runs ExecuTorch's C++ runner through the wheel, which fails on aarch64
# the same way the verify job does (finding 30).
runs-on: blacksmith-8vcpu-ubuntu-2404
runs-on: namespace-profile-execupack
timeout-minutes: 350
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
Expand Down
16 changes: 15 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,24 @@ permissions:
jobs:
test:
# Lint and unit tests only.
runs-on: blacksmith-2vcpu-ubuntu-2404-arm
runs-on: namespace-profile-execupack
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Runner
# What the Namespace profile really has, read by config/pipeline.yaml's runner_tiers.
run: |
echo "arch $(uname -m), $(nproc) CPUs"
grep -E 'MemTotal|MemAvailable|SwapTotal' /proc/meminfo
df -B1 / /mnt 2>/dev/null || df -B1 /
if sudo -n fallocate -l 1G /exe-expo-swaptest 2>/dev/null && sudo mkswap /exe-expo-swaptest >/dev/null \
&& sudo swapon /exe-expo-swaptest; then
echo "swap file: works"; sudo swapoff /exe-expo-swaptest
else
echo "swap file: refused"
fi
sudo rm -f /exe-expo-swaptest

- name: Read versions
id: versions
run: grep -E '^[A-Z_]+=' config/versions.env >>"$GITHUB_OUTPUT"
Expand Down
9 changes: 5 additions & 4 deletions .github/workflows/export-mtk.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ on:
default: ""
type: string
max_chunks:
description: Maximum chunks to divide the model into (LFM2.5 at 16k built with at most one attention layer per chunk: 8 for the 1.2B, 10 for the 2.6B); empty for mtk.max_chunks in config/pipeline.yaml
description: "Maximum chunks to divide the model into (LFM2.5 at 16k built with at most one attention layer per chunk, 8 for the 1.2B and 10 for the 2.6B); empty for mtk.max_chunks in config/pipeline.yaml"
default: ""
type: string
precision:
Expand Down Expand Up @@ -50,9 +50,8 @@ concurrency:

jobs:
targets:
# Hub metadata only, no model download. ARM is fine here precisely because it never
# touches MediaTek's x86-only toolchain; it reads a config.json and does arithmetic.
runs-on: blacksmith-2vcpu-ubuntu-2404-arm
# Hub metadata only, no model download: it reads a config.json and does arithmetic.
runs-on: namespace-profile-execupack
env:
# The matrix now reads the model's config to size each window's runner, and some of
# the watched repos are private, so this job needs the token the export job has.
Expand Down Expand Up @@ -89,6 +88,8 @@ jobs:

export:
needs: targets
# An empty matrix is an error to GitHub, so a model no runner can carry stops at targets.
if: ${{ needs.targets.outputs.entries != '[]' }}
strategy:
fail-fast: false
matrix:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/export-qnn.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ concurrency:
jobs:
targets:
# Hub metadata only, no model download.
runs-on: blacksmith-2vcpu-ubuntu-2404-arm
runs-on: namespace-profile-execupack
outputs:
socs: ${{ steps.socs.outputs.socs }}
contexts: ${{ steps.contexts.outputs.contexts }}
Expand Down Expand Up @@ -87,7 +87,7 @@ jobs:
# Stays on x86: the Qualcomm backend ships only in the executorch Linux x86_64
# wheel, and the QAIRT SDK it downloads is x86_64 (this workflow checks for
# libQnnHtp.so under an x86_64 path).
runs-on: blacksmith-8vcpu-ubuntu-2404
runs-on: namespace-profile-execupack
timeout-minutes: 350
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
Expand Down
11 changes: 6 additions & 5 deletions .github/workflows/export-vulkan.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ concurrency:
jobs:
targets:
# Hub metadata only, no model download: reads config.json and does arithmetic.
runs-on: blacksmith-2vcpu-ubuntu-2404-arm
runs-on: namespace-profile-execupack
env:
# Sizing reads the model's config, and gated repos (Llama 3.2) need the token.
HF_TOKEN: ${{ secrets.HF_TOKEN }}
Expand All @@ -60,16 +60,17 @@ jobs:

export:
needs: targets
# An empty matrix is an error to GitHub, so a model no runner can carry stops at targets.
if: ${{ needs.targets.outputs.entries != '[]' }}
strategy:
fail-fast: false
matrix:
entry: ${{ fromJSON(needs.targets.outputs.entries) }}
# Sized per window by the targets job: the export peak grows with the window squared
# (every attention layer holds a window x window causal mask), so Qwen3-4B needs ~33 GiB
# at 2k and ~108 GiB at 32k. The old fixed 8vcpu (32 GB + 24 GiB swap) could not carry
# any 32k window of these families but Llama-3.2-1B's.
# Exports stay on x86: the 128 GB tier exists only there (ARM tops out at 96 GB), and the
# runner has no GPU, so the file gets a structural check either way (findings 28, 30).
# at 2k and ~108 GiB at 32k. A window no tier in vulkan.runner_tiers can carry is left
# out; since 2026-10-07 that tier is the 15.7 GiB Namespace profile, with no swap. The
# runner has no GPU, so the file gets a structural check (findings 28, 30).
runs-on: ${{ matrix.entry.runner }}
timeout-minutes: 350
env:
Expand Down
30 changes: 16 additions & 14 deletions .github/workflows/export-xnnpack.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ concurrency:
jobs:
targets:
# Hub metadata only, no model download: reads config.json and does arithmetic.
runs-on: blacksmith-2vcpu-ubuntu-2404-arm
runs-on: namespace-profile-execupack
env:
# Sizing reads the model's config, and gated repos (Llama 3.2) need the token.
HF_TOKEN: ${{ secrets.HF_TOKEN }}
Expand All @@ -67,10 +67,12 @@ jobs:
echo "$entries"
echo "entries=$entries" >>"$GITHUB_OUTPUT"
echo "contexts=$(python3 -c 'import json,sys; print(json.dumps([e["context"] for e in json.loads(sys.argv[1])]))' "$entries")" >>"$GITHUB_OUTPUT"
# The solve holds the fp32 model (16 GB for a 4B) and one layer's Hessians. Every
# solve so far, up to LFM2.5-2.6B, fit the 24 GB size; past 3B it gets the 48 GB one.
# The solve holds the fp32 model (16 GB for a 4B) and one layer's Hessians. On
# Blacksmith every solve up to LFM2.5-2.6B fit 23.4 GiB and past 3B took 48 GB. One
# Namespace profile (15.7 GiB) now runs it; the models that would not fit it have no
# window that fits an export either, so their solve is skipped with the empty matrix.
params=$(python3 -c 'import sys; from pipeline import hub; print(hub.fetch(sys.argv[1], sys.argv[2]).total_params or 0)' "$MODEL_ID" "$REVISION")
if [ "$params" -gt 3000000000 ]; then solve=blacksmith-16vcpu-ubuntu-2404-arm; else solve=blacksmith-8vcpu-ubuntu-2404-arm; fi
solve=namespace-profile-execupack
echo "solve runner: $solve ($params parameters)"
echo "solve_runner=$solve" >>"$GITHUB_OUTPUT"

Expand All @@ -79,9 +81,8 @@ jobs:
# the KV cache and the masks, never the weights. Every window's export reuses this
# artifact, which is why it is worth a job of its own.
needs: targets
if: ${{ inputs.gptq }}
# ARM like the export: the solve works on the eager PyTorch module and never loads the
# C++ runner, which is the only thing aarch64 cannot do (finding 30).
# Nothing to solve for when no window fits a runner (an empty matrix).
if: ${{ inputs.gptq && needs.targets.outputs.entries != '[]' }}
# Sized on memory: it holds the fp32 model and one layer's Hessians (targets job).
runs-on: ${{ needs.targets.outputs.solve_runner }}
timeout-minutes: 350
Expand Down Expand Up @@ -117,15 +118,16 @@ jobs:

export:
needs: [targets, solve]
if: ${{ always() && needs.targets.result == 'success' && needs.solve.result != 'failure' }}
# An empty matrix is an error to GitHub, so a model no runner can carry stops at targets.
if: ${{ always() && needs.targets.result == 'success' && needs.targets.outputs.entries != '[]' && needs.solve.result != 'failure' }}
strategy:
fail-fast: false
matrix:
entry: ${{ fromJSON(needs.targets.outputs.entries) }}
# Sized per window by the targets job: every attention layer holds a window x window
# causal mask, so the fixed 24 GB ARM runner this used to be could not carry 32k for any
# Qwen3, Qwen2.5, Llama-3.2-3B or SmolLM2 export. x86 tiers, for the 128 GB one (ARM tops
# out at 96); the export is arch-clean and the smoke test stays in its own x86 job.
# causal mask, so the need grows with the window squared, and a window no tier in
# xnnpack.runner_tiers can carry is left out. Since 2026-10-07 that tier is the 15.7 GiB
# Namespace profile (x86, no swap). The smoke test stays in its own job (verify).
runs-on: ${{ matrix.entry.runner }}
timeout-minutes: 350
env:
Expand Down Expand Up @@ -185,7 +187,7 @@ jobs:

verify:
# The smoke test, and only then the publish. ExecuTorch's runner raises IndexError: stoi
# on aarch64 (finding 30), so this is the one job that has to be x86; it loads the .pte
# on aarch64 (finding 30), so this job has to be x86, as every job is now; it loads the .pte
# the export built and asks the same question through the same C++ runner the app uses.
needs: [targets, export]
# Not `needs.export.result == 'success'`: the export matrix is fail-fast: false so that
Expand All @@ -195,12 +197,12 @@ jobs:
# verified and never published because of it. Each verify job already stands down on
# its own when its window uploaded no artifact, which is the per-window check this
# should have relied on all along.
if: ${{ always() && needs.targets.result == 'success' }}
if: ${{ always() && needs.targets.result == 'success' && needs.targets.outputs.entries != '[]' }}
strategy:
fail-fast: false
matrix:
context: ${{ fromJSON(needs.targets.outputs.contexts) }}
runs-on: blacksmith-4vcpu-ubuntu-2404
runs-on: namespace-profile-execupack
timeout-minutes: 60
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
Expand Down
28 changes: 17 additions & 11 deletions .github/workflows/probe-runner.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ on:
type: string
contexts:
description: JSON list of windows, one job each
default: "[2048, 16384]"
default: "[2048, 8192]"
type: string

permissions:
Expand All @@ -23,15 +23,11 @@ jobs:
fail-fast: false
matrix:
context: ${{ fromJSON(inputs.contexts) }}
# Sized on memory, not cores: 32 GB, because the causal masks grow with the window
# squared and have killed a 32k export before (docs/research findings 2 and 28). The
# CPU knee is not yet measured.
# Exports stay on x86 until the runner works on aarch64. The toolchain is arm64-clean and
# the export itself succeeded there, but the wheel's C++ TextLLMRunner raises
# `IndexError: stoi` on ARM, and that runner is the smoke test: without it a published
# file drops from "generated text and answered" to "the methods exist", which is the
# standard the QNN and MediaTek programs were withdrawn for (findings 28, 30).
runs-on: blacksmith-8vcpu-ubuntu-2404
# The causal masks grow with the window squared and have killed a 32k export before
# (docs/research findings 2 and 28). x86, because the wheel's C++ TextLLMRunner raises
# `IndexError: stoi` on aarch64 (finding 30). The Namespace profile has no swap, so the
# swap asked for below is not added there.
runs-on: namespace-profile-execupack
timeout-minutes: 120
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
Expand All @@ -52,7 +48,17 @@ jobs:
swap-gib: "24"

- name: Export at a forced window
run: python -m pipeline export-xnnpack "$MODEL_ID" --context "$CONTEXT" --out "$WORK_DIR/out" --work "$WORK_DIR/work"
# Exit 4 is the gate refusing a window this runner cannot hold (16k and up for
# Qwen3-0.6B on the Namespace profile): a measurement in itself, not a failure.
run: |
set +e
python -m pipeline export-xnnpack "$MODEL_ID" --context "$CONTEXT" --out "$WORK_DIR/out" --work "$WORK_DIR/work"
code=$?
if [ "$code" -eq 4 ]; then
echo "::notice::Window $CONTEXT skipped: it does not fit this runner."
exit 0
fi
exit "$code"

- name: Summary
if: always()
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/watch-hf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ concurrency:
jobs:
watch:
# Hub metadata only, no model download.
runs-on: blacksmith-2vcpu-ubuntu-2404-arm
runs-on: namespace-profile-execupack
timeout-minutes: 30
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

exe-expo exports small dense open-weight LLMs (≤4B, no MoE) from Hugging Face to ExecuTorch `.pte` files for the
[openweights](https://github.com/alpharomercoma/openweights) Android app, running entirely on hosted CI runners
(Blacksmith, Ubuntu ARM for everything except the smoke test: the solve and the export run on `blacksmith-8vcpu-ubuntu-2404-arm`, and a `verify` job on x86 runs ExecuTorch's C++ runner, which fails on aarch64, before anything publishes. See finding 30.) `docs/PLAN.md` holds the decisions, measured costs, phase status and known
(Namespace since 2026-10-07: every job runs on `namespace-profile-execupack`, 4 x86 vCPUs, 15.7 GiB of RAM and no swap, which the container refuses; a `verify` job runs ExecuTorch's C++ runner before anything publishes. The sizing tiers in `config/pipeline.yaml` are that one profile, so a window that does not fit its RAM is skipped (finding 40); a larger Namespace profile added there as a second tier brings the wider windows back.) `docs/PLAN.md` holds the decisions, measured costs, phase status and known
limits. Read it before changing sizing, backends or naming, and update it when a decision or measurement changes.

## Commands
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,17 @@

Exports small open-weight LLMs from Hugging Face to ExecuTorch `.pte` files for the
[openweights](https://github.com/alpharomercoma/openweights) Android app, on
[Blacksmith](https://www.blacksmith.sh)'s hosted CI runners: ARM for the GPTQ solve and the
XNNPACK export, x86 for the runner check before anything publishes, and x86 at 8, 16 or 32
vCPUs for MediaTek, sized per context window. Design and decisions: [docs/PLAN.md](docs/PLAN.md).
[Namespace](https://namespace.so) runners (`namespace-profile-execupack`: 4 x86 vCPUs, 16 GB,
no swap), with every window checked against that memory before a job is started. Design and decisions: [docs/PLAN.md](docs/PLAN.md).

What has shipped to [`experimentalmachines`](https://huggingface.co/experimentalmachines) on
Hugging Face, and what is built but not published:

| Backend | Status |
|---|---|
| XNNPACK (CPU) | **Published**: LFM2.5 1.2B and 2.6B (and their heretic finetunes), Qwen3, Qwen2.5, Llama 3.2, SmolLM2, at every window from 2k to 32k the runner could build. Each window is now compared with its fp32 model before it publishes. Twelve models were published before that gate existed and are being re-checked by **Audit published**; the two audited so far, SmolLM2-135M and 360M, score 0.19 and 0.22 on its prompts against about 0.58 for their fp32 models |
| MediaTek NeuroPilot (MT6991) | **Published**: LFM2.5 1.2B and 2.6B (and their heretic finetunes) at MediaTek's default 512-token window. The sizing now puts 2k to 32k within the 32-vCPU tier; a 4k export has been built and answered correctly on a Dimensity 9400, and nothing above 512 is published yet. Checked structurally on the runner, which has no NPU |
| Vulkan (GPU) | Workflow in place, same models and recipe as XNNPACK through ExecuTorch's Vulkan delegate, checked structurally (no GPU on the runner). Nothing published yet |
| MediaTek NeuroPilot (MT6991) | **Published**: LFM2.5 1.2B and 2.6B (and their heretic finetunes) from 512 up to 8k, and the heretic finetunes at 16k; Qwen3, Llama 3.2 and SmolLM2 at the windows that finished before the 2026-10-04 cancel. Built on Blacksmith runners of up to 128 GB; the Namespace profile used since 2026-10-07 cannot build LFM2.5 at any window (finding 40). Checked structurally on the runner, which has no NPU |
| Vulkan (GPU) | **Published**: Qwen3, Qwen2.5, Llama 3.2 and SmolLM2 at 2k to 32k, same recipe as XNNPACK through ExecuTorch's Vulkan delegate, checked structurally on the runner (no GPU) and run through both apps on a Mali and two Adreno GPUs (finding 38) |
| Qualcomm QNN (SM8750; SM8650 optional) | Workflow in place for the checkpoints in ExecuTorch 1.4.0's Qualcomm registry (Qwen3, Qwen2.5 base, Gemma 3 1B, SmolLM2 135M, SmolLM3 3B, Llama 3.2). Nothing published yet |
| Samsung Exynos (ENN) | Waiting for an LLM path in ExecuTorch (1.4.0 ships the delegate with CNN examples only) |
| HF watcher (auto-dispatch) | Run by hand: its hourly schedule has been off since 2026-09-19. Dispatches XNNPACK for every model first, then Vulkan, then QNN, then MediaTek; state on the `state` branch |
Expand Down Expand Up @@ -79,8 +78,9 @@ stages are 87-93% of it:
- The first MediaTek export (Qwen3-0.6B, MT6991, 512-token cache) took 1,695 s: calibration
736 s (43%), NeuroPilot lowering 472 s (28%), over 4 chunks. Its limit was calibration
memory, which held every prompt at once and pinned every window at 512. Holding one prompt at
a time moved the rest to disk, so the runner for each window is now picked by RAM and disk
together, and the sizing puts 2k to 32k within the 32-vCPU tier (4k built so far).
a time moved the rest to disk, so the runner for each window is picked by RAM and disk
together. On the Blacksmith tiers that put 2k to 32k within the 32-vCPU one; on the one
Namespace profile used since 2026-10-07 (15.7 GiB, no swap) no LFM2.5 window fits (finding 40).

## Locally

Expand Down
Loading