Skip to content

Latest commit

 

History

History
447 lines (371 loc) · 19.4 KB

host-sdk-descriptors.md

File metadata and controls

447 lines (371 loc) · 19.4 KB

Host SDK: descriptors, resources, and effects

This guide describes the current Host SDK authoring surface, where a #[pd_host_function] declaration is the single source of a function's guest schema, runtime binding, typed resource requirements, named-struct contract, and hidden host-state effects. A module then lists its functions explicitly; the guest catalog is derived from those descriptors.

1. The model

One host function declaration produces, in a single macro expansion:

Part Meaning
schema The guest ABI: parameter names, types, passing modes, return type. This is the only fingerprint input.
binding The dispatch class (Static, StaticStack, StaticStackRuntimeOwned, StaticArgs, StaticNonYieldingArgs, Owned).
adapter The concrete adapter or owned-dispatch factory installed into a registry.
effects Guest resource effects (borrow/borrow-mut/take-owned/create) and hidden host-state read/write effects. Runtime-only metadata; excluded from the fingerprint.
resource_types The concrete resource-type declarations this function contributes.

A HostModuleDescriptor aggregates functions in a deterministic, author-declared order:

use vm::HostModuleDescriptor;

pub fn demo_module() -> HostModuleDescriptor {
    HostModuleDescriptor {
        name: "demo",
        functions: &[
            make_counter_descriptor,
            read_counter_descriptor,
        ],
        resources: &[counter_resource], // extra resource declarations
    }
}

There is deliberately no linker inventory: the module lists its functions, so feature gates, ordering, WebAssembly builds, and dead-code behavior are all explicit and reproducible.

2. Authoring a module

use pd_host_function::pd_host_function;
use vm::{resource, ResourceOwned, ResourceRef, Vm, VmError, VmResult};

/// A typed host resource with a canonical declaration.
pub struct Counter(u64);

impl resource::HostResource for Counter {}

impl vm::HostResourceType for Counter {
    const KEY: &'static str = "demo.counter";
    const DESCRIPTION: &'static str = "A monotonic counter";
}

fn counter_resource() -> vm::HostResourceTypeMeta {
    vm::HostResourceTypeMeta::of::<Counter>()
}

fn counter_key() -> vm::ResourceTypeKey {
    counter_resource().schema.key
}

fn make_counter_contract() -> vm::HostFunctionSchema {
    vm::HostFunctionSchema::with_return(
        "demo::make_counter",
        vec![vm::HostParamSchema::value("seed", vm::HostTypeSchema::Int)],
        vm::HostTypeSchema::Resource(counter_key()),
    )
}

/// Creates a counter and returns its handle.
#[pd_host_function(name = "demo::make_counter", contract = make_counter_contract)]
pub fn make_counter(vm: &mut Vm, seed: i64) -> VmResult<i64> {
    let token = vm
        .host_context()
        .push_resource(Counter(seed as u64))
        .map_err(|error| VmError::HostError(error.to_string()))?;
    Ok(token.handle().raw() as i64)
}

/// Reads the current count.
#[pd_host_function(name = "demo::read_counter")]
pub fn read_counter(counter: ResourceRef<'_, Counter>) -> VmResult<i64> {
    Ok(counter.get().0 as i64)
}

/// Consumes the counter.
#[pd_host_function(name = "demo::close_counter")]
pub fn close_counter(counter: ResourceOwned<Counter>) -> VmResult<bool> {
    let _ = counter.into_inner();
    Ok(true)
}

Rules that follow from the table above:

  • Typed resource wrappers carry the effect. ResourceRef<'_, T> is a Borrow<T> effect, ResourceMut<'_, T> is BorrowMut<T>, ResourceOwned<T> is TakeOwned<T>. A creation path inserts T through HostContext, returns the raw i64 token expected by the current adapter, and declares a typed resource return contract; that contract supplies the Create<T> effect.
  • One declaration per resource type. Implement HostResourceType once. Every function that mentions T contributes that declaration, identical duplicates dedupe, and the same key claimed by a different Rust type fails before any registry mutation.
  • Resource borrows cannot share the mutable VM. A synchronous function cannot take both &mut Vm and a ResourceRef/ResourceMut parameter; the generated wrapper borrows the same VM. Use ResourceOwned, or split the work into two calls.
  • Documentation is required. Every declaration needs a doc comment.

Hidden host state

Per-VM and per-scope dependencies that guests must never see are declared as hidden parameters. They do not count toward guest arity, never appear in the schema or fingerprint, and are resolved through the generic host-state table:

#[pd_host_function(name = "re::match")]
pub fn re_match(
    cache: vm::HostStateMut<RegexCache>, // hidden: a WriteState<RegexCache> effect
    pattern: &str,
    text: &str,
) -> VmResult<bool> {
    cache.is_match(pattern, text)
}

HostStateRef<T> produces a read effect, HostStateMut<T> a write effect. Both carry a lifecycle (per VM or per scope) and an initializer (required or lazy default), which the module validates when it installs. Hidden state cannot be combined with resource parameters in one function, and cannot cross an async boundary — resolve it in a separate host call.

Named structs

Rust signatures cannot spell field names. For a fixed-shape value, implement HostNamedStruct and mark the declaration. HostNamedStruct supplies only the schema; the returned Rust value must also use the adapter's runtime conversion:

pub struct JitConfig {
    enabled: bool,
}

impl vm::HostNamedStruct for JitConfig {
    const NAME: &'static str = "JitConfig";

    fn host_struct_fields() -> Vec<vm::HostStructField> {
        vec![vm::HostStructField::new("enabled", vm::HostTypeSchema::Bool)]
    }
}

impl vm::IntoHostCallOutcome for JitConfig {
    fn into_host_call_outcome(self) -> vm::CallOutcome {
        vm::CallOutcome::Return(vm::return_one(vm::Value::map(vec![(
            vm::Value::string("enabled"),
            vm::Value::Bool(self.enabled),
        )])))
    }
}

/// Returns the current JIT configuration.
#[pd_host_function(name = "jit::get_config")]
#[pd_host_named_struct]
pub fn get_config() -> VmResult<JitConfig> {
    Ok(JitConfig { enabled: true })
}

The generated descriptor emits the full Named { name, fields } schema and the module catalog derives the named struct from it. The adapter has a blanket IntoHostCallOutcome implementation for built-in scalar and container outputs that implement IntoVmValue. The public SDK re-exports IntoHostCallOutcome, so a custom named Rust type implements that trait and returns its map-shaped Value explicitly, as shown. Do not model public host request/result/event values as Map(unknown) or unknown; the tests/typed_host_no_dynamic_contract_tests.rs guard fails the build if a public standard function regresses to a dynamic schema.

3. Async functions and declared contracts

Standard I/O and SQLite use ordinary macro-generated async host functions. A function may keep an internal raw i64 execution-scope token while its explicit contract exposes a typed guest resource. Put that contract on the same function instead of duplicating a catalog entry.

Every value that crosses an .await must be owned. Implement CaptureAsyncHostContext to copy or clone the required VM state before the VM borrow ends, and use owned parameters such as String, VmMap, or VmArrayHandle. The current io::open shape is representative (body details are abbreviated):

#[derive(Clone)]
pub(crate) struct IoPolicyContext {
    policy: Option<IoPolicy>,
}

impl vm::CaptureAsyncHostContext for IoPolicyContext {
    fn capture(vm: &mut Vm) -> VmResult<Self> {
        Ok(Self { policy: io_policy(vm) })
    }
}

#[pd_host_function(name = "io::open", contract = super::io_open_contract)]
pub(crate) async fn builtin_io_open(
    #[pd_host_context] context: IoPolicyContext,
    path: String,
    mode: String,
) -> VmResult<vm::HostFutureOutput<Value>> {
    // Mode validation and `OpenOptions` setup are omitted here.
    let path = authorize_io_path(context.policy.as_ref(), &path, writes).await?;
    let file = match options.open(path).await {
        Ok(file) => file,
        // A recoverable failure is the contract's `Err` branch, never a VM fault.
        Err(error) => return Ok(io_error_output("io_open", error)),
    };
    let handle = IoHandle::File(BufReader::new(file));
    Ok(vm::HostFutureOutput::complete(move |vm| {
        publish_open_handle(vm, handle, super::io_open_contract().return_type)
    }))
}

A contract may also declare a fallible return. io::open, io::popen, io::read_all, io::read_line, io::write, io::flush and io::close declare Result<T, string>, where only ordinary file-system outcomes take the Err branch: invalid or closed handles, ownership/scope errors, policy denials, byte limits, unsupported modes and capability rejections stay fatal VmErrors.

An owned resource cannot cross the future boundary inside a GuestResult (the macro rejects it), so the async adapter returns the assembled guest carrier through vm::HostFutureOutput<Value> and publishes it with crate::result::publish_owned, which inserts the resource, validates the carrier against the declared contract and rolls the insertion back on failure. That path is specific to the successful branch: it is the one that must agree with io::open / io::popen's Ok(resource<io.file>) outcome. The Err branches (an unopenable path, a process that cannot be started) are plain Value strings built by the adapter itself, so they carry no resource and are never routed through publish_owned — there is nothing to insert or roll back. They are not unchecked, though: every return, including these, still passes the generic validate_host_call_return check (src/vm/host.rs:2141 → validate_host_value) when the operation completes, which validates the value against the declared Result schema. What they deliberately skip is resource ownership bookkeeping, not schema validation.

io::read_line's success payload is the line without its terminator: the adapter strips one trailing \n, so a file ending in \r\n leaves the \r in the payload. End of file is Ok(None), and a present but empty line is Ok(Some("")); the two are never conflated.

When context depends on a resource-token argument, implement capture_with_args(vm, args) to validate the token and clone the owned resource state before submission. IoHandleContext, SqliteConnectionContext, and SqliteCloseContext use that form; the raw token parameter remains in the Rust signature for contract binding but is not borrowed across suspension.

The generated wrapper captures #[pd_host_context] and converts ordinary parameters before creating the future. It then calls Vm::submit_host_future, which assigns an operation id and submits the boxed future to the embedding's generic HostAsyncBridge. The bridge owns polling on its executor; the VM tracks that submitted operation and resolves the result through the declared contract. The embedding must keep the bridge's executor and runtime alive and driven until its submitted futures finish. Process-backed I/O also requires a live Tokio runtime while close or reset cleanup is polled, as detailed below.

Owned callable schemas, including bare function types such as VmCallable<fn(Event) -> Action>, are accepted. Use HostFutureOutput::complete when the resolved future must briefly re-enter the VM to insert or retire a resource. When an async opening phase must hand control to generic VM continuation machinery, return HostFutureOutput<T> and use HostFutureOutput::continue_with; value mapping preserves that continuation. The HTTP SSE builtin uses this only to transfer an opened Hyper response into the generic callable-stream driver.

Tokio runtime lease for process-backed IO

With the async feature, io::popen and io::close use Tokio process and pipe types exclusively. The embedding Tokio runtime that spawns a process-backed IO resource must remain alive and actively driven until io::close completes or a VM reset has been polled to quiescence. This includes Windows process pipes: do not shut the runtime down while a process pipe operation or process cleanup is still active.

Process close and reset cleanup must be polled while a live Tokio runtime is current. If reset cleanup is polled outside such a context, it returns the retryable resource error runtime_required immediately. The child, its pipes, the closing execution scope, and the pending reset remain owned by the VM; the VM cannot be reused. Re-enter and drive the runtime, then poll the same reset again. Vm::clear_async_bridge also rejects an execution scope with live resources or operations, and rejects a pending reset, so the bridge can be released only after scope quiescence.

Vm, process IO resource, and process IO handle Drop paths are strictly nonblocking. They issue only immediate best-effort direct-child kill requests and, on Unix, a process-group signal. They do not poll cleanup futures, wait for the leader, or run Windows tree termination. Abrupt Drop therefore has no eventual-reap guarantee. Embeddings that require deterministic tree termination and leader reaping must finish explicit io::close or reset while the runtime lease is valid.

What the contract does and does not change:

  • The contract replaces only the guest schema. The adapter, binding class, async submission, and host-state effects still come from the one macro expansion — there is no second registration path to keep in sync.
  • Guest resource effects are derived from the contract schema, so the raw token signature cannot drift from what the guest sees.
  • The contract's declared name is validated against the function name at construction; a renamed function cannot silently keep a stale contract.
  • Resource declarations stay with the module: implement HostResourceType once for the concrete type and list it in HostModuleDescriptor::resources. Then the resource key, its description, and its Rust type identity have exactly one source. The current I/O module follows this rule with io_file_key(), which returns io_file_resource().schema.key; contract code should likewise obtain the key from its canonical resource factory instead of repeating a key string.

4. Installing a module

pub struct DemoExtension;

impl vm::HostExtension for DemoExtension {
    fn register(&self, registry: &mut vm::HostFunctionRegistry) -> VmResult<()> {
        demo_module().install(registry).map(|_| ())
    }

    fn install(&self, vm: &mut vm::Vm) {
        demo_module()
            .install_state_requirements(vm)
            .expect("descriptor state requirements install");
    }
}

Installation is transactional and fail-closed:

  • The whole module is validated first — function schemas, resource declarations, named-struct bodies, binding/adapter agreement, and hidden host-state requirements. A later failure rolls back every adapter the call installed.
  • Conflicting resource keys or conflicting state providers fail before the registry changes at all.
  • install_from_catalog(registry, catalog) is the descriptor path for embedders that compose their own catalog or a subcatalog: every descriptor must match exactly one import in the supplied snapshot (same name, labels, schemas, passing modes, return type), the adapters stay the module's own, and each installed import is granted its capability. A registry built with HostFunctionRegistry::restricted() keeps denying every import the module does not declare.

5. Owned dispatch

Functions whose arguments are transferred rather than borrowed (timer callbacks are the standard example) use owned dispatch. Descriptors express it with HostBindingKind::Owned and a 'static factory:

struct RegisterTimerFactory { repeating: bool }

impl vm::HostOwnedAdapterFactory for RegisterTimerFactory {
    fn create(&self, context: vm::OwnedHostContext<'_>) -> Box<dyn vm::HostOwnedFunction> {
        Box::new(RegisterTimer { registry: context.registry().clone(), repeating: self.repeating })
    }
}

static TIMER_AT_FACTORY: RegisterTimerFactory = RegisterTimerFactory { repeating: false };

OwnedHostCall::take_arg(index) drains and transfers the operand; the factory receives the immutable registry and binding configuration it needs to spawn an isolated owned-value execution VM. The owned path keeps its guarantees: exact operand draining, callback provenance, isolated callback VM, and transactional owned-resource transfer. Do not route an owned function through borrowed or static dispatch.

Runtime-owned pending dispatch

A stack adapter whose call suspends — the HTTP request and SQLite statement families are the standard examples — resolves its pending operation from the generic VM operation/stream registries (submitted host operations and stream drivers) instead of from a per-function operation driver. Descriptors express that with HostBindingKind::StaticStackRuntimeOwned and the stack adapter:

HostFunctionDescriptor {
    schema: http_request_contract(),
    binding: HostBindingDescriptor { kind: HostBindingKind::StaticStackRuntimeOwned },
    effects: vec![],
    adapter: HostAdapterDescriptor::StaticStackRuntimeOwned(builtin_http_client_request),
    resource_types: vec![],
}

The adapter is a StaticHostStackFunction, exactly like StaticStack: the two classes register identically. The difference is the install step — after the import registers, the module install path calls HostFunctionRegistry::mark_exact_runtime_owned_pending(&name), which fails closed for a name that is not registered:

cannot mark unregistered host import '<name>' as runtime-owned

The marker records no per-import state: submitted operations are keyed by their reserved host-operation id, so there is no parallel op-driver table to maintain. Choose StaticStackRuntimeOwned only when the call's pending operation is owned by those generic registries. A module that schedules its own concrete operation driver (the io::* family) keeps plain StaticStack, and an owned-dispatch function that transfers its operands keeps Owned.

6. Aggregation and feature gates

A build composes modules through one deterministic list. The standard library's aggregation is the reference implementation (src/builtins/runtime/host_modules.rs):

let mut modules: Vec<StandardHostModule> = vec![ /* always-present modules */ ];
#[cfg(all(feature = "http-client", not(target_family = "wasm")))]
modules.push(super::http::http_host_module());
modules.push(super::io::io_host_module());
#[cfg(all(feature = "sqlite", not(target_family = "wasm")))]
modules.push(super::sqlite_schema::sqlite_standard_host_module());
modules.push(super::timer::timer_host_module());
modules.sort_by_key(|module| module.name);

Each module declares two things:

  • catalog — the guest catalog surface it publishes. Derive it from the module's descriptors; a module with no guest surface returns an empty surface.
  • owned — every descriptor the module owns, including functions still dispatched through the generated namespaced-builtin path.

tests/standard_host_descriptor_arch_tests.rs checks the composed descriptors and catalogs directly: module names are unique and deterministically ordered, every published function belongs to its module's ownership list, and the merged catalog preserves the expected module and aggregate fingerprints. It also checks that resource declarations and named structs follow the selected modules, public returns stay typed, and native-only HTTP and SQLite surfaces do not appear when their build gates are off.