diff --git a/AGENTS.md b/AGENTS.md index 301d223f..2118c0ca 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -200,8 +200,8 @@ in the same change that adds, removes or renames a folder it lists. apart from site or development additions. - **model-local engine configuration**: one model's launch recipe on one engine (artifact bindings, approved engine artifacts, request kinds, sizing, hardware support). -- **model profile**: a preset of defaults for one engine configuration; never holds total context - or context per request. +- **model profile**: a preset of defaults and engine parameter visibility overrides for one engine + configuration; never holds a total context or context per request value. - **engine release**: one version of an engine (llama.cpp, vLLM) and the parameters it exposes. An **engine artifact** is one runnable build of it. - **hardware class**: a kind of execution hardware, nothing model-specific. **Hardware support** is diff --git a/apps/comfyui/AGENTS.md b/apps/comfyui/AGENTS.md index bdd32d79..7121c87a 100644 --- a/apps/comfyui/AGENTS.md +++ b/apps/comfyui/AGENTS.md @@ -4,8 +4,8 @@ * [`Dockerfile`](Dockerfile) builds the upstream source at `COMFYUI_REV` in [`docker-image.nix`](docker-image.nix). [`entrypoint.sh`](entrypoint.sh) prepares the deployment's - storage, which mounts on ComfyUI's base directory. The shared models storage mounts on the base - directory's models folder. + storage, which mounts on ComfyUI's base directory. The models storage, shared by the app or per + deployment, mounts on the base directory's models folder. * The download node in [`custom_nodes/dotcore_model_downloads`](custom_nodes/dotcore_model_downloads) decides which missing-model downloads are allowed. Read it before changing download behavior. It ships in the image, outside the storage, so each image version brings its own copy. @@ -36,7 +36,7 @@ frontend still handing the dialog's downloads to it. 2. Gated, on an NVIDIA host: give the Docker application execution target the accelerator device, deploy ComfyUI, and open the endpoint the console reports. -3. Place a small checkpoint in the shared models storage, then queue one workflow from the editor. +3. Place a small checkpoint in the models storage, then queue one workflow from the editor. 4. The queue completes the workflow, `/system_stats` reports the accelerator, and the deployment's inspection attributes accelerator memory to its service. The placed checkpoint and the generated image are still on storage after a restart, and the placed checkpoint is still there diff --git a/apps/comfyui/README.md b/apps/comfyui/README.md index 49650be0..dc1fc6d1 100644 --- a/apps/comfyui/README.md +++ b/apps/comfyui/README.md @@ -8,18 +8,20 @@ video. * Open it from the console. * When a workflow needs a model you don't have, the editor offers to download it onto the server. Public models from Hugging Face and Civitai work. Models that need a login don't; download them - yourself and place them in the shared models storage. -* Downloaded models live in storage shared by every ComfyUI deployment on the node. They survive - restarts and removing a deployment, and stay until the site is reset or an operator removes them. + yourself and place them in the models storage. +* By default, downloaded models live in storage shared by every ComfyUI deployment on the node. + They survive restarts and removing a deployment, and stay until the site is reset or an operator + removes them. Choosing "This deployment" for models in Advanced keeps them with the deployment + and deletes them at teardown. * Custom nodes, inputs, outputs and user state live in the deployment's own storage. Removing the deployment deletes them. ## Good to know * It has no login, and custom nodes can run any code. Use it on a trusted network only. -* Deployments on one node share their models, so one deployment can place a model file, including - formats that run code when loaded, that another loads. Each already runs as root without a login, - so this grants no new authority. +* With shared models, deployments on one node share them, so one deployment can place a model file, + including formats that run code when loaded, that another loads. Each already runs as root without + a login, so this grants no new authority. * It uses an NVIDIA GPU when the machine has one (driver 580 or newer), and the CPU otherwise, as on a Mac. * Keep the VRAM mode setting on `auto`. The other modes need an NVIDIA GPU. diff --git a/apps/hermes/AGENTS.md b/apps/hermes/AGENTS.md index 0dc0e23d..b7e212ff 100644 --- a/apps/hermes/AGENTS.md +++ b/apps/hermes/AGENTS.md @@ -8,9 +8,10 @@ its data. Read it for what it checks and sets. Its placeholder key `dotcore-no-key` is not a secret: Hermes lists models only for a provider that has a key. * The dashboard refuses a non-loopback bind without a login, and the Chat page's WebSockets work - only behind it. The catalog sets Hermes' `basic` login through literal bindings. Its username and - password are deliberate public placeholders, shown in the catalog `notes`, the same for every - deployment; they are not a leaked secret. They set no session secret, so a restart logs users out. + only behind it. The catalog sets Hermes' `basic` login from two parameters: a username (default + `admin`) and a generated secret password, bound to the environment. Hermes hashes the password in + memory and never writes it to its configuration. They set no session secret, so a restart logs + users out. Its test reaches the login and chat through a published port, never from inside the container, which sees loopback. * The catalog declares no storage binding, so `/opt/data` is container-local on every platform. @@ -32,10 +33,10 @@ ## Validate 1. `just image-test`: against the image tagged `dotcore/hermes:dev`, the startup wrapper's - configuration and refusals, gateway readiness, the Dotcore provider selection, and the login and - Chat page WebSocket through a published port, as a browser reaches them. + configuration and refusals, gateway readiness, the Dotcore provider selection, and the login (with a + generated password) and Chat page WebSocket through a published port, as a browser reaches them. 2. Deploy Hermes with a linked inference route and open the dashboard after readiness, logging in - with the login from the console's app panel: Dotcore is the selected provider and the selectable + with the login from the deployment's sidebar: Dotcore is the selected provider and the selectable model is the route's current name. 3. Open the dashboard's Chat page and send one prompt; the backend log shows `POST /v1/chat/completions` with that route name as `model_id`. Rename the route and check the diff --git a/apps/hermes/README.md b/apps/hermes/README.md index 7bc3eb75..194f42a4 100644 --- a/apps/hermes/README.md +++ b/apps/hermes/README.md @@ -6,18 +6,16 @@ web dashboard. ## Using it * Open the dashboard from the console. Its address stays the same for the deployment's life. -* Log in with the username and password shown in the console's app panel. +* Log in with the username and password in the deployment's sidebar. The password is generated at + deploy unless you type one. * Chat with the agent on the dashboard's Chat page. * Link an inference route to make it the default model. Without one, pick a route in the dashboard before chatting. ## Good to know -* The login is the same for every deployment, so it doesn't protect the dashboard from anyone who - can reach it. * The chat runs the agent with full tool and terminal access inside its container. * Restarting the app logs you out. -* Dotcore doesn't manage its credentials or secrets. * The Hermes agent API is off. * Hermes keeps its data (sessions, memory, settings) inside the app. It is lost when the app is redeployed or recreated. diff --git a/apps/hermes/image-workflow.mjs b/apps/hermes/image-workflow.mjs index 6aaf9670..50cf7978 100644 --- a/apps/hermes/image-workflow.mjs +++ b/apps/hermes/image-workflow.mjs @@ -1,3 +1,4 @@ +import { randomInt } from "node:crypto"; import { readFileSync } from "node:fs"; import { chmod, @@ -40,10 +41,10 @@ const INVALID_STARTUPS = [ ['http://127.0.0.1:50052/"/v1', ROUTE], [INFERENCE_URL, "Fixture route"], ]; -// The login the catalog sets for the dashboard: deliberate public placeholders, the same for every -// deployment. +// The login the catalog binds for the dashboard: the username's default and a generated-style +// password (32 alphanumeric characters) that differs on every run. const LOGIN_USERNAME = "admin"; -const LOGIN_PASSWORD = "dotcore"; +const LOGIN_PASSWORD = randomAlphanumeric(32); const BASE_IMAGE = dockerfileBaseImage( readFileSync(new URL("./Dockerfile", import.meta.url), "utf8"), /^nousresearch\/hermes-agent@sha256:[a-f0-9]{64}$/, @@ -394,3 +395,13 @@ print(f"gateway running, pid {state['pid']}, dashboard 200") `; console.log(JSON.stringify(await verifyHermesImage(), null, 2)); + +/** Returns a random string of letters and digits, like a generated secret parameter. */ +function randomAlphanumeric(length) { + const alphabet = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"; + return Array.from( + { length }, + () => alphabet[randomInt(alphabet.length)], + ).join(""); +} diff --git a/apps/n8n/AGENTS.md b/apps/n8n/AGENTS.md index b28cabfe..6e4017cc 100644 --- a/apps/n8n/AGENTS.md +++ b/apps/n8n/AGENTS.md @@ -4,7 +4,7 @@ * [`docker-image.lock.json`](docker-image.lock.json) pins the upstream image; nothing is built here. * The catalog declaration owns the start command, which imports the `Dotcore Inference` credential, - the fixed owner login and n8n's settings. Read it before changing them. The command stays on one + n8n's settings and the owner login (email and password parameters; the password is bound in bcrypt form because n8n accepts only a hash). Read it before changing them. The command stays on one line because the Docker runtime rejects control characters. * Webhook and editor URLs are fixed from the node's host and port when the deployment is created. * A deployment keeps the declaration `notes` it was created with. @@ -17,7 +17,7 @@ ## Validate -1. On a Docker apps site, deploy a model and n8n, open the endpoint the console shows, and sign in. +1. On a Docker apps site, deploy a model and n8n, open the endpoint the console shows, and sign in with the sidebar login; a second deployment gets a different generated password and `admin` is refused. 2. Credentials lists one `Dotcore Inference`; an OpenAI Chat Model node using it lists the site's routes, and one execution reaches a route (backend log `POST /v1/chat/completions` with that route as `model_id`). diff --git a/apps/n8n/README.md b/apps/n8n/README.md index 8ffd1583..85e5ed44 100644 --- a/apps/n8n/README.md +++ b/apps/n8n/README.md @@ -4,7 +4,8 @@ ## Using it -* Open it from the console and sign in with the login it shows (`admin@dotcore.local` / `admin`). +* Open it from the console and sign in with the email and password in the deployment's sidebar. Set them when you deploy, or + leave the password empty to get a generated one. * A ready-made `Dotcore Inference` credential connects n8n's OpenAI nodes to your models. * In the OpenAI node's "Message a model", turn off the "Store" option. Dotcore doesn't store responses, so the request fails with it on. @@ -17,7 +18,6 @@ ## Good to know -* The login is fixed and public, so keep the app on a trusted network. * Workflows, credentials and settings live in the deployment's data. Losing it loses them. * Telemetry, version notifications and the AI Assistant are off. diff --git a/apps/open-webui/AGENTS.md b/apps/open-webui/AGENTS.md index 5ea14fa4..ae484aad 100644 --- a/apps/open-webui/AGENTS.md +++ b/apps/open-webui/AGENTS.md @@ -3,8 +3,16 @@ ## How it is built * [`docker-image.lock.json`](docker-image.lock.json) pins the upstream image; nothing is built here. -* `WEBUI_AUTH` is off. Configuration bindings set every Open WebUI environment value, persistent - configuration is off, and `OPENAI_API_BASE_URL` gets the site's inference endpoint. Data lives at +* Login is off by default, and sign-up is always off. The `require-login` parameter drives + `WEBUI_AUTH`, and only the admin email and password bindings depend on it: with login off they are + not set at all, because Open WebUI creates the admin from them even with login off and then + refuses visitors with an "existing users" error. +* With login on, Open WebUI creates the admin only when its data folder has no user, so login needs + a fresh data folder, which each deployment has. A data folder opened with login off holds a default + admin that stays valid and skips the parameters' admin. `WEBUI_ADMIN_NAME`, `WEBUI_SECRET_KEY` and + `DEFAULT_USER_ROLE` stay unbound. +* Configuration bindings set every Open WebUI environment value, persistent configuration is off, + and `OPENAI_API_BASE_URL` gets the site's inference endpoint. Data lives at `/app/backend/data`. ## Update @@ -15,4 +23,8 @@ ## Validate 1. On a Docker apps site, deploy a model and Open WebUI, open the endpoint the console shows. -2. The model list shows the site's routes; one chat returns an answer. +2. With login off (the default), the app opens straight into the chat, and the container has no + admin email or password variable. +3. Deploy again with "Require login" on: sign in with the sidebar's email and password; + unauthenticated access is refused. +4. The model list shows the site's routes; one chat returns an answer. diff --git a/apps/open-webui/README.md b/apps/open-webui/README.md index a97b4a3d..049ddf6c 100644 --- a/apps/open-webui/README.md +++ b/apps/open-webui/README.md @@ -4,11 +4,16 @@ ## Using it -* Open it from the console. It lists the site's inference routes as models. +* Open it from the console. Login is off by default, so it opens straight into the chat. It lists + the site's inference routes as models. +* To require login, turn on "Require login" when you deploy, then sign in with the email and + password in the deployment's sidebar. ## Good to know -* It has no login. Anyone who can reach it can use it. +* The admin email and password apply only when login is required. Sign-up is closed. +* The password is the admin's initial one. If you change it inside Open WebUI, the sidebar still + shows the old one. * Each start re-applies the settings Dotcore sets, so changes you make to them in the app don't last. * Chats and settings live in the deployment's data. diff --git a/backend/Cargo.lock b/backend/Cargo.lock index b3a26573..293b962d 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -288,6 +288,19 @@ version = "0.23.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5" +[[package]] +name = "bcrypt" +version = "0.19.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a0cd0bd35a28836d528d2b58ad499bc3c5641d59379421b1be9eeb0c2f2b912a" +dependencies = [ + "base64 0.23.1", + "blowfish", + "getrandom 0.4.3", + "subtle", + "zeroize", +] + [[package]] name = "bit-vec" version = "0.9.1" @@ -325,6 +338,16 @@ dependencies = [ "hybrid-array", ] +[[package]] +name = "blowfish" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "62ce3946557b35e71d1bbe07ec385073ce9eda05043f95de134eb578fcf1a298" +dependencies = [ + "byteorder", + "cipher", +] + [[package]] name = "bollard" version = "0.21.1" @@ -420,6 +443,12 @@ version = "1.25.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "95832e849adfb21180ccb6826a99da14e5d266ae5c2e668e1602cf234f153797" +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + [[package]] name = "byteorder-lite" version = "0.1.0" @@ -489,6 +518,16 @@ dependencies = [ "windows-link", ] +[[package]] +name = "cipher" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8cf2a2c93cd704877c0858356ed03480ff301ee950b43f1cbe4573b088bfa6c" +dependencies = [ + "crypto-common", + "inout", +] + [[package]] name = "clap" version = "4.6.7" @@ -981,6 +1020,7 @@ dependencies = [ "anyhow", "async-openai", "base64 0.23.1", + "bcrypt", "bollard", "bytes", "dotcore", @@ -2019,6 +2059,15 @@ dependencies = [ "rustversion", ] +[[package]] +name = "inout" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a580c13f0ffc335ef42dd3f6b1a41b726051c5e1656d9a3aa6fddb3c5dfbbf9" +dependencies = [ + "hybrid-array", +] + [[package]] name = "instability" version = "0.3.14" diff --git a/backend/catalog/src/lib.rs b/backend/catalog/src/lib.rs index 3ea5813f..623ffbf5 100644 --- a/backend/catalog/src/lib.rs +++ b/backend/catalog/src/lib.rs @@ -16,15 +16,16 @@ compile_error!("Dotcore supports only Linux and macOS targets"); use std::{collections::BTreeMap, error::Error as StdError, fmt, path::Path}; use dotcore_proto::dotcore::catalog::v1::{ - Application, EngineArtifact, EngineParameterValues, EngineRelease, HardwareClass, - HardwareClassRecommendations, HardwareRequirements, Model, ModelArtifact, ModelArtifactBinding, - ModelEngineConfiguration, ModelOptionValues, ModelProfile, UpstreamModelTemplate, + Application, EngineArtifact, EngineRelease, HardwareClass, HardwareClassRecommendations, + HardwareRequirements, Model, ModelArtifact, ModelArtifactBinding, ModelEngineConfiguration, + ModelOptionValues, ModelProfile, ParameterValues, UpstreamModelTemplate, }; #[cfg(test)] use dotcore_proto::dotcore::model::v1::ResolvedLocalModelDeploymentSnapshot; pub mod options; mod selector; +mod sensitive_names; pub mod service; pub mod shared_parameters; mod sizing; @@ -35,6 +36,7 @@ pub mod typed_parameters; mod validation; pub use selector::{SelectorRelation, selector_relation}; +pub use sensitive_names::is_sensitive_name; pub use sizing::{ ACCELERATOR_MEMORY_RESOURCE, HOST_MEMORY_RESOURCE, ScalarResourceEstimate, SizingError, SizingFitModel, @@ -44,9 +46,10 @@ use source::{CatalogSourceRecords, CatalogSourceRole, load_catalog_source}; pub use validation::CatalogDiagnostic; pub use validation::EngineDeclarationValidator; pub use validation::{ - overlay_parameter_defaults, valid_configuration_value, valid_environment_variable, - valid_pinned_hugging_face_artifact, validate_application_with_policy, - validate_parameter_values, validate_supplied_parameter_values, + is_secret_parameter, overlay_parameter_defaults, valid_configuration_value, + valid_environment_variable, valid_pinned_hugging_face_artifact, + validate_application_with_policy, validate_parameter_values, + validate_parameter_values_before_secrets, validate_supplied_parameter_values, validate_upstream_model_template, }; @@ -219,7 +222,7 @@ impl Catalog { &self, model_id: &CatalogModelId, profile_id: &str, - ) -> Option<(String, EngineParameterValues)> { + ) -> Option<(String, ParameterValues)> { let (model, profile) = self.selected_model_profile(model_id, profile_id)?; let engine_defaults = self .engine_defaults @@ -357,7 +360,7 @@ impl Catalog { configuration_id: &str, hardware_class_id: &str, binding_id: &str, - engine_parameter_values: &EngineParameterValues, + engine_parameter_values: &ParameterValues, option_values: &[ModelOptionValues], ) -> Result, SizingError> { let key = SizingKey { @@ -383,7 +386,7 @@ impl Catalog { binding_id: &str, resolved_model_artifact_id: Option<&str>, resolved_auxiliary_artifact_ids: &[String], - engine_parameter_values: &EngineParameterValues, + engine_parameter_values: &ParameterValues, option_values: &[ModelOptionValues], ) -> Result, SizingError> { let key = SizingKey { @@ -960,9 +963,8 @@ mod tests { use dotcore_domain::test_support::oci::ServiceArtifactOci as _; use dotcore_proto::dotcore::{ catalog::v1::{ - ApplicationDependencyCondition, ApplicationServiceDependency, EngineParameterValue, - EngineParameterValues, EngineRelease, UpstreamWireProtocol, UseCase, - engine_parameter_value::Value as ParameterValue, + ApplicationDependencyCondition, ApplicationServiceDependency, EngineRelease, + ParameterValue, ParameterValues, UpstreamWireProtocol, UseCase, parameter_value::Value, }, engine::v1::EngineKind, model::v1::ResolvedLocalModelDeploymentSnapshot, @@ -1089,18 +1091,19 @@ parameterSchema: parameters: - parameterId: "total-context" displayName: "Total context" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: {{ minimum: "1024", maximum: "131072", unit: "tokens", defaultValue: "32768" }} - parameterId: "context-per-request" displayName: "Context per request" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: {{ minimum: "1024", maximum: "131072", unit: "tokens" }} - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true + visibility: "PARAMETER_VISIBILITY_ADVANCED" unsignedInteger: {{ minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" }} - parameterId: "cache-mib" displayName: "Cache" + visibility: "PARAMETER_VISIBILITY_HIDDEN" unsignedInteger: {{ minimum: "0", maximum: "{cache_maximum}", unit: "MiB" }} artifacts: - artifactId: "linux-amd64" @@ -1375,10 +1378,10 @@ artifacts: "Catalog sizing evaluation: resolved local selection does not match sizing binding" ); - let expanded = EngineParameterValues { - values: vec![EngineParameterValue { + let expanded = ParameterValues { + values: vec![ParameterValue { parameter_id: "total-context".to_owned(), - value: Some(ParameterValue::UnsignedInteger(50_177)), + value: Some(Value::UnsignedInteger(50_177)), }], }; let expanded_local = resolved_fixture_snapshot("sized-weights", "nvidia", expanded); @@ -1786,7 +1789,7 @@ artifacts: "local", "nvidia", "second", - &EngineParameterValues::default(), + &ParameterValues::default(), &[], ) .expect("derivation evaluates"), @@ -1794,6 +1797,57 @@ artifacts: ); } + #[test] + fn a_profile_hides_only_parameters_with_a_default_or_a_derivation_on_every_binding() { + let hidden = r#" engineParameterVisibilityOverrides: + - { parameterId: "prompt-cache-memory-mib", visibility: "PARAMETER_VISIBILITY_HIDDEN" } +"#; + let load = |case: &str, sizing: &str, binding_sizing: &str, values: &str| { + let root = temporary_catalog_root().join(case); + create_empty_catalog_source(&root); + write_sizing_engine(&root); + write_sizing_model_with(&root, sizing, binding_sizing, &format!("{values}{hidden}")); + Catalog::load_with_policy( + &root, + OciArtifactPolicy::Development, + &ACCEPT_ENGINE_DECLARATIONS, + ) + }; + + let error = load("unset", "", "", " values: []\n") + .expect_err("a hidden parameter without a value must reject") + .to_string(); + assert!( + error.contains( + "profile \"balanced\" hides engine parameter \"prompt-cache-memory-mib\", which has no default and which sizing does not derive on every binding and hardware class" + ), + "{error}" + ); + assert!(!error.contains("omits engine parameter"), "{error}"); + + load( + "defaulted", + "", + "", + &format!(" values:\n{}", SIZING_ENGINE_CACHE_DEFAULT), + ) + .expect("a profile default gives the hidden parameter its value"); + load( + "derived", + r#" sizing: + derivedParameters: + prompt-cache-memory-mib: "4096" + estimates: + acceleratorMemory: + unit: bytes + equation: "${prompt-cache-memory-mib}" +"#, + UNPARAMETERIZED_BINDING_SIZING, + " values: []\n", + ) + .expect("sizing gives the hidden parameter its value on every binding"); + } + #[test] fn profile_defaults_overlay_the_engine_defaults_per_parameter() { let load = |profile_defaults: &str| { @@ -1883,6 +1937,90 @@ artifacts: ); } + #[test] + fn a_secret_engine_parameter_needs_no_default_and_takes_no_profile_default_or_sizing() { + let load = |sizing: &str, profile_defaults: &str| { + let root = temporary_catalog_root(); + create_empty_catalog_source(&root); + write_sizing_engine(&root); + let engine_path = root.join("engines/fixture.yaml"); + let engine = std::fs::read_to_string(&engine_path).expect("sizing engine"); + std::fs::write( + &engine_path, + engine.replace( + "artifacts:\n", + " - parameterId: \"api-key\"\n displayName: \"API key\"\n visibility: \"PARAMETER_VISIBILITY_ADVANCED\"\n secret: { generatedLength: 32, minimumLength: 16 }\nartifacts:\n", + ), + ) + .expect("sizing engine with a secret"); + write_sizing_model_with( + &root, + sizing, + UNPARAMETERIZED_BINDING_SIZING, + profile_defaults, + ); + Catalog::load_with_policy( + &root, + OciArtifactPolicy::Development, + &ACCEPT_ENGINE_DECLARATIONS, + ) + }; + let sizing = |estimate: &str, derived: &str| { + format!( + " sizing:\n{derived} estimates:\n acceleratorMemory:\n unit: bytes\n equation: \"{estimate}\"\n" + ) + }; + let defaults = format!(" values:\n{SIZING_ENGINE_CACHE_DEFAULT}"); + + let catalog = load(&sizing("${total-context}", ""), &defaults).expect("secret engine"); + assert_eq!( + catalog + .local_defaults_for(&CatalogModelId("fixture-model".to_owned()), "balanced") + .expect("profile defaults") + .1, + unsigned_values(&[ + ("prompt-cache-memory-mib", 0), + ("total-context", 32_768), + ("parallel-requests", 4), + ]) + ); + + let error = load( + &sizing("${total-context}", ""), + &format!( + "{defaults} - parameterId: \"api-key\"\n string: \"catalog-literal-key\"\n" + ), + ) + .expect_err("a profile default on a secret") + .to_string(); + assert!( + error.contains( + "profile \"balanced\" sets secret engine parameter \"api-key\", which each deployment generates or receives" + ), + "{error}" + ); + assert!(!error.contains("catalog-literal-key"), "{error}"); + + let error = load(&sizing("${total-context} + ${api-key}", ""), &defaults) + .expect_err("an estimate reading a secret") + .to_string(); + assert!(error.contains("api-key"), "{error}"); + + let error = load( + &sizing( + "${total-context}", + " derivedParameters:\n api-key: \"1\"\n", + ), + &defaults, + ) + .expect_err("a derivation of a secret") + .to_string(); + assert!( + error.contains("derived parameter \"api-key\" is not an unsigned engine parameter"), + "{error}" + ); + } + #[test] fn the_linearity_guard_reports_sizing_that_bends_with_the_total_context() { let guard = |case: &str, equation: &str, allowance: &str| { @@ -2510,6 +2648,53 @@ useCases: fs::remove_dir_all(root).expect("Catalog fixture cleanup"); } + #[test] + fn only_the_master_catalog_rejects_literals_on_sensitive_variables() { + let root = temporary_catalog_root(); + let master = root.join("master"); + let layer = root.join("layer"); + create_empty_catalog_source(&master); + create_empty_catalog_source(&layer); + let mut application = super::test_support::application(); + let binding = + &mut application.graph.as_mut().expect("graph").services[0].configuration_bindings[0]; + binding.environment_variable = "FIXTURE_TOKEN".to_owned(); + binding.source = Some( + dotcore_proto::dotcore::catalog::v1::application_configuration_binding::Source::LiteralValue( + "fixture-literal".to_owned(), + ), + ); + write_application(&layer, "layer.yaml", &application); + + let development = + Catalog::load_development_sources(&master, &layer, &ACCEPT_ENGINE_DECLARATIONS) + .expect("a development catalog may bind a literal to a sensitive variable"); + let site = Catalog::load_sources_with_policy( + &master, + &layer, + OciArtifactPolicy::Development, + &ACCEPT_ENGINE_DECLARATIONS, + ) + .expect("a site catalog may bind a literal to a sensitive variable"); + write_application(&master, "master.yaml", &application); + let error = Catalog::load_with_policy( + &master, + OciArtifactPolicy::Development, + &ACCEPT_ENGINE_DECLARATIONS, + ) + .expect_err("the master catalog must not bind a literal to a sensitive variable") + .to_string(); + + assert_eq!(development.list_application_releases().len(), 1); + assert_eq!(site.list_application_releases().len(), 1); + assert_eq!( + error, + "Catalog diagnostics: applications/master.yaml: application \"fixture-application\" service \"web\" configuration binding \"fixture-flag\" sets a catalog value on sensitive variable \"FIXTURE_TOKEN\"; bind a secret parameter instead" + ); + assert!(!error.contains("fixture-literal")); + fs::remove_dir_all(root).expect("Catalog fixture cleanup"); + } + #[test] fn complete_validation_prevents_partial_catalog_publication() { let valid = super::test_support::application(); @@ -2575,9 +2760,7 @@ useCases: .expect("parameter defaults") .values[0] .value = Some( - dotcore_proto::dotcore::catalog::v1::engine_parameter_value::Value::UnsignedInteger( - 1_000_000, - ), + dotcore_proto::dotcore::catalog::v1::parameter_value::Value::UnsignedInteger(1_000_000), ); model.profiles.push(duplicate); @@ -2910,18 +3093,19 @@ parameterSchema: parameters: - parameterId: "total-context" displayName: "Total context" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: {{ minimum: "1024", maximum: "131072", unit: "tokens", defaultValue: "32768" }} - parameterId: "context-per-request" displayName: "Context per request" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: {{ minimum: "1024", maximum: "131072", unit: "tokens" }} - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true + visibility: "PARAMETER_VISIBILITY_ADVANCED" unsignedInteger: {{ minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" }} - parameterId: "cache-mib" displayName: "Cache" + visibility: "PARAMETER_VISIBILITY_HIDDEN" unsignedInteger: {{ minimum: "0", maximum: "{cache_maximum}", unit: "MiB" }} artifacts: - artifactId: "{id}-artifact" @@ -3011,13 +3195,13 @@ profiles: ) } - fn unsigned_values(values: &[(&str, u64)]) -> EngineParameterValues { - EngineParameterValues { + fn unsigned_values(values: &[(&str, u64)]) -> ParameterValues { + ParameterValues { values: values .iter() - .map(|(parameter_id, value)| EngineParameterValue { + .map(|(parameter_id, value)| ParameterValue { parameter_id: (*parameter_id).to_owned(), - value: Some(ParameterValue::UnsignedInteger(*value)), + value: Some(Value::UnsignedInteger(*value)), }) .collect(), } @@ -3035,7 +3219,7 @@ profiles: fn resolved_fixture_snapshot( model_artifact_id: &str, hardware_class_id: &str, - engine_parameter_values: EngineParameterValues, + engine_parameter_values: ParameterValues, ) -> ResolvedLocalModelDeploymentSnapshot { ResolvedLocalModelDeploymentSnapshot { model_id: "fixture-model".to_owned(), diff --git a/backend/catalog/src/sensitive_names.rs b/backend/catalog/src/sensitive_names.rs new file mode 100644 index 00000000..229edda4 --- /dev/null +++ b/backend/catalog/src/sensitive_names.rs @@ -0,0 +1,73 @@ +//! Recognizes names that mark a value as secret, such as an environment variable or a launch flag. +//! +//! A name is split into words on `-` and `_` and compared without regard to case, so a sensitive +//! word matches only as a whole word: an API token variable matches, a tokenizer path does not. +//! Callers use it to mask values they record and to refuse literal values where a secret belongs. + +/// Words, or runs of words joined by `_`, whose presence marks a name as carrying a secret. +// `TOKENS` is left out: engine flags such as a batch's maximum tokens count tokens and carry no +// secret. +const SENSITIVE_NAMES: [&str; 9] = [ + "TOKEN", + "SECRET", + "SECRETS", + "PASSWORD", + "PASSWORDS", + "API_KEY", + "API_KEYS", + "CREDENTIAL", + "CREDENTIALS", +]; + +/// Returns whether `name` contains a sensitive name as whole words. +pub fn is_sensitive_name(name: &str) -> bool { + let words = name + .split(['-', '_']) + .map(str::to_ascii_uppercase) + .collect::>(); + SENSITIVE_NAMES.iter().any(|sensitive| { + let sensitive_words = sensitive.split('_').collect::>(); + words.windows(sensitive_words.len()).any(|window| { + window + .iter() + .map(String::as_str) + .eq(sensitive_words.iter().copied()) + }) + }) +} + +#[cfg(test)] +mod tests { + use super::is_sensitive_name; + + #[test] + fn sensitive_names_match_whole_words_in_any_case() { + for name in [ + "hf_token", + "HF_TOKEN", + "MY_API_KEY_FILE", + "api-key", + "N8N_BASIC_AUTH_PASSWORD", + "CLIENT_SECRET", + "aws-credential", + "GOOGLE_APPLICATION_CREDENTIALS", + "OPENAI_API_KEYS", + "DB_PASSWORDS", + "app-secrets", + ] { + assert!(is_sensitive_name(name), "{name}"); + } + for name in [ + "TOKENIZER_PATH", + "max-num-batched-tokens", + "CUDA_VISIBLE_DEVICES", + "API_URL", + "KEY", + "PASSWORDLESS", + "KEYS", + "", + ] { + assert!(!is_sensitive_name(name), "{name}"); + } + } +} diff --git a/backend/catalog/src/shared_parameters.rs b/backend/catalog/src/shared_parameters.rs index f7db621d..d9337ec0 100644 --- a/backend/catalog/src/shared_parameters.rs +++ b/backend/catalog/src/shared_parameters.rs @@ -1,15 +1,15 @@ //! The engine parameters every engine release shares: total context, context per request and //! parallel requests. //! -//! The catalog owns their ids and their fixed shape, so controllers, adapters and the console -//! name them the same way on every engine, while each release sets its own maxima, default values +//! The catalog owns their ids and their fixed shape and visibility, so controllers, adapters and +//! the console name and place them the same way on every engine, while each release sets its own maxima, default values //! and descriptions. It also fills the context per request a deployment leaves out, from the //! maximum context and the total context. use dotcore_proto::dotcore::catalog::v1::{ - EngineParameterDefinition, EngineParameterSchema, EngineParameterValue, EngineParameterValues, - Model, ModelEngineConfiguration, engine_parameter_definition::ParameterType, - engine_parameter_value::Value, + EngineParameterSchema, Model, ModelEngineConfiguration, ParameterDefinition, ParameterValue, + ParameterValues, ParameterVisibility, parameter_definition::ParameterType, + parameter_value::Value, }; /// Id of the total context: the KV pool an engine holds at start, shared by active requests. @@ -49,9 +49,9 @@ pub fn maximum_context_tokens(model: &Model, configuration: &ModelEngineConfigur /// values already supply one, or they hold no total context to bound it. pub fn filled_context_per_request( schema: &EngineParameterSchema, - values: &EngineParameterValues, + values: &ParameterValues, maximum_context_tokens: u32, -) -> Option { +) -> Option { let declared = schema .parameters .iter() @@ -73,7 +73,7 @@ pub fn filled_context_per_request( _ => None, } })?; - Some(EngineParameterValue { + Some(ParameterValue { parameter_id: CONTEXT_PER_REQUEST_PARAMETER_ID.to_owned(), value: Some(Value::UnsignedInteger( total_context.min(u64::from(maximum_context_tokens)), @@ -90,6 +90,31 @@ pub(crate) fn profile_may_default(parameter_id: &str) -> bool { .is_none_or(|shared| shared.profile_default) } +/// Whether the parameter is fixed in the main view: the context parameters are, and may only be +/// hidden there; an engine or a profile may show any other parameter there too. +pub(crate) fn in_main_view(parameter_id: &str) -> bool { + SHARED_PARAMETERS.iter().any(|shared| { + shared.parameter_id == parameter_id && shared.visibility == ParameterVisibility::Basic + }) +} + +/// Returns why a profile may not give the parameter this visibility, or none when it may. The +/// context parameters belong to the main view: a profile may hide them but not move them to the +/// advanced view. A profile may give any other parameter any visibility. +pub(crate) fn profile_visibility_problem( + parameter_id: &str, + visibility: ParameterVisibility, +) -> Option<&'static str> { + let main_view = in_main_view(parameter_id); + match visibility { + ParameterVisibility::Unspecified => Some("requires a basic, advanced or hidden visibility"), + ParameterVisibility::Advanced if main_view => Some("may be hidden but not advanced"), + ParameterVisibility::Basic + | ParameterVisibility::Advanced + | ParameterVisibility::Hidden => None, + } +} + /// Checks that an engine parameter schema declares every shared parameter with its fixed shape, /// returning one error per problem. Engine payloads are left to the engine's own validation. pub(crate) fn validate_shared_parameters(schema: &EngineParameterSchema) -> Vec { @@ -125,6 +150,7 @@ const SHARED_PARAMETERS: [SharedParameter; 3] = [ display_name: "Total context", unit: "tokens", minimum: MINIMUM_CONTEXT_TOKENS, + visibility: ParameterVisibility::Basic, engine_default: true, profile_default: false, }, @@ -133,6 +159,7 @@ const SHARED_PARAMETERS: [SharedParameter; 3] = [ display_name: "Context per request", unit: "tokens", minimum: MINIMUM_CONTEXT_TOKENS, + visibility: ParameterVisibility::Basic, engine_default: false, profile_default: false, }, @@ -141,17 +168,20 @@ const SHARED_PARAMETERS: [SharedParameter; 3] = [ display_name: "Parallel requests", unit: "requests", minimum: 1, + visibility: ParameterVisibility::Advanced, engine_default: true, profile_default: true, }, ]; -/// One shared parameter's fixed shape. Every shared parameter is an exposed unsigned integer. +/// One shared parameter's fixed shape. Every shared parameter is an unsigned integer the console +/// shows: the contexts in its main view, the parallel requests in its advanced view. struct SharedParameter { parameter_id: &'static str, display_name: &'static str, unit: &'static str, minimum: u64, + visibility: ParameterVisibility, /// Whether every release must declare an engine default; otherwise none may. engine_default: bool, @@ -162,15 +192,15 @@ struct SharedParameter { impl SharedParameter { /// Returns how a declared definition departs from the fixed shape, or none when it matches. - fn shape_problem(&self, definition: &EngineParameterDefinition) -> Option { + fn shape_problem(&self, definition: &ParameterDefinition) -> Option { let Some(ParameterType::UnsignedInteger(range)) = definition.parameter_type.as_ref() else { return Some("must be an unsigned integer".to_owned()); }; if definition.display_name != self.display_name { return Some(format!("must be named {:?}", self.display_name)); } - if !definition.exposed { - return Some("must be exposed".to_owned()); + if definition.visibility() != self.visibility { + return Some(format!("must be {}", visibility_name(self.visibility))); } if range.unit != self.unit { return Some(format!("must be in {:?}", self.unit)); @@ -186,9 +216,19 @@ impl SharedParameter { } } +/// Returns the visibility's lowercase name, for messages. +fn visibility_name(visibility: ParameterVisibility) -> &'static str { + match visibility { + ParameterVisibility::Unspecified => "unspecified", + ParameterVisibility::Basic => "basic", + ParameterVisibility::Advanced => "advanced", + ParameterVisibility::Hidden => "hidden", + } +} + #[cfg(test)] mod tests { - use dotcore_proto::dotcore::catalog::v1::UnsignedIntegerEngineParameterSchema; + use dotcore_proto::dotcore::catalog::v1::UnsignedIntegerParameterSchema; use super::*; @@ -214,7 +254,7 @@ mod tests { #[test] fn a_shared_parameter_in_the_wrong_shape_is_rejected() { - let cases: [(&str, ShapeChange, &str); 7] = [ + let cases: [(&str, ShapeChange, &str); 9] = [ ( TOTAL_CONTEXT_PARAMETER_ID, |definition| { @@ -229,8 +269,18 @@ mod tests { ), ( PARALLEL_REQUESTS_PARAMETER_ID, - |definition| definition.exposed = false, - "must be exposed", + |definition| definition.set_visibility(ParameterVisibility::Hidden), + "must be advanced", + ), + ( + TOTAL_CONTEXT_PARAMETER_ID, + |definition| definition.set_visibility(ParameterVisibility::Advanced), + "must be basic", + ), + ( + CONTEXT_PER_REQUEST_PARAMETER_ID, + |definition| definition.set_visibility(ParameterVisibility::Hidden), + "must be basic", ), ( PARALLEL_REQUESTS_PARAMETER_ID, @@ -306,6 +356,29 @@ mod tests { assert!(profile_may_default("prompt-cache-memory-mib")); } + #[test] + fn profiles_may_hide_any_parameter_and_keep_the_contexts_basic() { + use ParameterVisibility::{Advanced, Basic, Hidden, Unspecified}; + + for parameter_id in [TOTAL_CONTEXT_PARAMETER_ID, CONTEXT_PER_REQUEST_PARAMETER_ID] { + assert_eq!(profile_visibility_problem(parameter_id, Basic), None); + assert_eq!(profile_visibility_problem(parameter_id, Hidden), None); + assert_eq!( + profile_visibility_problem(parameter_id, Advanced), + Some("may be hidden but not advanced") + ); + } + for parameter_id in [PARALLEL_REQUESTS_PARAMETER_ID, "prompt-cache-memory-mib"] { + assert_eq!(profile_visibility_problem(parameter_id, Basic), None); + assert_eq!(profile_visibility_problem(parameter_id, Advanced), None); + assert_eq!(profile_visibility_problem(parameter_id, Hidden), None); + } + assert_eq!( + profile_visibility_problem(TOTAL_CONTEXT_PARAMETER_ID, Unspecified), + Some("requires a basic, advanced or hidden visibility") + ); + } + #[test] fn the_omitted_context_per_request_is_the_smaller_of_the_maximum_and_the_total_context() { let schema = shared_schema(); @@ -340,7 +413,7 @@ mod tests { None ); assert_eq!( - filled_context_per_request(&schema, &EngineParameterValues::default(), 262_144), + filled_context_per_request(&schema, &ParameterValues::default(), 262_144), None ); } @@ -371,7 +444,7 @@ mod tests { } /// Edits one shared parameter definition away from its fixed shape. - type ShapeChange = fn(&mut EngineParameterDefinition); + type ShapeChange = fn(&mut ParameterDefinition); fn messages(schema: &EngineParameterSchema) -> Vec { validate_shared_parameters(schema) @@ -415,43 +488,45 @@ mod tests { unit: &str, minimum: u64, default_value: Option, - ) -> EngineParameterDefinition { - EngineParameterDefinition { + ) -> ParameterDefinition { + ParameterDefinition { parameter_id: parameter_id.to_owned(), display_name: display_name.to_owned(), parameter_type: Some(ParameterType::UnsignedInteger( - UnsignedIntegerEngineParameterSchema { + UnsignedIntegerParameterSchema { minimum, maximum: 16_777_216, unit: unit.to_owned(), default_value, }, )), - exposed: true, + visibility: SHARED_PARAMETERS + .iter() + .find(|shared| shared.parameter_id == parameter_id) + .map_or(ParameterVisibility::Advanced, |shared| shared.visibility) + as i32, description: String::new(), } } - fn range( - definition: &mut EngineParameterDefinition, - ) -> &mut UnsignedIntegerEngineParameterSchema { + fn range(definition: &mut ParameterDefinition) -> &mut UnsignedIntegerParameterSchema { match definition.parameter_type.as_mut() { Some(ParameterType::UnsignedInteger(range)) => range, _ => panic!("unsigned shared parameter"), } } - fn total(value: u64) -> EngineParameterValues { - EngineParameterValues { - values: vec![EngineParameterValue { + fn total(value: u64) -> ParameterValues { + ParameterValues { + values: vec![ParameterValue { parameter_id: TOTAL_CONTEXT_PARAMETER_ID.to_owned(), value: Some(Value::UnsignedInteger(value)), }], } } - fn context_per_request(value: u64) -> EngineParameterValue { - EngineParameterValue { + fn context_per_request(value: u64) -> ParameterValue { + ParameterValue { parameter_id: CONTEXT_PER_REQUEST_PARAMETER_ID.to_owned(), value: Some(Value::UnsignedInteger(value)), } diff --git a/backend/catalog/src/sizing/mod.rs b/backend/catalog/src/sizing/mod.rs index fbf74561..84e687c5 100644 --- a/backend/catalog/src/sizing/mod.rs +++ b/backend/catalog/src/sizing/mod.rs @@ -7,12 +7,11 @@ use std::collections::{BTreeMap, BTreeSet}; use dotcore_proto::dotcore::catalog::v1::{ - EngineParameterSchema, EngineParameterValues, ModelEngineConfiguration, - ModelOptionParameterValue, ModelOptionValues, - engine_parameter_definition::ParameterType as EngineParameterType, - engine_parameter_value::Value as EngineParameterValue, - model_option_parameter_definition::ParameterType as OptionParameterType, + EngineParameterSchema, ModelEngineConfiguration, ModelOptionParameterValue, ModelOptionValues, + ParameterValues, model_option_parameter_definition::ParameterType as OptionParameterType, model_option_parameter_value::Value as OptionParameterValue, + parameter_definition::ParameterType as EngineParameterType, + parameter_value::Value as EngineParameterValueKind, }; use serde::Deserialize; @@ -79,7 +78,7 @@ impl CompiledSizing { /// parameter id order. Supplied values always take precedence over a derivation. pub(super) fn derive( &self, - engine_values: &EngineParameterValues, + engine_values: &ParameterValues, option_values: &[ModelOptionValues], ) -> Result, SizingError> { let values = self.symbol_values(engine_values, option_values)?; @@ -114,7 +113,7 @@ impl CompiledSizing { /// Evaluates available results in deterministic resource-name order. pub(super) fn evaluate( &self, - engine_values: &EngineParameterValues, + engine_values: &ParameterValues, option_values: &[ModelOptionValues], ) -> Result, SizingError> { let values = self.symbol_values(engine_values, option_values)?; @@ -230,7 +229,7 @@ impl CompiledSizing { fn symbol_values( &self, - engine_values: &EngineParameterValues, + engine_values: &ParameterValues, option_values: &[ModelOptionValues], ) -> Result, SizingError> { let mut values = self @@ -534,7 +533,7 @@ impl SizingFitModel { /// Evaluates one point's constant and coefficients, or none when the estimate is unavailable. pub fn point( &self, - engine_values: &EngineParameterValues, + engine_values: &ParameterValues, option_values: &[ModelOptionValues], extra_values: &BTreeMap, ) -> Result, SizingError> { @@ -574,7 +573,7 @@ impl SizingFitModel { pub fn predict( &self, parameters: &BTreeMap, - engine_values: &EngineParameterValues, + engine_values: &ParameterValues, option_values: &[ModelOptionValues], extra_values: &BTreeMap, ) -> Result { @@ -588,7 +587,7 @@ impl SizingFitModel { /// appear in an allowance, so it needs only the point's values. pub fn allowance( &self, - engine_values: &EngineParameterValues, + engine_values: &ParameterValues, option_values: &[ModelOptionValues], extra_values: &BTreeMap, ) -> Result { @@ -603,7 +602,7 @@ impl SizingFitModel { fn values( &self, - engine_values: &EngineParameterValues, + engine_values: &ParameterValues, option_values: &[ModelOptionValues], extra_values: &BTreeMap, parameters: Option<&BTreeMap>, @@ -842,16 +841,17 @@ fn option_symbols( fn insert_engine_values( symbols: &mut BTreeMap, - values: &EngineParameterValues, + values: &ParameterValues, ) -> Result<(), SizingError> { for parameter in &values.values { match parameter.value.as_ref() { - Some(EngineParameterValue::UnsignedInteger(value)) => insert_value( + Some(EngineParameterValueKind::UnsignedInteger(value)) => insert_value( symbols, parameter.parameter_id.clone(), expression::Value::Unsigned(*value), )?, - Some(EngineParameterValue::Boolean(_)) => {} + // Sizing references integers only. + Some(EngineParameterValueKind::Boolean(_) | EngineParameterValueKind::String(_)) => {} None => { return Err(SizingError::evaluation(format!( "Engine parameter {:?} has no value", @@ -918,15 +918,15 @@ mod tests { use std::collections::{BTreeMap, BTreeSet}; use dotcore_proto::dotcore::catalog::v1::{ - BooleanEngineParameterSchema, BooleanModelOptionParameterSchema, EngineParameterDefinition, - EngineParameterSchema, EngineParameterValue, EngineParameterValues, + BooleanModelOptionParameterSchema, BooleanParameterSchema, EngineParameterSchema, ModelConfigurationOption, ModelEngineConfiguration, ModelOptionParameterDefinition, - ModelOptionParameterValue, ModelOptionValues, StringEnumModelOptionParameterSchema, - UnsignedIntegerEngineParameterSchema, UnsignedIntegerModelOptionParameterSchema, - engine_parameter_definition::ParameterType as EngineParameterType, - engine_parameter_value::Value as EngineParameterValueKind, + ModelOptionParameterValue, ModelOptionValues, ParameterDefinition, ParameterValue, + ParameterValues, ParameterVisibility, StringEnumModelOptionParameterSchema, + UnsignedIntegerModelOptionParameterSchema, UnsignedIntegerParameterSchema, model_option_parameter_definition::ParameterType as OptionParameterType, model_option_parameter_value::Value as OptionParameterValueKind, + parameter_definition::ParameterType as EngineParameterType, + parameter_value::Value as EngineParameterValueKind, }; use super::{EstimateSource, ScalarResourceEstimate, SizingSource, compile}; @@ -934,18 +934,18 @@ mod tests { fn schema() -> EngineParameterSchema { EngineParameterSchema { schema_id: "test-schema".to_owned(), - parameters: vec![EngineParameterDefinition { + parameters: vec![ParameterDefinition { parameter_id: "total-context".to_owned(), display_name: "Total context".to_owned(), parameter_type: Some(EngineParameterType::UnsignedInteger( - UnsignedIntegerEngineParameterSchema { + UnsignedIntegerParameterSchema { minimum: 1, maximum: 131_072, unit: "tokens".to_owned(), default_value: None, }, )), - exposed: true, + visibility: ParameterVisibility::Basic as i32, description: "Maximum context.".to_owned(), }], } @@ -1014,18 +1014,18 @@ mod tests { } } - fn engine_values(total_context: u64) -> EngineParameterValues { - EngineParameterValues { - values: vec![EngineParameterValue { + fn engine_values(total_context: u64) -> ParameterValues { + ParameterValues { + values: vec![ParameterValue { parameter_id: "total-context".to_owned(), value: Some(EngineParameterValueKind::UnsignedInteger(total_context)), }], } } - fn engine_values_with_boolean(total_context: u64, enabled: bool) -> EngineParameterValues { + fn engine_values_with_boolean(total_context: u64, enabled: bool) -> ParameterValues { let mut values = engine_values(total_context); - values.values.push(EngineParameterValue { + values.values.push(ParameterValue { parameter_id: "feature-enabled".to_owned(), value: Some(EngineParameterValueKind::Boolean(enabled)), }); @@ -1574,27 +1574,27 @@ mod tests { fn derived_schema() -> EngineParameterSchema { let mut schema = schema(); - schema.parameters.push(EngineParameterDefinition { + schema.parameters.push(ParameterDefinition { parameter_id: "cache-bytes".to_owned(), display_name: "Cache bytes".to_owned(), parameter_type: Some(EngineParameterType::UnsignedInteger( - UnsignedIntegerEngineParameterSchema { + UnsignedIntegerParameterSchema { minimum: 1, maximum: u64::MAX, unit: "bytes".to_owned(), default_value: None, }, )), - exposed: false, + visibility: ParameterVisibility::Hidden as i32, description: "Cache size.".to_owned(), }); - schema.parameters.push(EngineParameterDefinition { + schema.parameters.push(ParameterDefinition { parameter_id: "feature-enabled".to_owned(), display_name: "Feature".to_owned(), parameter_type: Some(EngineParameterType::Boolean( - BooleanEngineParameterSchema {}, + BooleanParameterSchema::default(), )), - exposed: false, + visibility: ParameterVisibility::Hidden as i32, description: "Feature toggle.".to_owned(), }); schema @@ -1609,8 +1609,8 @@ mod tests { declaration } - fn cache_bytes_value(value: u64) -> EngineParameterValue { - EngineParameterValue { + fn cache_bytes_value(value: u64) -> ParameterValue { + ParameterValue { parameter_id: "cache-bytes".to_owned(), value: Some(EngineParameterValueKind::UnsignedInteger(value)), } diff --git a/backend/catalog/src/source.rs b/backend/catalog/src/source.rs index 75a2ee65..f0f65c1c 100644 --- a/backend/catalog/src/source.rs +++ b/backend/catalog/src/source.rs @@ -23,7 +23,7 @@ use serde_json::{Map as JsonMap, Value as JsonValue}; use super::{ sizing::{CatalogSizingSource, ConfigurationSizingSource, SupportSizingSource}, - validation::CatalogDiagnostic, + validation::{CatalogDiagnostic, validate_sensitive_literals}, }; pub(super) fn load_catalog_source( @@ -57,7 +57,13 @@ pub(super) fn load_catalog_source( &mut records.recommendations, diagnostics, ); - load_applications(catalog_root, source, &mut records.applications, diagnostics); + load_applications( + catalog_root, + source, + role, + &mut records.applications, + diagnostics, + ); load_upstream_model_templates( catalog_root, source, @@ -297,12 +303,24 @@ fn load_recommendations( fn load_applications( catalog_root: &Path, source: &str, + role: CatalogSourceRole, applications: &mut Vec<(String, Application)>, diagnostics: &mut Vec, ) { for file in optional_catalog_files(catalog_root, source, APPLICATION_DIRECTORY, diagnostics) { match yaml::from_file(&file.path) { - Ok(application) => applications.push((file.location, application)), + Ok(application) => { + // Site and development additions belong to their operators; only the master + // catalog ships to every site. + if role == CatalogSourceRole::Master { + diagnostics.extend( + validate_sensitive_literals(&application) + .into_iter() + .map(|error| CatalogDiagnostic::new(file.location.clone(), error)), + ); + } + applications.push((file.location, application)); + } Err(error) => diagnostics.push(CatalogDiagnostic::new( file.location, anyhow::Error::new(error), diff --git a/backend/catalog/src/test_support.rs b/backend/catalog/src/test_support.rs index 6fd2a2d2..48d8706d 100644 --- a/backend/catalog/src/test_support.rs +++ b/backend/catalog/src/test_support.rs @@ -8,10 +8,10 @@ use std::{collections::BTreeMap, fs, path::PathBuf}; use dotcore_domain::test_support::{canonical_temp_dir, unique_suffix}; use dotcore_proto::dotcore::catalog::v1::{ - Application, EngineParameterValue, EngineRelease, HardwareClass, HardwareSupport, Model, - ModelEngineConfiguration, ModelOptionParameterValue, ModelOptionValues, ModelProfile, Selector, - UpstreamModelTemplate, UpstreamWireProtocol, engine_parameter_value, - model_option_parameter_definition, model_option_parameter_value, + Application, EngineRelease, HardwareClass, HardwareSupport, Model, ModelEngineConfiguration, + ModelOptionParameterValue, ModelOptionValues, ModelProfile, ParameterValue, Selector, + UpstreamModelTemplate, UpstreamWireProtocol, model_option_parameter_definition, + model_option_parameter_value, parameter_value, }; use crate::{ @@ -290,7 +290,7 @@ pub fn write_sizing_engine(root: &std::path::Path) { parameters: - parameterId: "total-context" displayName: "Total context" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: minimum: "1024" maximum: "16777216" @@ -298,14 +298,14 @@ pub fn write_sizing_engine(root: &std::path::Path) { defaultValue: "32768" - parameterId: "context-per-request" displayName: "Context per request" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: minimum: "1024" maximum: "16777216" unit: "tokens" - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true + visibility: "PARAMETER_VISIBILITY_ADVANCED" unsignedInteger: minimum: "1" maximum: "64" @@ -313,6 +313,7 @@ pub fn write_sizing_engine(root: &std::path::Path) { defaultValue: "4" - parameterId: "prompt-cache-memory-mib" displayName: "Prompt cache memory" + visibility: "PARAMETER_VISIBILITY_HIDDEN" unsignedInteger: minimum: "0" maximum: "65536" @@ -578,9 +579,9 @@ fn choice_combinations( combinations } -fn unsigned_value(parameter_id: &str, value: u64) -> EngineParameterValue { - EngineParameterValue { +fn unsigned_value(parameter_id: &str, value: u64) -> ParameterValue { + ParameterValue { parameter_id: parameter_id.to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(value)), + value: Some(parameter_value::Value::UnsignedInteger(value)), } } diff --git a/backend/catalog/src/typed_parameters.rs b/backend/catalog/src/typed_parameters.rs index 42e1fe2a..e6c91e3c 100644 --- a/backend/catalog/src/typed_parameters.rs +++ b/backend/catalog/src/typed_parameters.rs @@ -1,20 +1,21 @@ -//! Owner-neutral typed parameters: definition validation, value checks against a definition, -//! and the canonical text form a value takes when projected into configuration. +//! Owner-neutral typed parameters: definition validation, defaults, value checks against a +//! definition, and the canonical text form a value takes when projected into configuration. //! -//! Application graphs use these definitions today; the model lane keeps its own engine and -//! option parameter schemas until it adopts this one. Owners layer their own rules on top, -//! such as whether a default is required. +//! Application graphs and engine releases declare their parameters with these definitions. Owners +//! layer their own rules on top, such as which types and visibilities they accept and whether a +//! default is required. use std::collections::BTreeSet; use anyhow::anyhow; use dotcore_domain::valid_id; use dotcore_proto::dotcore::catalog::v1::{ - ParameterDefinition, ParameterValue, parameter_definition::ParameterType, + ParameterDefinition, ParameterValue, ParameterVisibility, parameter_definition::ParameterType, parameter_value::Value, }; -/// Validates one definition's identity, label, and type constraints, including any default. +/// Validates one definition's identity, label, visibility and type constraints, including any +/// default. pub(crate) fn validate_parameter_definition( definition: &ParameterDefinition, ) -> Vec { @@ -28,6 +29,16 @@ pub(crate) fn validate_parameter_definition( if definition.display_name.trim().is_empty() { errors.push(anyhow!("parameter {id:?} requires a display name")); } + if !matches!( + ParameterVisibility::try_from(definition.visibility), + Ok(ParameterVisibility::Basic + | ParameterVisibility::Advanced + | ParameterVisibility::Hidden) + ) { + errors.push(anyhow!( + "parameter {id:?} requires a basic, advanced or hidden visibility" + )); + } match definition.parameter_type.as_ref() { Some(ParameterType::Boolean(_)) => {} Some(ParameterType::UnsignedInteger(range)) if range.minimum > range.maximum => { @@ -53,6 +64,40 @@ pub(crate) fn validate_parameter_definition( )); } } + Some(ParameterType::Text(schema)) => { + if schema.maximum_length == 0 { + errors.push(anyhow!( + "parameter {id:?} maximum length must be at least 1" + )); + } + if schema.minimum_length > schema.maximum_length { + errors.push(anyhow!( + "parameter {id:?} minimum length must not exceed its maximum length" + )); + } + if schema.maximum_length > MAXIMUM_FREE_TEXT_LENGTH { + errors.push(anyhow!( + "parameter {id:?} maximum length must not exceed {MAXIMUM_FREE_TEXT_LENGTH}" + )); + } + } + Some(ParameterType::Secret(schema)) => { + if schema.minimum_length == 0 { + errors.push(anyhow!( + "parameter {id:?} minimum length must be at least 1" + )); + } + if schema.minimum_length > schema.generated_length { + errors.push(anyhow!( + "parameter {id:?} minimum length must not exceed its generated length" + )); + } + if schema.generated_length > MAXIMUM_FREE_TEXT_LENGTH { + errors.push(anyhow!( + "parameter {id:?} generated length must not exceed {MAXIMUM_FREE_TEXT_LENGTH}" + )); + } + } None => errors.push(anyhow!("parameter {id:?} requires a type")), } if errors.is_empty() @@ -64,16 +109,24 @@ pub(crate) fn validate_parameter_definition( errors } -/// Returns the definition's declared default, if any. +/// Returns the definition's declared default, if any. A secret never has one. pub fn default_value(definition: &ParameterDefinition) -> Option { match definition.parameter_type.as_ref()? { ParameterType::Boolean(schema) => schema.default_value.map(Value::Boolean), ParameterType::UnsignedInteger(schema) => schema.default_value.map(Value::UnsignedInteger), ParameterType::StringEnum(schema) => schema.default_value.clone().map(Value::String), + ParameterType::Text(schema) => schema.default_value.clone().map(Value::String), + ParameterType::Secret(_) => None, } } -/// Checks that a value has the definition's type and lies within its range or choices. +/// Returns whether the definition declares a secret. +pub fn is_secret(definition: &ParameterDefinition) -> bool { + matches!(definition.parameter_type, Some(ParameterType::Secret(_))) +} + +/// Checks that a value has the definition's type and lies within its range, choices or length +/// bounds. Errors never quote a text or secret value. pub fn check_value(definition: &ParameterDefinition, value: &Value) -> anyhow::Result<()> { match (definition.parameter_type.as_ref(), value) { (Some(ParameterType::Boolean(_)), Value::Boolean(_)) => Ok(()), @@ -95,6 +148,12 @@ pub fn check_value(definition: &ParameterDefinition, value: &Value) -> anyhow::R Err(anyhow!("{value:?} is not one of {:?}", schema.values)) } } + (Some(ParameterType::Text(schema)), Value::String(value)) => { + check_free_text(value, schema.minimum_length, schema.maximum_length) + } + (Some(ParameterType::Secret(schema)), Value::String(value)) => { + check_free_text(value, schema.minimum_length, MAXIMUM_FREE_TEXT_LENGTH) + } _ => Err(anyhow!("value type does not match the parameter type")), } } @@ -118,10 +177,33 @@ pub fn canonical_value_text(values: &[ParameterValue], parameter_id: &str) -> Op .map(canonical_text) } +/// Longest text or secret value in characters, which keeps every value a reasonable environment +/// value; a secret declares no maximum of its own. +const MAXIMUM_FREE_TEXT_LENGTH: u32 = 1024; + +/// Checks a text or secret value's length in characters and that it is safe as an environment +/// value: no `=` and no control characters. +fn check_free_text(value: &str, minimum: u32, maximum: u32) -> anyhow::Result<()> { + if value.contains('=') || value.chars().any(char::is_control) { + return Err(anyhow!("it contains `=` or a control character")); + } + let length = value.chars().count(); + if length < minimum as usize { + return Err(anyhow!( + "it is shorter than the minimum length of {minimum}" + )); + } + if length > maximum as usize { + return Err(anyhow!("it is longer than the maximum length of {maximum}")); + } + Ok(()) +} + #[cfg(test)] mod tests { use dotcore_proto::dotcore::catalog::v1::{ - BooleanParameterSchema, StringEnumParameterSchema, UnsignedIntegerParameterSchema, + BooleanParameterSchema, SecretParameterSchema, StringEnumParameterSchema, + TextParameterSchema, UnsignedIntegerParameterSchema, }; use super::*; @@ -171,6 +253,86 @@ mod tests { ); } + #[test] + fn check_value_bounds_text_and_secrets_without_quoting_them() { + let text = definition(ParameterType::Text(TextParameterSchema { + minimum_length: 2, + maximum_length: 4, + default_value: None, + })); + let secret = definition(ParameterType::Secret(SecretParameterSchema { + generated_length: 32, + minimum_length: 3, + })); + let check = |definition: &ParameterDefinition, value: &str| { + check_value(definition, &Value::String(value.to_owned())) + .map_err(|error| error.to_string()) + }; + + assert_eq!(check(&text, "ab"), Ok(())); + assert_eq!(check(&text, "éèêë"), Ok(())); + assert_eq!( + check(&text, "a"), + Err("it is shorter than the minimum length of 2".to_owned()) + ); + assert_eq!( + check(&text, "abcde"), + Err("it is longer than the maximum length of 4".to_owned()) + ); + assert_eq!(check(&secret, &"s".repeat(1024)), Ok(())); + assert_eq!( + check(&secret, &"s".repeat(1025)), + Err("it is longer than the maximum length of 1024".to_owned()) + ); + assert_eq!( + check(&secret, ""), + Err("it is shorter than the minimum length of 3".to_owned()) + ); + for unsafe_value in ["a=bc", "ab\tc", "ab\nc"] { + assert_eq!( + check(&secret, unsafe_value), + Err("it contains `=` or a control character".to_owned()) + ); + } + assert_eq!( + check_value(&secret, &Value::Boolean(true)) + .unwrap_err() + .to_string(), + "value type does not match the parameter type" + ); + assert_eq!(default_value(&secret), None); + assert!(is_secret(&secret)); + assert!(!is_secret(&text)); + } + + #[test] + fn definition_validation_caps_text_and_generated_secret_lengths() { + assert_errors( + &definition(ParameterType::Text(TextParameterSchema { + minimum_length: 0, + maximum_length: 1025, + default_value: None, + })), + &["parameter \"mode\" maximum length must not exceed 1024"], + ); + assert_errors( + &definition(ParameterType::Secret(SecretParameterSchema { + generated_length: 1025, + minimum_length: 12, + })), + &["parameter \"mode\" generated length must not exceed 1024"], + ); + assert!( + validate_parameter_definition(&definition(ParameterType::Secret( + SecretParameterSchema { + generated_length: 1024, + minimum_length: 12, + } + ))) + .is_empty() + ); + } + #[test] fn definition_validation_rejects_invalid_shapes() { let mut bad_id = definition(ParameterType::Boolean(BooleanParameterSchema { @@ -208,6 +370,12 @@ mod tests { let mut untyped = enum_definition(&["auto"], None); untyped.parameter_type = None; assert_errors(&untyped, &["parameter \"mode\" requires a type"]); + let mut unplaced = enum_definition(&["auto"], Some("auto")); + unplaced.visibility = ParameterVisibility::Unspecified as i32; + assert_errors( + &unplaced, + &["parameter \"mode\" requires a basic, advanced or hidden visibility"], + ); assert!( validate_parameter_definition(&enum_definition(&["auto"], Some("auto"))).is_empty() ); @@ -233,6 +401,7 @@ mod tests { parameter_id: "mode".into(), display_name: "Mode".into(), description: String::new(), + visibility: ParameterVisibility::Basic as i32, parameter_type: Some(parameter_type), } } diff --git a/backend/catalog/src/validation/application.rs b/backend/catalog/src/validation/application.rs index 4a73f6cf..380a2c01 100644 --- a/backend/catalog/src/validation/application.rs +++ b/backend/catalog/src/validation/application.rs @@ -59,11 +59,12 @@ mod tests { Application, ApplicationConfigurationBinding, ApplicationDependencyCondition, ApplicationDeploymentInput, ApplicationEndpointProtocol, ApplicationGraph, ApplicationHttpReadiness, ApplicationInferenceRouteInput, - ApplicationInferenceRouteProjection, ApplicationParameter, ApplicationPrimaryEndpoint, - ApplicationSecretBinding, ApplicationService, ApplicationServiceDependency, + ApplicationInferenceRouteProjection, ApplicationPrimaryEndpoint, ApplicationSecretBinding, + ApplicationSecretForm, ApplicationService, ApplicationServiceDependency, ApplicationServicePort, ApplicationServiceReadiness, BooleanParameterSchema, NixArtifact, - NixDockerImage, NixSystemStorePath, OciPlatform, ParameterDefinition, StorageBinding, - StorageKind, StorageScope, StringEnumParameterSchema, UnsignedIntegerParameterSchema, + NixDockerImage, NixSystemStorePath, OciPlatform, ParameterDefinition, ParameterVisibility, + SecretParameterSchema, StorageBinding, StorageKind, StoragePlacement, + StringEnumParameterSchema, TextParameterSchema, UnsignedIntegerParameterSchema, application_configuration_binding, application_service_artifact::Kind as ServiceArtifactKind, nix_artifact::Provides, parameter_definition::ParameterType, @@ -119,12 +120,13 @@ mod tests { "postgres://database".to_owned(), )), input: None, + enabled_by_parameter_id: None, }], storage_bindings: vec![StorageBinding { binding_name: "data".to_owned(), mount_path: "/app/data".to_owned(), kind: StorageKind::Data as i32, - scope: StorageScope::Deployment as i32, + placements: vec![StoragePlacement::Deployment as i32], environment: Vec::new(), }], dependencies: vec![ApplicationServiceDependency { @@ -427,6 +429,7 @@ mod tests { "not=safe".to_owned(), )), input: None, + enabled_by_parameter_id: None, }); web.configuration_bindings .push(ApplicationConfigurationBinding { @@ -438,10 +441,14 @@ mod tests { ), ), input: None, + enabled_by_parameter_id: None, }); web.secret_bindings.push(ApplicationSecretBinding { binding_name: "secret".to_owned(), environment_variable: "DATABASE_URL".to_owned(), + parameter_id: "password".to_owned(), + form: ApplicationSecretForm::Plain as i32, + enabled_by_parameter_id: None, }); let errors = validate_application(&application) @@ -634,7 +641,29 @@ mod tests { ], &["mode"], ), - "application parameter \"debug\" is not used by any configuration binding", + "application parameter \"debug\" is not used by any binding", + ), + ( + parameterized_application( + vec![parameter_with_visibility( + "mode", + string_enum(&["auto"], Some("auto")), + ParameterVisibility::Hidden, + )], + &["mode"], + ), + "application parameter \"mode\" must be basic or advanced, not hidden", + ), + ( + parameterized_application( + vec![parameter_with_visibility( + "mode", + string_enum(&["auto"], Some("auto")), + ParameterVisibility::Unspecified, + )], + &["mode"], + ), + "parameter \"mode\" requires a basic, advanced or hidden visibility", ), ]; @@ -647,6 +676,201 @@ mod tests { } } + #[test] + fn application_validation_accepts_text_and_secret_parameters() { + let mut application = parameterized_application( + vec![ + parameter("username", text(1, 64, Some("admin"))), + parameter("password", secret(32, 12)), + ], + &["username"], + ); + bind_secret( + &mut application, + "password", + "password", + ApplicationSecretForm::Plain, + ); + bind_secret( + &mut application, + "password-hash", + "password", + ApplicationSecretForm::Bcrypt, + ); + + assert!(validate_application(&application).is_empty()); + } + + #[test] + fn application_validation_rejects_invalid_text_and_secret_parameters() { + let with_form_binding = + |parameters, bound_ids: &[&str], secret_id: &str, form: ApplicationSecretForm| { + let mut application = parameterized_application(parameters, bound_ids); + bind_secret(&mut application, "password", secret_id, form); + application + }; + let with_secret_binding = |parameters, bound_ids: &[&str], secret_id: &str| { + with_form_binding( + parameters, + bound_ids, + secret_id, + ApplicationSecretForm::Plain, + ) + }; + let cases = [ + ( + with_secret_binding(vec![parameter("password", secret(8, 12))], &[], "password"), + "parameter \"password\" minimum length must not exceed its generated length", + ), + ( + with_secret_binding(vec![parameter("password", secret(32, 0))], &[], "password"), + "parameter \"password\" minimum length must be at least 1", + ), + ( + parameterized_application( + vec![parameter("username", text(1, 4, Some("administrator")))], + &["username"], + ), + "parameter \"username\" default is invalid: it is longer than the maximum length of 4", + ), + ( + parameterized_application( + vec![parameter("username", text(6, 64, Some("admin")))], + &["username"], + ), + "parameter \"username\" default is invalid: it is shorter than the minimum length of 6", + ), + ( + parameterized_application( + vec![parameter("username", text(1, 64, Some("a=b")))], + &["username"], + ), + "parameter \"username\" default is invalid: it contains `=` or a control character", + ), + ( + parameterized_application( + vec![parameter("username", text(0, 0, Some("")))], + &["username"], + ), + "parameter \"username\" maximum length must be at least 1", + ), + ( + parameterized_application( + vec![parameter("username", text(1, 64, None))], + &["username"], + ), + "application parameter \"username\" requires a default", + ), + ( + with_secret_binding(vec![parameter("password", secret(32, 12))], &[], "missing"), + "application secret binding \"password\" on service \"web\" references missing parameter \"missing\"", + ), + ( + with_secret_binding( + vec![ + parameter("username", text(1, 64, Some("admin"))), + parameter("password", secret(32, 12)), + ], + &["password"], + "username", + ), + "application configuration binding \"password\" on service \"web\" references secret parameter \"password\"; secrets use secret bindings", + ), + ]; + + for (application, expected) in cases { + let errors = validate_application(&application) + .into_iter() + .map(|error| error.to_string()) + .collect::>(); + assert!( + errors.iter().any(|error| error == expected), + "expected {expected:?} in {errors:?}" + ); + } + + for form in [ApplicationSecretForm::Plain, ApplicationSecretForm::Bcrypt] { + let not_secret = with_form_binding( + vec![parameter("username", text(1, 64, Some("admin")))], + &[], + "username", + form, + ); + assert_eq!( + validate_application(¬_secret) + .into_iter() + .map(|error| error.to_string()) + .collect::>(), + [ + "application secret binding \"password\" on service \"web\" references parameter \"username\", which is not a secret" + ] + ); + } + + let mut unknown_form = + parameterized_application(vec![parameter("password", secret(32, 12))], &[]); + bind_secret( + &mut unknown_form, + "password", + "password", + ApplicationSecretForm::Plain, + ); + unknown_form.graph.as_mut().expect("graph").services[1].secret_bindings[0].form = + ApplicationSecretForm::Unspecified as i32; + assert_eq!( + validate_application(&unknown_form) + .into_iter() + .map(|error| error.to_string()) + .collect::>(), + ["application secret binding \"password\" requires a known form"] + ); + } + + #[test] + fn application_yaml_declares_secret_parameters_without_a_default() { + let graph = dotcore_proto::yaml::from_str::( + r#"parameters: + - parameterId: "password" + displayName: "password" + visibility: "PARAMETER_VISIBILITY_BASIC" + secret: + generatedLength: 32 + minimumLength: 12 +services: + - secretBindings: + - bindingName: "password" + environmentVariable: "PASSWORD" + parameterId: "password" + form: "APPLICATION_SECRET_FORM_PLAIN" +"#, + ) + .expect("secret graph decodes"); + + assert_eq!(graph.parameters, [parameter("password", secret(32, 12))]); + assert_eq!( + graph.services[0].secret_bindings, + [ApplicationSecretBinding { + binding_name: "password".to_owned(), + environment_variable: "PASSWORD".to_owned(), + parameter_id: "password".to_owned(), + form: ApplicationSecretForm::Plain as i32, + enabled_by_parameter_id: None, + }] + ); + dotcore_proto::yaml::from_str::( + r#"parameters: + - parameterId: "password" + displayName: "password" + visibility: "PARAMETER_VISIBILITY_BASIC" + secret: + generatedLength: 32 + minimumLength: 12 + defaultValue: "dotcore" +"#, + ) + .expect_err("a secret parameter cannot declare a default"); + } + #[test] fn application_validation_rejects_a_parameter_binding_with_a_route_input() { let mut application = parameterized_application( @@ -678,17 +902,85 @@ mod tests { ); } + #[test] + fn application_validation_checks_the_parameter_enabling_a_binding() { + let enabled_by = |parameter_id: &str| { + let mut application = parameterized_application( + vec![ + parameter("login", boolean(Some(false))), + parameter("mode", string_enum(&["auto"], Some("auto"))), + parameter("password", secret(32, 1)), + ], + &["login", "mode"], + ); + bind_secret( + &mut application, + "password", + "password", + ApplicationSecretForm::Plain, + ); + let web = &mut application.graph.as_mut().expect("graph").services[1]; + web.configuration_bindings + .last_mut() + .expect("mode binding") + .enabled_by_parameter_id = Some(parameter_id.to_owned()); + web.secret_bindings[0].enabled_by_parameter_id = Some(parameter_id.to_owned()); + application + }; + let errors = |application: &Application| { + validate_application(application) + .into_iter() + .map(|error| error.to_string()) + .collect::>() + }; + + assert!(errors(&enabled_by("login")).is_empty()); + assert_eq!( + errors(&enabled_by("missing")), + [ + "application configuration binding \"mode\" on service \"web\" is enabled by missing parameter \"missing\"", + "application secret binding \"password\" on service \"web\" is enabled by missing parameter \"missing\"", + ] + ); + assert_eq!( + errors(&enabled_by("mode")), + [ + "application configuration binding \"mode\" on service \"web\" is enabled by parameter \"mode\", which is not a boolean", + "application secret binding \"password\" on service \"web\" is enabled by parameter \"mode\", which is not a boolean", + ] + ); + + let mut with_route_input = enabled_by("login"); + let binding = &mut with_route_input.graph.as_mut().expect("graph").services[1] + .configuration_bindings[0]; + binding.enabled_by_parameter_id = Some("login".to_owned()); + binding.input = Some(ApplicationDeploymentInput { + kind: Some( + dotcore_proto::dotcore::catalog::v1::application_deployment_input::Kind::InferenceRoute( + ApplicationInferenceRouteInput { + projection: ApplicationInferenceRouteProjection::RouteName as i32, + }, + ), + ), + }); + assert_eq!( + errors(&with_route_input), + [ + "application configuration binding \"database-url\" on service \"web\" takes a route input, so it cannot be enabled by a parameter" + ] + ); + } + #[test] fn application_yaml_declares_parameters_and_parameter_bindings() { let graph = dotcore_proto::yaml::from_str::( r#"parameters: - - advanced: true - definition: - parameterId: "mode" - displayName: "mode" - stringEnum: - values: ["auto", "highvram"] - defaultValue: "auto" + - parameterId: "mode" + displayName: "mode" + visibility: "PARAMETER_VISIBILITY_ADVANCED" + stringEnum: + values: ["auto", "highvram"] + defaultValue: "auto" services: - configurationBindings: - bindingName: "mode" @@ -703,7 +995,7 @@ services: [parameter_with_visibility( "mode", string_enum(&["auto", "highvram"], Some("auto")), - true + ParameterVisibility::Advanced )] ); assert_eq!( @@ -747,7 +1039,7 @@ services: } fn parameterized_application( - parameters: Vec, + parameters: Vec, bound_ids: &[&str], ) -> Application { let mut application = application(); @@ -763,28 +1055,44 @@ services: (*id).to_owned(), )), input: None, + enabled_by_parameter_id: None, }); } application } - fn parameter(id: &str, parameter_type: ParameterType) -> ApplicationParameter { - parameter_with_visibility(id, parameter_type, false) + fn bind_secret( + application: &mut Application, + binding_name: &str, + parameter_id: &str, + form: ApplicationSecretForm, + ) { + application.graph.as_mut().expect("graph").services[1] + .secret_bindings + .push(ApplicationSecretBinding { + binding_name: binding_name.to_owned(), + environment_variable: binding_name.to_uppercase().replace('-', "_"), + parameter_id: parameter_id.to_owned(), + form: form as i32, + enabled_by_parameter_id: None, + }); + } + + fn parameter(id: &str, parameter_type: ParameterType) -> ParameterDefinition { + parameter_with_visibility(id, parameter_type, ParameterVisibility::Basic) } fn parameter_with_visibility( id: &str, parameter_type: ParameterType, - advanced: bool, - ) -> ApplicationParameter { - ApplicationParameter { - definition: Some(ParameterDefinition { - parameter_id: id.to_owned(), - display_name: id.to_owned(), - description: String::new(), - parameter_type: Some(parameter_type), - }), - advanced, + visibility: ParameterVisibility, + ) -> ParameterDefinition { + ParameterDefinition { + parameter_id: id.to_owned(), + display_name: id.to_owned(), + description: String::new(), + visibility: visibility as i32, + parameter_type: Some(parameter_type), } } @@ -801,6 +1109,25 @@ services: }) } + fn text( + minimum_length: u32, + maximum_length: u32, + default_value: Option<&str>, + ) -> ParameterType { + ParameterType::Text(TextParameterSchema { + minimum_length, + maximum_length, + default_value: default_value.map(str::to_owned), + }) + } + + fn secret(generated_length: u32, minimum_length: u32) -> ParameterType { + ParameterType::Secret(SecretParameterSchema { + generated_length, + minimum_length, + }) + } + fn string_enum(values: &[&str], default_value: Option<&str>) -> ParameterType { ParameterType::StringEnum(StringEnumParameterSchema { values: values.iter().map(|value| (*value).to_owned()).collect(), diff --git a/backend/catalog/src/validation/engine_release.rs b/backend/catalog/src/validation/engine_release.rs index 1f64cb81..5770d8d9 100644 --- a/backend/catalog/src/validation/engine_release.rs +++ b/backend/catalog/src/validation/engine_release.rs @@ -9,14 +9,17 @@ use std::collections::BTreeSet; use dotcore_domain::{Accelerator, valid_id}; use dotcore_proto::dotcore::catalog::v1::{ - EngineArtifact, EngineParameterDefinition, EngineParameterSchema, EngineRelease, Selector, - engine_artifact::Kind as EngineArtifactKind, engine_parameter_definition::ParameterType, + EngineArtifact, EngineParameterSchema, EngineRelease, ParameterDefinition, Selector, + engine_artifact::Kind as EngineArtifactKind, parameter_definition::ParameterType, }; use super::common::{valid_selector_memory_architecture, validate_oci_reference}; use super::nix::{AcceptedProvides, nix_systems, validate_nix_artifact}; use super::storage::{StorageOwner, validate_storage_bindings}; -use crate::{OciArtifactPolicy, shared_parameters::validate_shared_parameters}; +use crate::{ + OciArtifactPolicy, shared_parameters::validate_shared_parameters, + typed_parameters::validate_parameter_definition as validate_shared_definition, +}; pub fn validate_engine_release_with_policy( engine: &EngineRelease, @@ -229,37 +232,33 @@ fn validate_parameter_schema( errors.extend(validate_shared_parameters(schema)); } +/// Validates one engine parameter definition: the shared definition rules, a unique id, and the +/// engine owner rule that parameters are booleans, unsigned integers with a unit, or secrets. fn validate_parameter_definition( - definition: &EngineParameterDefinition, + definition: &ParameterDefinition, ids: &mut BTreeSet, errors: &mut Vec, ) { - if !ids.insert(definition.parameter_id.clone()) - || !valid_id(&definition.parameter_id) - || definition.display_name.trim().is_empty() - { - errors.push(anyhow::anyhow!( - "engine parameter requires a unique valid ID and display name" - )); + let id = &definition.parameter_id; + if !ids.insert(id.clone()) { + errors.push(anyhow::anyhow!("duplicate engine parameter {id:?}")); } - if definition - .parameter_type - .as_ref() - .is_none_or(|kind| match kind { - ParameterType::UnsignedInteger(schema) => { - schema.minimum > schema.maximum - || schema.unit.trim().is_empty() - || schema.default_value.is_some_and(|default| { - !(schema.minimum..=schema.maximum).contains(&default) - }) - } - ParameterType::Boolean(_) => false, - }) - { - errors.push(anyhow::anyhow!( - "engine parameter {:?} requires a valid typed range holding its default", - definition.parameter_id - )); + errors.extend(validate_shared_definition(definition)); + match definition.parameter_type.as_ref() { + Some(ParameterType::UnsignedInteger(schema)) if schema.unit.trim().is_empty() => { + errors.push(anyhow::anyhow!("engine parameter {id:?} requires a unit")); + } + Some(ParameterType::StringEnum(_) | ParameterType::Text(_)) => { + errors.push(anyhow::anyhow!( + "engine parameter {id:?} must be a boolean, an unsigned integer or a secret" + )) + } + Some( + ParameterType::Boolean(_) + | ParameterType::UnsignedInteger(_) + | ParameterType::Secret(_), + ) + | None => {} } } @@ -267,8 +266,8 @@ fn validate_parameter_definition( mod tests { use dotcore_proto::dotcore::{ catalog::v1::{ - NixArtifact, NixDockerImage, StorageBinding, StorageKind, StorageScope, - nix_artifact::Provides, + NixArtifact, NixDockerImage, ParameterVisibility, StorageBinding, StorageKind, + StoragePlacement, nix_artifact::Provides, }, observation::v1::AcceleratorMemoryArchitecture, }; @@ -403,18 +402,18 @@ mod tests { #[test] fn parameter_schemas_reject_an_out_of_range_default_and_a_missing_shared_parameter() { let unsigned = |parameter_id: &str, display_name: &str, unit: &str, default_value| { - EngineParameterDefinition { + ParameterDefinition { parameter_id: parameter_id.to_owned(), display_name: display_name.to_owned(), parameter_type: Some(ParameterType::UnsignedInteger( - dotcore_proto::dotcore::catalog::v1::UnsignedIntegerEngineParameterSchema { + dotcore_proto::dotcore::catalog::v1::UnsignedIntegerParameterSchema { minimum: 1024, maximum: 65_536, unit: unit.to_owned(), default_value, }, )), - exposed: true, + visibility: ParameterVisibility::Basic as i32, description: String::new(), } }; @@ -432,14 +431,89 @@ mod tests { assert_eq!( errors.iter().map(ToString::to_string).collect::>(), [ - "engine parameter \"total-context\" requires a valid typed range holding its default", + "parameter \"total-context\" default is invalid: 131072 is outside 1024..=65536", "engine parameter schema \"fixture-shared\" must declare the shared parameter \"parallel-requests\"", ] ); } #[test] - fn engine_storage_requires_engine_scopes_cache_kind_and_container_artifacts() { + fn engine_parameters_are_booleans_unsigned_integers_with_a_unit_or_secrets() { + let definition = |parameter_type| ParameterDefinition { + parameter_id: "feature".to_owned(), + display_name: "Feature".to_owned(), + description: String::new(), + visibility: ParameterVisibility::Advanced as i32, + parameter_type: Some(parameter_type), + }; + let errors = |definition: &ParameterDefinition| { + let mut errors = Vec::new(); + validate_parameter_definition(definition, &mut BTreeSet::new(), &mut errors); + errors.iter().map(ToString::to_string).collect::>() + }; + let unsigned = |unit: &str| { + ParameterType::UnsignedInteger( + dotcore_proto::dotcore::catalog::v1::UnsignedIntegerParameterSchema { + minimum: 0, + maximum: 8, + unit: unit.to_owned(), + default_value: Some(4), + }, + ) + }; + let boolean = ParameterType::Boolean( + dotcore_proto::dotcore::catalog::v1::BooleanParameterSchema { + default_value: Some(true), + }, + ); + + assert_eq!(errors(&definition(boolean.clone())), Vec::::new()); + assert_eq!(errors(&definition(unsigned("MiB"))), Vec::::new()); + assert_eq!( + errors(&definition(unsigned(" "))), + ["engine parameter \"feature\" requires a unit"] + ); + assert_eq!( + errors(&definition(ParameterType::StringEnum( + dotcore_proto::dotcore::catalog::v1::StringEnumParameterSchema { + values: vec!["on".to_owned()], + default_value: Some("on".to_owned()), + }, + ))), + ["engine parameter \"feature\" must be a boolean, an unsigned integer or a secret"] + ); + assert_eq!( + errors(&definition(ParameterType::Text( + dotcore_proto::dotcore::catalog::v1::TextParameterSchema { + minimum_length: 0, + maximum_length: 8, + default_value: Some("on".to_owned()), + }, + ))), + ["engine parameter \"feature\" must be a boolean, an unsigned integer or a secret"] + ); + assert_eq!( + errors(&definition(ParameterType::Secret( + dotcore_proto::dotcore::catalog::v1::SecretParameterSchema { + generated_length: 32, + minimum_length: 12, + }, + ))), + Vec::::new() + ); + let mut basic = definition(boolean.clone()); + basic.visibility = ParameterVisibility::Basic as i32; + assert_eq!(errors(&basic), Vec::::new()); + let mut unplaced = definition(boolean); + unplaced.visibility = ParameterVisibility::Unspecified as i32; + assert_eq!( + errors(&unplaced), + ["parameter \"feature\" requires a basic, advanced or hidden visibility"] + ); + } + + #[test] + fn engine_storage_requires_engine_placements_cache_kind_and_container_artifacts() { let mut image = nix("self"); image.provides = Some(Provides::DockerImage(NixDockerImage {})); let artifact = |artifact_id: &str, kind| EngineArtifact { @@ -447,13 +521,14 @@ mod tests { kind: Some(kind), ..Default::default() }; - let binding = |binding_name: &str, kind: StorageKind, scope: StorageScope| StorageBinding { - binding_name: binding_name.to_owned(), - mount_path: format!("/cache/{binding_name}"), - kind: kind as i32, - scope: scope as i32, - environment: Vec::new(), - }; + let binding = + |binding_name: &str, kind: StorageKind, placement: StoragePlacement| StorageBinding { + binding_name: binding_name.to_owned(), + mount_path: format!("/cache/{binding_name}"), + kind: kind as i32, + placements: vec![placement as i32], + environment: Vec::new(), + }; let mut engine = EngineRelease { engine_release_id: "fixture".to_owned(), artifacts: vec![ @@ -461,8 +536,12 @@ mod tests { artifact("image", EngineArtifactKind::Nix(image)), ], storage_bindings: vec![ - binding("compile", StorageKind::Cache, StorageScope::EngineArtifact), - binding("scratch", StorageKind::Cache, StorageScope::Deployment), + binding( + "compile", + StorageKind::Cache, + StoragePlacement::EngineArtifact, + ), + binding("scratch", StorageKind::Cache, StoragePlacement::Deployment), ], ..Default::default() }; @@ -478,15 +557,15 @@ mod tests { .artifacts .push(artifact("native", EngineArtifactKind::Nix(nix("self")))); engine.storage_bindings.extend([ - binding("data", StorageKind::Data, StorageScope::Deployment), - binding("shared", StorageKind::Cache, StorageScope::Application), + binding("data", StorageKind::Data, StoragePlacement::Deployment), + binding("shared", StorageKind::Cache, StoragePlacement::Application), ]); assert_eq!( errors(&engine), [ "engine release \"fixture\" storage binding \"data\" cannot use kind STORAGE_KIND_DATA", - "engine release \"fixture\" storage binding \"shared\" cannot use scope STORAGE_SCOPE_APPLICATION", + "engine release \"fixture\" storage binding \"shared\" cannot use placement STORAGE_PLACEMENT_APPLICATION", "engine release \"fixture\" declares storage bindings, so engine artifact \"native\" must be a container image", ] ); diff --git a/backend/catalog/src/validation/graph.rs b/backend/catalog/src/validation/graph.rs index 15633580..1b59a4ae 100644 --- a/backend/catalog/src/validation/graph.rs +++ b/backend/catalog/src/validation/graph.rs @@ -5,13 +5,14 @@ use std::collections::{BTreeMap, BTreeSet}; use dotcore_domain::valid_id; use dotcore_proto::dotcore::catalog::v1::{ ApplicationDependencyCondition, ApplicationEndpointProtocol, ApplicationGraph, - ApplicationService, application_configuration_binding, parameter_definition::ParameterType, + ApplicationService, ParameterVisibility, application_configuration_binding, + parameter_definition::ParameterType, }; use super::application::valid_configuration_value; use super::service::validate_application_service; use crate::OciArtifactPolicy; -use crate::typed_parameters::{default_value, validate_parameter_definition}; +use crate::typed_parameters::{default_value, is_secret, validate_parameter_definition}; pub(super) fn validate_application_graph( graph: &ApplicationGraph, @@ -179,17 +180,14 @@ fn validate_primary_endpoint( } } -/// Checks parameter definitions and that bindings and parameters reference each other exactly: -/// every binding names a declared parameter and every parameter supplies at least one binding. +/// Checks parameter definitions, each shown as basic or advanced and, except a secret, with a +/// default, and that bindings and parameters reference each other exactly: every binding names a +/// declared parameter, secrets only through secret bindings and other parameters only through +/// configuration bindings, a binding is enabled only by a boolean parameter, and every parameter +/// supplies or enables at least one binding. fn validate_parameters(graph: &ApplicationGraph, errors: &mut Vec) { let mut parameter_ids = BTreeSet::new(); - for parameter in &graph.parameters { - let Some(definition) = parameter.definition.as_ref() else { - errors.push(anyhow::anyhow!( - "application parameter requires a definition" - )); - continue; - }; + for definition in &graph.parameters { let id = &definition.parameter_id; if !parameter_ids.insert(id.as_str()) { errors.push(anyhow::anyhow!("duplicate application parameter {id:?}")); @@ -197,11 +195,17 @@ fn validate_parameters(graph: &ApplicationGraph, errors: &mut Vec let definition_errors = validate_parameter_definition(definition); let valid_definition = definition_errors.is_empty(); errors.extend(definition_errors); - if valid_definition && default_value(definition).is_none() { + if valid_definition && !is_secret(definition) && default_value(definition).is_none() { errors.push(anyhow::anyhow!( "application parameter {id:?} requires a default" )); } + // A value the operator never sees belongs in the graph as a literal. + if definition.visibility() == ParameterVisibility::Hidden { + errors.push(anyhow::anyhow!( + "application parameter {id:?} must be basic or advanced, not hidden" + )); + } // Enum values are projected verbatim into environment variables. if let Some(ParameterType::StringEnum(schema)) = definition.parameter_type.as_ref() && !schema @@ -215,31 +219,118 @@ fn validate_parameters(graph: &ApplicationGraph, errors: &mut Vec } } + let parameter_kinds = graph + .parameters + .iter() + .map(|definition| { + let kind = match definition.parameter_type { + Some(ParameterType::Secret(_)) => ParameterKind::Secret, + Some(ParameterType::Boolean(_)) => ParameterKind::Boolean, + _ => ParameterKind::Other, + }; + (definition.parameter_id.as_str(), kind) + }) + .collect::>(); let mut referenced_ids = BTreeSet::new(); for service in &graph.services { + let service_name = &service.service_name; for binding in &service.configuration_bindings { + let binding_name = &binding.binding_name; + if let Some(id) = &binding.enabled_by_parameter_id { + referenced_ids.insert(id.as_str()); + check_enabler( + ¶meter_kinds, + "configuration", + binding_name, + service_name, + id, + errors, + ); + // Launch projects the route name in place of a route-input binding's attachment, + // so nothing would observe its switch. + if binding.input.is_some() { + errors.push(anyhow::anyhow!( + "application configuration binding {binding_name:?} on service {service_name:?} takes a route input, so it cannot be enabled by a parameter" + )); + } + } let Some(application_configuration_binding::Source::ParameterId(id)) = binding.source.as_ref() else { continue; }; referenced_ids.insert(id.as_str()); - if !parameter_ids.contains(id.as_str()) { - errors.push(anyhow::anyhow!( - "application configuration binding {:?} on service {:?} references missing parameter {id:?}", - binding.binding_name, - service.service_name - )); + match parameter_kinds.get(id.as_str()) { + None => errors.push(anyhow::anyhow!( + "application configuration binding {binding_name:?} on service {service_name:?} references missing parameter {id:?}" + )), + Some(ParameterKind::Secret) => errors.push(anyhow::anyhow!( + "application configuration binding {binding_name:?} on service {service_name:?} references secret parameter {id:?}; secrets use secret bindings" + )), + Some(_) => {} + } + } + for binding in &service.secret_bindings { + let binding_name = &binding.binding_name; + if let Some(id) = &binding.enabled_by_parameter_id { + referenced_ids.insert(id.as_str()); + check_enabler( + ¶meter_kinds, + "secret", + binding_name, + service_name, + id, + errors, + ); + } + let id = binding.parameter_id.as_str(); + referenced_ids.insert(id); + match parameter_kinds.get(id) { + None => errors.push(anyhow::anyhow!( + "application secret binding {binding_name:?} on service {service_name:?} references missing parameter {id:?}" + )), + Some(ParameterKind::Secret) => {} + Some(_) => errors.push(anyhow::anyhow!( + "application secret binding {binding_name:?} on service {service_name:?} references parameter {id:?}, which is not a secret" + )), } } } for id in parameter_ids.difference(&referenced_ids) { errors.push(anyhow::anyhow!( - "application parameter {id:?} is not used by any configuration binding" + "application parameter {id:?} is not used by any binding" )); } } +/// The kind of a graph parameter that decides which bindings may reference it. +#[derive(Clone, Copy)] +enum ParameterKind { + Secret, + Boolean, + Other, +} + +/// Checks that the parameter enabling one binding is a declared boolean. +fn check_enabler( + parameter_kinds: &BTreeMap<&str, ParameterKind>, + kind: &str, + binding_name: &str, + service_name: &str, + id: &str, + errors: &mut Vec, +) { + match parameter_kinds.get(id) { + None => errors.push(anyhow::anyhow!( + "application {kind} binding {binding_name:?} on service {service_name:?} is enabled by missing parameter {id:?}" + )), + Some(ParameterKind::Boolean) => {} + Some(_) => errors.push(anyhow::anyhow!( + "application {kind} binding {binding_name:?} on service {service_name:?} is enabled by parameter {id:?}, which is not a boolean" + )), + } +} + fn application_dependencies_are_cyclic( graph: &ApplicationGraph, service_names: &BTreeSet<&str>, diff --git a/backend/catalog/src/validation/mod.rs b/backend/catalog/src/validation/mod.rs index 3631a63d..78945519 100644 --- a/backend/catalog/src/validation/mod.rs +++ b/backend/catalog/src/validation/mod.rs @@ -10,8 +10,9 @@ use std::collections::{BTreeMap, BTreeSet}; use dotcore_proto::dotcore::catalog::v1::{ - Application, EngineParameterDefinition, EngineParameterValues, EngineRelease, HardwareClass, - HardwareClassRecommendations, Model, ModelEngineConfiguration, UpstreamModelTemplate, + Application, EngineRelease, HardwareClass, HardwareClassRecommendations, Model, + ModelEngineConfiguration, ParameterDefinition, ParameterValues, ParameterVisibility, + UpstreamModelTemplate, }; use super::{ @@ -23,9 +24,11 @@ pub use application::{ valid_configuration_value, valid_environment_variable, validate_application_with_policy, }; pub use model::{ - overlay_parameter_defaults, valid_pinned_hugging_face_artifact, validate_parameter_values, + is_secret_parameter, overlay_parameter_defaults, valid_pinned_hugging_face_artifact, + validate_parameter_values, validate_parameter_values_before_secrets, validate_supplied_parameter_values, }; +pub(super) use sensitive_literal::validate_sensitive_literals; pub use upstream_model_template::validate_upstream_model_template; mod application; @@ -36,6 +39,7 @@ mod hardware_class; mod model; mod nix; mod recommendation; +mod sensitive_literal; mod service; mod storage; mod upstream_model_template; @@ -72,8 +76,8 @@ pub trait EngineDeclarationValidator: Send + Sync { _model: &Model, _configuration: &ModelEngineConfiguration, _engine: &EngineRelease, - ) -> EngineParameterValues { - EngineParameterValues::default() + ) -> ParameterValues { + ParameterValues::default() } } @@ -218,7 +222,7 @@ pub(super) fn validate_catalog( } /// Engine parameter defaults of each configuration, keyed by model and configuration id. -pub(super) type ConfigurationEngineDefaults = BTreeMap<(String, String), EngineParameterValues>; +pub(super) type ConfigurationEngineDefaults = BTreeMap<(String, String), ParameterValues>; pub(super) struct ValidatedCatalog { pub(super) models: BTreeMap, @@ -453,11 +457,38 @@ fn validate_sizing_coverage( continue; }; let defaulted = model::overlay_parameter_defaults(values, &engine_defaults); - // Resolution fills the context per request, so no sizing needs to derive it. + // Resolution fills the context per request and deployment creation generates + // secrets, so no sizing needs to derive either. for parameter_id in model::missing_parameter_ids(&defaulted, engine) .into_iter() - .filter(|parameter_id| parameter_id != CONTEXT_PER_REQUEST_PARAMETER_ID) + .filter(|parameter_id| { + parameter_id != CONTEXT_PER_REQUEST_PARAMETER_ID + && !model::is_secret_parameter(engine, parameter_id) + }) { + let hidden = profile.engine_parameter_visibility_overrides.iter().any( + |visibility_override| { + visibility_override.parameter_id == parameter_id + && visibility_override.visibility() == ParameterVisibility::Hidden + }, + ); + // A hidden parameter is one the operator can never set, so it needs a value + // on every binding and class. + if hidden + && class_bindings().any(|(hardware_class_id, binding)| { + !binding_sizing(hardware_class_id, &binding.binding_id) + .is_some_and(|sizing| sizing.derives(¶meter_id)) + }) + { + diagnostics.push(CatalogDiagnostic::new( + &location, + anyhow::anyhow!( + "profile {:?} hides engine parameter {parameter_id:?}, which has no default and which sizing does not derive on every binding and hardware class", + profile.profile_id + ), + )); + continue; + } for (hardware_class_id, binding) in class_bindings() { if !binding_sizing(hardware_class_id, &binding.binding_id) .is_some_and(|sizing| sizing.derives(¶meter_id)) @@ -479,9 +510,9 @@ fn validate_sizing_coverage( } fn parameter_definition<'a>( - parameters: &'a [EngineParameterDefinition], + parameters: &'a [ParameterDefinition], parameter_id: &str, -) -> Option<&'a EngineParameterDefinition> { +) -> Option<&'a ParameterDefinition> { parameters .iter() .find(|parameter| parameter.parameter_id == parameter_id) diff --git a/backend/catalog/src/validation/model.rs b/backend/catalog/src/validation/model.rs index b6e3cd8a..be4ad525 100644 --- a/backend/catalog/src/validation/model.rs +++ b/backend/catalog/src/validation/model.rs @@ -11,15 +11,15 @@ use std::collections::{BTreeMap, BTreeSet}; use dotcore_domain::valid_id; use dotcore_proto::dotcore::catalog::v1::{ - EngineArtifact, EngineParameterValue, EngineParameterValues, EngineRelease, HardwareClass, - HuggingFaceArtifactSource, InferenceRequestKind, IntelligenceScore, License, Model, - ModelArtifact, ModelConfigurationOption, ModelEngineConfiguration, ModelInputCapabilities, - ModelInputModality, ModelOptionParameterDefinition, SpeculativeDecodingMethod, - engine_parameter_definition::ParameterType as EngineParameterType, - engine_parameter_value::Value as EngineParameterValueKind, - hugging_face_artifact_source::Content as HuggingFaceContent, + EngineArtifact, EngineRelease, HardwareClass, HuggingFaceArtifactSource, InferenceRequestKind, + IntelligenceScore, License, Model, ModelArtifact, ModelConfigurationOption, + ModelEngineConfiguration, ModelInputCapabilities, ModelInputModality, + ModelOptionParameterDefinition, ModelProfile, ParameterValue, ParameterValues, + SpeculativeDecodingMethod, hugging_face_artifact_source::Content as HuggingFaceContent, model_option::Kind as ModelOptionKind, model_option_parameter_definition::ParameterType as ModelOptionParameterType, + parameter_definition::ParameterType as EngineParameterType, + parameter_value::Value as EngineParameterValueKind, }; use super::common::lower_hex; @@ -30,7 +30,8 @@ use crate::{ valid_option_values, }, selector::{SelectorRelation, selector_relation}, - shared_parameters::profile_may_default, + shared_parameters::{profile_may_default, profile_visibility_problem}, + typed_parameters::{check_value, default_value, is_secret}, validation::{EngineDeclarationValidator, configuration_engine_release}, }; @@ -818,24 +819,7 @@ fn validate_profiles( let Some(engine) = configuration_engine_release(configuration, engines) else { continue; }; - if !profile.exposed_parameter_ids.is_empty() { - let mut exposed_ids = BTreeSet::new(); - let schema = engine.parameter_schema.as_ref(); - if profile.exposed_parameter_ids.iter().any(|parameter_id| { - !exposed_ids.insert(parameter_id) - || schema.is_none_or(|schema| { - !schema - .parameters - .iter() - .any(|definition| definition.parameter_id == *parameter_id) - }) - }) { - errors.push(anyhow::anyhow!( - "profile {:?} has invalid exposed engine parameters", - profile.profile_id - )); - } - } + validate_visibility_overrides(profile, engine, errors); match profile.engine_parameter_defaults.as_ref() { Some(values) => { validate_supplied_parameter_values(values, engine, &profile.profile_id, errors); @@ -850,6 +834,18 @@ fn validate_profiles( value.parameter_id )); } + // A secret default would be a literal every deployment shares. + for value in values + .values + .iter() + .filter(|value| is_secret_parameter(engine, &value.parameter_id)) + { + errors.push(anyhow::anyhow!( + "profile {:?} sets secret engine parameter {:?}, which each deployment generates or receives", + profile.profile_id, + value.parameter_id + )); + } } None => errors.push(anyhow::anyhow!( "profile {:?} requires engine parameter defaults", @@ -867,6 +863,44 @@ fn validate_profiles( } } +/// Validates a profile's engine parameter visibility overrides: each names a parameter of the +/// engine release once, with a visibility the shared parameter rules allow for profiles. +fn validate_visibility_overrides( + profile: &ModelProfile, + engine: &EngineRelease, + errors: &mut Vec, +) { + let profile_id = &profile.profile_id; + let mut ids = BTreeSet::new(); + for visibility_override in &profile.engine_parameter_visibility_overrides { + let parameter_id = &visibility_override.parameter_id; + if !engine + .parameter_schema + .iter() + .flat_map(|schema| &schema.parameters) + .any(|definition| &definition.parameter_id == parameter_id) + { + errors.push(anyhow::anyhow!( + "profile {profile_id:?} overrides the visibility of engine parameter {parameter_id:?}, which is not a parameter of the engine release" + )); + continue; + } + if !ids.insert(parameter_id) { + errors.push(anyhow::anyhow!( + "profile {profile_id:?} overrides the visibility of engine parameter {parameter_id:?} more than once" + )); + continue; + } + if let Some(problem) = + profile_visibility_problem(parameter_id, visibility_override.visibility()) + { + errors.push(anyhow::anyhow!( + "profile {profile_id:?} engine parameter {parameter_id:?} {problem}" + )); + } + } +} + fn validate_profile_option_defaults( profile: &dotcore_proto::dotcore::catalog::v1::ModelProfile, configuration: &ModelEngineConfiguration, @@ -891,26 +925,50 @@ fn validate_profile_option_defaults( /// Validates a complete set of engine parameter values against the engine schema. pub fn validate_parameter_values( - values: &EngineParameterValues, + values: &ParameterValues, engine: &EngineRelease, profile_id: &str, errors: &mut Vec, ) { - let missing = missing_parameter_ids(values, engine); - if !missing.is_empty() { - errors.push(anyhow::anyhow!( - "profile {profile_id:?} engine parameter values are incomplete; missing {}", - missing.join(", ") - )); - return; - } - validate_supplied_parameter_values(values, engine, profile_id, errors); + validate_values( + values, + engine, + profile_id, + SecretCompleteness::Required, + errors, + ); +} + +/// Validates engine parameter values that are complete except, possibly, for secrets, whose +/// values exist only once a deployment is created. +pub fn validate_parameter_values_before_secrets( + values: &ParameterValues, + engine: &EngineRelease, + profile_id: &str, + errors: &mut Vec, +) { + validate_values( + values, + engine, + profile_id, + SecretCompleteness::Optional, + errors, + ); +} + +/// Whether the engine release declares `parameter_id` as a secret. +pub fn is_secret_parameter(engine: &EngineRelease, parameter_id: &str) -> bool { + engine + .parameter_schema + .iter() + .flat_map(|schema| &schema.parameters) + .any(|definition| definition.parameter_id == parameter_id && is_secret(definition)) } /// Validates supplied engine parameter values, which may omit parameters that sizing derives. /// Completeness is checked once derivations are known. pub fn validate_supplied_parameter_values( - values: &EngineParameterValues, + values: &ParameterValues, engine: &EngineRelease, profile_id: &str, errors: &mut Vec, @@ -952,6 +1010,13 @@ pub fn validate_supplied_parameter_values( } (Some(EngineParameterType::Boolean(_)), Some(EngineParameterValueKind::Boolean(_))) => { } + (Some(EngineParameterType::Secret(_)), Some(value @ EngineParameterValueKind::String(_))) => { + if let Err(error) = check_value(definition, value) { + errors.push(anyhow::anyhow!( + "profile {profile_id:?} engine parameter {parameter_id:?} value is invalid: {error}" + )); + } + } _ => errors.push(anyhow::anyhow!( "profile {profile_id:?} engine parameter {parameter_id:?} has a value of the wrong type" )), @@ -962,9 +1027,9 @@ pub fn validate_supplied_parameter_values( /// Overlays defaults on supplied engine parameter values, per parameter: the supplied values /// keep their order, followed by each default whose parameter they leave out. pub fn overlay_parameter_defaults( - values: &EngineParameterValues, - defaults: &EngineParameterValues, -) -> EngineParameterValues { + values: &ParameterValues, + defaults: &ParameterValues, +) -> ParameterValues { let mut overlaid = values.clone(); overlaid.values.extend( defaults @@ -982,30 +1047,24 @@ pub fn overlay_parameter_defaults( } /// Engine defaults the release declares for its parameters, in schema order. -pub(super) fn engine_default_values(engine: &EngineRelease) -> EngineParameterValues { - EngineParameterValues { +pub(super) fn engine_default_values(engine: &EngineRelease) -> ParameterValues { + ParameterValues { values: engine .parameter_schema .iter() .flat_map(|schema| &schema.parameters) - .filter_map(|definition| match definition.parameter_type.as_ref() { - Some(EngineParameterType::UnsignedInteger(range)) => { - range.default_value.map(|default| EngineParameterValue { - parameter_id: definition.parameter_id.clone(), - value: Some(EngineParameterValueKind::UnsignedInteger(default)), - }) - } - _ => None, + .filter_map(|definition| { + default_value(definition).map(|default| ParameterValue { + parameter_id: definition.parameter_id.clone(), + value: Some(default), + }) }) .collect(), } } /// Engine schema parameter ids without a supplied value, in schema order. -pub fn missing_parameter_ids( - values: &EngineParameterValues, - engine: &EngineRelease, -) -> Vec { +pub fn missing_parameter_ids(values: &ParameterValues, engine: &EngineRelease) -> Vec { engine .parameter_schema .as_ref() @@ -1025,6 +1084,36 @@ pub fn missing_parameter_ids( .unwrap_or_default() } +/// Whether value completeness covers secret parameters. +#[derive(Clone, Copy, PartialEq, Eq)] +enum SecretCompleteness { + Required, + Optional, +} + +fn validate_values( + values: &ParameterValues, + engine: &EngineRelease, + profile_id: &str, + secrets: SecretCompleteness, + errors: &mut Vec, +) { + let missing = missing_parameter_ids(values, engine) + .into_iter() + .filter(|parameter_id| { + secrets == SecretCompleteness::Required || !is_secret_parameter(engine, parameter_id) + }) + .collect::>(); + if !missing.is_empty() { + errors.push(anyhow::anyhow!( + "profile {profile_id:?} engine parameter values are incomplete; missing {}", + missing.join(", ") + )); + return; + } + validate_supplied_parameter_values(values, engine, profile_id, errors); +} + fn valid_weight_quantization(value: &str) -> bool { value.is_empty() || (value.len() <= MAX_WEIGHT_QUANTIZATION_BYTES @@ -1133,23 +1222,73 @@ mod tests { use std::collections::{BTreeMap, BTreeSet}; use dotcore_proto::dotcore::catalog::v1::{ - BooleanModelOptionParameterSchema, EngineArtifact, EngineArtifactReference, - EngineParameterValues, EngineRelease, HardwareClass, InferenceRequestKind, - KvCachePrecisionModelOption, Model, ModelArtifactBinding, ModelConfigurationOption, - ModelEngineConfiguration, ModelOption, ModelOptionParameterDefinition, - ModelOptionParameterValue, ModelOptionValues, ModelProfile, OciPlatform, Selector, - SpeculativeDecodingMethod, SpeculativeDecodingModelOption, - StringEnumModelOptionParameterSchema, UnsignedIntegerModelOptionParameterSchema, - model_option, model_option_parameter_definition::ParameterType, - model_option_parameter_value::Value, + BooleanModelOptionParameterSchema, EngineArtifact, EngineArtifactReference, EngineRelease, + HardwareClass, InferenceRequestKind, KvCachePrecisionModelOption, Model, + ModelArtifactBinding, ModelConfigurationOption, ModelEngineConfiguration, ModelOption, + ModelOptionParameterDefinition, ModelOptionParameterValue, ModelOptionValues, ModelProfile, + OciPlatform, ParameterValues, Selector, SpeculativeDecodingMethod, + SpeculativeDecodingModelOption, StringEnumModelOptionParameterSchema, + UnsignedIntegerModelOptionParameterSchema, model_option, + model_option_parameter_definition::ParameterType, model_option_parameter_value::Value, }; use super::{ - overlapping_artifacts, valid_request_kinds, validate_class_artifact_coverage, - validate_configuration_options, validate_model_options, validate_profiles, + engine_default_values, overlapping_artifacts, valid_request_kinds, + validate_class_artifact_coverage, validate_configuration_options, validate_model_options, + validate_profiles, }; use crate::test_support::vendor_hardware_class; + #[test] + fn engine_defaults_cover_every_parameter_declaring_one_booleans_included() { + use dotcore_proto::dotcore::catalog::v1::{ + BooleanParameterSchema, EngineParameterSchema, ParameterDefinition, ParameterValue, + UnsignedIntegerParameterSchema, parameter_definition::ParameterType as Type, + parameter_value::Value as ParameterValueKind, + }; + + let definition = |parameter_id: &str, parameter_type| ParameterDefinition { + parameter_id: parameter_id.to_owned(), + parameter_type: Some(parameter_type), + ..Default::default() + }; + let unsigned = |default_value| { + Type::UnsignedInteger(UnsignedIntegerParameterSchema { + minimum: 0, + maximum: 64, + unit: "requests".to_owned(), + default_value, + }) + }; + let boolean = |default_value| Type::Boolean(BooleanParameterSchema { default_value }); + let engine = EngineRelease { + parameter_schema: Some(EngineParameterSchema { + schema_id: "fixture".to_owned(), + parameters: vec![ + definition("parallel-requests", unsigned(Some(4))), + definition("flash-attention", boolean(Some(false))), + definition("cache-mib", unsigned(None)), + definition("mlock", boolean(None)), + ], + }), + ..Default::default() + }; + let value = |parameter_id: &str, value| ParameterValue { + parameter_id: parameter_id.to_owned(), + value: Some(value), + }; + + assert_eq!( + engine_default_values(&engine), + ParameterValues { + values: vec![ + value("parallel-requests", ParameterValueKind::UnsignedInteger(4)), + value("flash-attention", ParameterValueKind::Boolean(false)), + ], + } + ); + } + #[test] fn request_kinds_must_be_non_empty_known_and_unique() { let chat = InferenceRequestKind::Chat as i32; @@ -1601,7 +1740,7 @@ mod tests { profile_id: "balanced".to_owned(), configuration_id: "local".to_owned(), default_model_artifact_binding_id: "weights".to_owned(), - engine_parameter_defaults: Some(EngineParameterValues::default()), + engine_parameter_defaults: Some(ParameterValues::default()), ..Default::default() }], ..Default::default() diff --git a/backend/catalog/src/validation/sensitive_literal.rs b/backend/catalog/src/validation/sensitive_literal.rs new file mode 100644 index 00000000..80c6ab5e --- /dev/null +++ b/backend/catalog/src/validation/sensitive_literal.rs @@ -0,0 +1,195 @@ +//! Keeps secrets out of the master catalog's application declarations. +//! +//! A value the catalog itself sets on a service's environment ships to every site, so a variable +//! whose name marks it as a secret must get its value from a secret parameter instead. An empty +//! value stays allowed: it only tells the service that no secret is set. + +use std::collections::BTreeMap; + +use anyhow::anyhow; +use dotcore_proto::dotcore::catalog::v1::{ + Application, ApplicationConfigurationBinding, ParameterDefinition, + application_configuration_binding::Source, +}; + +use crate::{ + is_sensitive_name, + typed_parameters::{canonical_text, default_value}, +}; + +/// Returns one error per configuration binding that sets a non-empty catalog value, either a +/// literal or a parameter default, on an environment variable with a sensitive name. Errors name +/// the application, service, binding and variable, never the value. +pub(crate) fn validate_sensitive_literals(application: &Application) -> Vec { + let Some(graph) = application.graph.as_ref() else { + return Vec::new(); + }; + let parameters = graph + .parameters + .iter() + .map(|definition| (definition.parameter_id.as_str(), definition)) + .collect::>(); + + let mut errors = Vec::new(); + for service in &graph.services { + for binding in &service.configuration_bindings { + if !is_sensitive_name(&binding.environment_variable) + || !binds_catalog_value(binding, ¶meters) + { + continue; + } + errors.push(anyhow!( + "application {:?} service {:?} configuration binding {:?} sets a catalog value on sensitive variable {:?}; bind a secret parameter instead", + application.application_id, + service.service_name, + binding.binding_name, + binding.environment_variable + )); + } + } + errors +} + +/// Whether the binding's environment value comes from the catalog and is not empty. +fn binds_catalog_value( + binding: &ApplicationConfigurationBinding, + parameters: &BTreeMap<&str, &ParameterDefinition>, +) -> bool { + match binding.source.as_ref() { + Some(Source::LiteralValue(value)) => !value.is_empty(), + Some(Source::ParameterId(id)) => parameters + .get(id.as_str()) + .and_then(|definition| default_value(definition)) + .is_some_and(|value| !canonical_text(&value).is_empty()), + Some(Source::ResolvedValueName(_)) | None => false, + } +} + +#[cfg(test)] +mod tests { + use dotcore_proto::dotcore::catalog::v1::{ + ApplicationConfigurationBinding, ParameterDefinition, ParameterVisibility, + TextParameterSchema, application_configuration_binding::Source, + parameter_definition::ParameterType, + }; + + use super::validate_sensitive_literals; + use crate::test_support::application; + + #[test] + fn non_empty_literal_on_a_sensitive_variable_is_rejected_without_its_value() { + let mut application = application(); + add_binding( + &mut application, + "api-key", + "SERVICE_API_KEY", + Source::LiteralValue("hunter2".to_owned()), + ); + + let errors = messages(&application); + + assert_eq!( + errors, + [ + "application \"fixture-application\" service \"web\" configuration binding \"api-key\" sets a catalog value on sensitive variable \"SERVICE_API_KEY\"; bind a secret parameter instead" + ] + ); + assert!(!errors[0].contains("hunter2")); + } + + #[test] + fn empty_literals_and_non_sensitive_variables_are_allowed() { + let mut application = application(); + add_binding( + &mut application, + "openai-api-key", + "OPENAI_API_KEY", + Source::LiteralValue(String::new()), + ); + add_binding( + &mut application, + "tokenizer", + "TOKENIZER_PATH", + Source::LiteralValue("/models/tokenizer".to_owned()), + ); + add_binding( + &mut application, + "endpoint-token", + "ENDPOINT_TOKEN", + Source::ResolvedValueName("published-endpoint-url".to_owned()), + ); + + assert_eq!(messages(&application), Vec::::new()); + } + + #[test] + fn parameter_default_on_a_sensitive_variable_is_rejected_unless_empty() { + let mut application = application(); + add_text_parameter(&mut application, "admin-password", "changeme"); + add_text_parameter(&mut application, "upstream-token", ""); + add_binding( + &mut application, + "admin-password", + "ADMIN_PASSWORD", + Source::ParameterId("admin-password".to_owned()), + ); + add_binding( + &mut application, + "upstream-token", + "UPSTREAM_TOKEN", + Source::ParameterId("upstream-token".to_owned()), + ); + + assert_eq!( + messages(&application), + [ + "application \"fixture-application\" service \"web\" configuration binding \"admin-password\" sets a catalog value on sensitive variable \"ADMIN_PASSWORD\"; bind a secret parameter instead" + ] + ); + } + + fn messages(application: &dotcore_proto::dotcore::catalog::v1::Application) -> Vec { + validate_sensitive_literals(application) + .into_iter() + .map(|error| error.to_string()) + .collect() + } + + fn add_binding( + application: &mut dotcore_proto::dotcore::catalog::v1::Application, + binding_name: &str, + environment_variable: &str, + source: Source, + ) { + application.graph.as_mut().expect("graph").services[0] + .configuration_bindings + .push(ApplicationConfigurationBinding { + binding_name: binding_name.to_owned(), + environment_variable: environment_variable.to_owned(), + source: Some(source), + input: None, + enabled_by_parameter_id: None, + }); + } + + fn add_text_parameter( + application: &mut dotcore_proto::dotcore::catalog::v1::Application, + parameter_id: &str, + default_value: &str, + ) { + application + .graph + .as_mut() + .expect("graph") + .parameters + .push(ParameterDefinition { + parameter_id: parameter_id.to_owned(), + visibility: ParameterVisibility::Basic as i32, + parameter_type: Some(ParameterType::Text(TextParameterSchema { + default_value: Some(default_value.to_owned()), + ..Default::default() + })), + ..Default::default() + }); + } +} diff --git a/backend/catalog/src/validation/service.rs b/backend/catalog/src/validation/service.rs index f7cce03f..55a192ff 100644 --- a/backend/catalog/src/validation/service.rs +++ b/backend/catalog/src/validation/service.rs @@ -4,8 +4,9 @@ use std::collections::BTreeSet; use dotcore_domain::valid_id; use dotcore_proto::dotcore::catalog::v1::{ - ApplicationInferenceRouteProjection, ApplicationService, application_configuration_binding, - application_deployment_input, application_service_artifact::Kind as ServiceArtifactKind, + ApplicationInferenceRouteProjection, ApplicationSecretForm, ApplicationService, + application_configuration_binding, application_deployment_input, + application_service_artifact::Kind as ServiceArtifactKind, }; use super::{ @@ -145,6 +146,15 @@ fn validate_bindings(service: &ApplicationService, errors: &mut Vec bool { + fn allows_placement(self, placement: StoragePlacement) -> bool { match self { - Self::ApplicationService => { - matches!(scope, StorageScope::Deployment | StorageScope::Application) - } - Self::EngineRelease => { - matches!( - scope, - StorageScope::Deployment | StorageScope::EngineArtifact - ) - } + Self::ApplicationService => matches!( + placement, + StoragePlacement::Deployment | StoragePlacement::Application + ), + Self::EngineRelease => matches!( + placement, + StoragePlacement::Deployment | StoragePlacement::EngineArtifact + ), } } } @@ -86,16 +87,7 @@ pub(super) fn validate_storage_bindings( )), Ok(_) => {} } - match StorageScope::try_from(binding.scope) { - Ok(StorageScope::Unspecified) | Err(_) => errors.push(anyhow::anyhow!( - "{subject} storage binding {name:?} requires a scope" - )), - Ok(scope) if !owner.allows_scope(scope) => errors.push(anyhow::anyhow!( - "{subject} storage binding {name:?} cannot use scope {}", - scope.as_str_name() - )), - Ok(_) => {} - } + validate_placements(binding, owner, subject, errors); for variable in &binding.environment { if !valid_environment_variable(&variable.name) { errors.push(anyhow::anyhow!( @@ -118,6 +110,44 @@ pub(super) fn validate_storage_bindings( } } +/// Checks that a binding lists at least one placement, each specified, allowed by its owner and +/// listed once, and that an engine binding lists exactly one. +fn validate_placements( + binding: &StorageBinding, + owner: StorageOwner, + subject: &str, + errors: &mut Vec, +) { + let name = &binding.binding_name; + if binding.placements.is_empty() { + errors.push(anyhow::anyhow!( + "{subject} storage binding {name:?} requires a placement" + )); + } + if owner == StorageOwner::EngineRelease && binding.placements.len() > 1 { + errors.push(anyhow::anyhow!( + "{subject} storage binding {name:?} must declare exactly one placement" + )); + } + let mut seen = BTreeSet::new(); + for &placement in &binding.placements { + match StoragePlacement::try_from(placement) { + Ok(StoragePlacement::Unspecified) | Err(_) => errors.push(anyhow::anyhow!( + "{subject} storage binding {name:?} has an unspecified placement" + )), + Ok(placement) if !owner.allows_placement(placement) => errors.push(anyhow::anyhow!( + "{subject} storage binding {name:?} cannot use placement {}", + placement.as_str_name() + )), + Ok(placement) if !seen.insert(placement) => errors.push(anyhow::anyhow!( + "{subject} storage binding {name:?} lists placement {} twice", + placement.as_str_name() + )), + Ok(_) => {} + } + } +} + /// Whether a value is an absolute container path other than `/`, without empty, `.` or `..` /// components. pub(super) fn valid_container_path(value: &str) -> bool { @@ -147,10 +177,14 @@ mod tests { #[test] fn valid_bindings_may_nest_and_record_their_names() { let bindings = [ - binding("data", "/data", StorageScope::Deployment), + binding("data", "/data", StoragePlacement::Deployment), StorageBinding { environment: vec![variable("MODELS_DIR", ""), variable("CHECKPOINTS", "a/b")], - ..binding("models", "/data/models", StorageScope::Application) + placements: vec![ + StoragePlacement::Application as i32, + StoragePlacement::Deployment as i32, + ], + ..binding("models", "/data/models", StoragePlacement::Application) }, ]; let mut names = BTreeSet::new(); @@ -175,14 +209,14 @@ mod tests { #[test] fn rejects_invalid_or_duplicate_names_and_mount_paths() { let bindings = [ - binding("data", "/data", StorageScope::Deployment), - binding("data", "/other", StorageScope::Deployment), - binding("logs", "/data", StorageScope::Deployment), - binding("Bad_Name", "/bad", StorageScope::Deployment), - binding("relative", "data", StorageScope::Deployment), - binding("root", "/", StorageScope::Deployment), - binding("escape", "/data/../etc", StorageScope::Deployment), - binding("taken", "/taken", StorageScope::Deployment), + binding("data", "/data", StoragePlacement::Deployment), + binding("data", "/other", StoragePlacement::Deployment), + binding("logs", "/data", StoragePlacement::Deployment), + binding("Bad_Name", "/bad", StoragePlacement::Deployment), + binding("relative", "data", StoragePlacement::Deployment), + binding("root", "/", StoragePlacement::Deployment), + binding("escape", "/data/../etc", StoragePlacement::Deployment), + binding("taken", "/taken", StoragePlacement::Deployment), ]; let mut names = BTreeSet::from(["taken".to_owned()]); @@ -206,11 +240,11 @@ mod tests { } #[test] - fn rejects_missing_kinds_and_scopes() { + fn rejects_missing_kinds_and_placements() { let bindings = [StorageBinding { kind: 0, - scope: 0, - ..binding("data", "/data", StorageScope::Deployment) + placements: Vec::new(), + ..binding("data", "/data", StoragePlacement::Deployment) }]; assert_eq!( @@ -222,29 +256,29 @@ mod tests { ), [ "service storage binding \"data\" requires a kind", - "service storage binding \"data\" requires a scope", + "service storage binding \"data\" requires a placement", ] ); } #[test] - fn rejects_kinds_and_scopes_the_owner_cannot_use() { + fn rejects_kinds_and_placements_the_owner_cannot_use() { let application = [ - binding("cache", "/cache", StorageScope::EngineArtifact), + binding("cache", "/cache", StoragePlacement::EngineArtifact), StorageBinding { kind: StorageKind::Cache as i32, - ..binding("scratch", "/scratch", StorageScope::Deployment) + ..binding("scratch", "/scratch", StoragePlacement::Deployment) }, ]; let engine = [ - binding("data", "/data", StorageScope::Deployment), + binding("data", "/data", StoragePlacement::Deployment), StorageBinding { kind: StorageKind::Cache as i32, - ..binding("shared", "/shared", StorageScope::Application) + ..binding("shared", "/shared", StoragePlacement::Application) }, StorageBinding { kind: StorageKind::Cache as i32, - ..binding("compile", "/compile", StorageScope::EngineArtifact) + ..binding("compile", "/compile", StoragePlacement::EngineArtifact) }, ]; @@ -256,7 +290,7 @@ mod tests { &mut BTreeSet::new() ), [ - "service storage binding \"cache\" cannot use scope STORAGE_SCOPE_ENGINE_ARTIFACT", + "service storage binding \"cache\" cannot use placement STORAGE_PLACEMENT_ENGINE_ARTIFACT", "service storage binding \"scratch\" cannot use kind STORAGE_KIND_CACHE", ] ); @@ -269,9 +303,66 @@ mod tests { ), [ "service storage binding \"data\" cannot use kind STORAGE_KIND_DATA", - "service storage binding \"shared\" cannot use scope STORAGE_SCOPE_APPLICATION", + "service storage binding \"shared\" cannot use placement STORAGE_PLACEMENT_APPLICATION", + ] + ); + } + + #[test] + fn rejects_unspecified_or_repeated_placements_and_several_engine_placements() { + let placements = |placements: &[StoragePlacement]| { + placements + .iter() + .map(|&placement| placement as i32) + .collect() + }; + let application = [ + StorageBinding { + placements: placements(&[ + StoragePlacement::Deployment, + StoragePlacement::Unspecified, + ]), + ..binding("unspecified", "/unspecified", StoragePlacement::Deployment) + }, + StorageBinding { + placements: placements(&[ + StoragePlacement::Application, + StoragePlacement::Deployment, + StoragePlacement::Application, + ]), + ..binding("repeated", "/repeated", StoragePlacement::Deployment) + }, + ]; + let engine = [StorageBinding { + kind: StorageKind::Cache as i32, + placements: placements(&[ + StoragePlacement::Deployment, + StoragePlacement::EngineArtifact, + ]), + ..binding("cache", "/cache", StoragePlacement::Deployment) + }]; + + assert_eq!( + validate( + &application, + StorageOwner::ApplicationService, + &mut BTreeSet::new(), + &mut BTreeSet::new() + ), + [ + "service storage binding \"unspecified\" has an unspecified placement", + "service storage binding \"repeated\" lists placement STORAGE_PLACEMENT_APPLICATION twice", ] ); + assert_eq!( + validate( + &engine, + StorageOwner::EngineRelease, + &mut BTreeSet::new(), + &mut BTreeSet::new() + ), + ["service storage binding \"cache\" must declare exactly one placement"] + ); } #[test] @@ -287,11 +378,11 @@ mod tests { variable("PARENT", "a/../b"), variable("TRAILING", "a/"), ], - ..binding("data", "/data", StorageScope::Deployment) + ..binding("data", "/data", StoragePlacement::Deployment) }, StorageBinding { environment: vec![variable("CACHE_HOME", "other")], - ..binding("other", "/other", StorageScope::Deployment) + ..binding("other", "/other", StoragePlacement::Deployment) }, ]; let mut variables = BTreeSet::from(["TAKEN".to_owned()]); @@ -326,12 +417,12 @@ mod tests { errors.iter().map(ToString::to_string).collect() } - fn binding(name: &str, mount_path: &str, scope: StorageScope) -> StorageBinding { + fn binding(name: &str, mount_path: &str, placement: StoragePlacement) -> StorageBinding { StorageBinding { binding_name: name.to_owned(), mount_path: mount_path.to_owned(), kind: StorageKind::Data as i32, - scope: scope as i32, + placements: vec![placement as i32], environment: Vec::new(), } } diff --git a/backend/cli/src/application/mod.rs b/backend/cli/src/application/mod.rs index e291cd51..cd414610 100644 --- a/backend/cli/src/application/mod.rs +++ b/backend/cli/src/application/mod.rs @@ -7,6 +7,7 @@ //! stdout, for example `DOTCORE_APPLICATION_DEPLOYMENT_STATE=Ready`. mod list; +mod parameters; mod projection; mod show; @@ -19,6 +20,7 @@ use dotcore_proto::dotcore::{ RemoveApplicationDeploymentRequest, ResumeDeploymentRequest, follow_deployment_logs_response, }, + catalog::v1::StoragePlacement, deployment::v1::ExactPlacement, }; use futures_util::StreamExt; @@ -27,7 +29,7 @@ use tonic::Status; use crate::management::{ DeploymentGroup, Lifecycle, LogEntry, LogEvent, LogEvents, LogPage, LogPosition, LogSource, LogSourceState, LoggedDeployment, LogsArgs, ManagedDeployment, Manager, ManagerArgs, NodeNames, - change_lifecycle, deployment_logs, teardown, + change_lifecycle, deployment_logs, single_line, teardown, }; use list::{ApplicationListArgs, application_list}; use projection::{desired_state_label, state_label}; @@ -58,7 +60,8 @@ enum ApplicationSubcommand { /// Ask the manager to resume a paused application deployment; returns without waiting. Resume(LifecycleArgs), - /// Remove an application deployment, its workloads and its data after confirmation. + /// Remove an application deployment, its workloads and the data placed with it after + /// confirmation; data shared by the app is kept. Teardown(ApplicationTeardownArgs), /// Print the most recent log lines of an application deployment, on any node, and follow @@ -190,6 +193,39 @@ impl ManagedDeployment for ApplicationDeployment { fn desired_state_label(&self) -> &'static str { desired_state_label(self.desired_state()) } + + fn teardown_consequence(&self) -> String { + teardown_consequence(self) + } +} + +/// What tearing a deployment down does to its data: storage placed with the deployment is +/// deleted, storage shared by the app is kept, each listed by binding name. +fn teardown_consequence(deployment: &ApplicationDeployment) -> String { + let bindings = |placement: StoragePlacement| { + deployment + .storage_attachments + .iter() + .filter(|attachment| attachment.placement == placement as i32) + .map(|attachment| single_line(&attachment.binding_name)) + .collect::>() + .join(", ") + }; + let deleted = bindings(StoragePlacement::Deployment); + let kept = bindings(StoragePlacement::Application); + + let mut text = "This stops and removes its workloads".to_owned(); + if !deleted.is_empty() { + text.push_str(&format!(" and deletes this deployment's data ({deleted})")); + } + if kept.is_empty() { + text.push_str(". The catalog application is kept."); + } else { + text.push_str(&format!( + ". It keeps the data shared by the app ({kept}) and the catalog application." + )); + } + text } impl LoggedDeployment for ApplicationDeployment { @@ -307,3 +343,42 @@ fn node_of<'a>(deployment: &'a ApplicationDeployment, nodes: &'a NodeNames) -> & fn placement(deployment: &ApplicationDeployment) -> Option<&ExactPlacement> { deployment.placement.as_ref() } + +#[cfg(test)] +mod tests { + use dotcore_proto::dotcore::application::v1::ApplicationStorageAttachment; + + use super::*; + + #[test] + fn teardown_consequence_names_the_deleted_and_the_kept_data() { + let with_storage = |attachments: &[(&str, StoragePlacement)]| ApplicationDeployment { + storage_attachments: attachments + .iter() + .map(|(binding, placement)| ApplicationStorageAttachment { + service_name: "web".to_owned(), + binding_name: (*binding).to_owned(), + storage_name: format!("web-{binding}"), + placement: *placement as i32, + }) + .collect(), + ..Default::default() + }; + + assert_eq!( + teardown_consequence(&with_storage(&[])), + "This stops and removes its workloads. The catalog application is kept." + ); + assert_eq!( + teardown_consequence(&with_storage(&[ + ("data", StoragePlacement::Deployment), + ("models", StoragePlacement::Deployment), + ])), + "This stops and removes its workloads and deletes this deployment's data (data, models). The catalog application is kept." + ); + assert_eq!( + teardown_consequence(&with_storage(&[("models", StoragePlacement::Application)])), + "This stops and removes its workloads. It keeps the data shared by the app (models) and the catalog application." + ); + } +} diff --git a/backend/cli/src/application/parameters.rs b/backend/cli/src/application/parameters.rs new file mode 100644 index 00000000..e252da92 --- /dev/null +++ b/backend/cli/src/application/parameters.rs @@ -0,0 +1,29 @@ +//! The parameters of one application deployment, as rows an operator can read. + +use dotcore_proto::dotcore::application::v1::ApplicationDeployment; + +use crate::parameters::{ParameterInput, ParameterRow, definition_default, retained}; + +/// The deployment's parameters in catalog order. A parameter without a retained value shows its +/// default, and one with neither is left out. +pub(super) fn rows(deployment: &ApplicationDeployment, reveal_secrets: bool) -> Vec { + let Some(snapshot) = deployment.snapshot.as_ref() else { + return Vec::new(); + }; + let Some(graph) = snapshot.graph.as_ref() else { + return Vec::new(); + }; + + graph + .parameters + .iter() + .filter_map(|definition| { + ParameterInput { + definition, + value: retained(&snapshot.parameter_values, &definition.parameter_id), + default: definition_default(definition), + } + .row(reveal_secrets) + }) + .collect() +} diff --git a/backend/cli/src/application/show.rs b/backend/cli/src/application/show.rs index 2406a116..e6c3ff30 100644 --- a/backend/cli/src/application/show.rs +++ b/backend/cli/src/application/show.rs @@ -8,7 +8,7 @@ use clap::Args; use dotcore_proto::dotcore::application::v1::ApplicationDeployment; use super::{ - node_of, placement, + node_of, parameters, placement, projection::{ application_display_name, desired_state_label, endpoint, observation_stale, presentation_name, readiness, readiness_label, service_label, state_label, @@ -18,6 +18,7 @@ use super::{ use crate::{ management::{ManagerArgs, NodeNames, find_deployment, single_line}, node::write_records, + parameters::{ParameterRow, record_lines}, terminal::{Message, TerminalCondition, controlling_terminal_condition, open_terminal}, }; @@ -28,6 +29,10 @@ pub struct ApplicationShowArgs { #[arg(value_name = "DEPLOYMENT")] reference: String, + /// Print secret parameter values instead of masking them. + #[arg(long)] + reveal_secrets: bool, + #[command(flatten)] manager: ManagerArgs, } @@ -38,7 +43,7 @@ pub(crate) async fn application_show(arguments: ApplicationShowArgs) -> anyhow:: let deployment = find_deployment::(&manager, &arguments.reference).await?; let nodes = manager.node_names().await; - let details = Details::of(&deployment, &nodes); + let details = Details::of(&deployment, &nodes, arguments.reveal_secrets); if controlling_terminal_condition() == TerminalCondition::Interactive { let mut terminal = open_terminal(); @@ -47,6 +52,16 @@ pub(crate) async fn application_show(arguments: ApplicationShowArgs) -> anyhow:: details.name )))?; terminal.message(&Message::KeyValues(details.pairs()))?; + if !details.parameters.is_empty() { + terminal.message(&Message::Heading("Parameters".to_owned()))?; + terminal.message(&Message::KeyValues( + details + .parameters + .iter() + .map(|row| (row.name.clone(), row.value.clone())) + .collect(), + ))?; + } return Ok(()); } @@ -70,10 +85,11 @@ struct Details { route: String, failure: String, message: String, + parameters: Vec, } impl Details { - fn of(deployment: &ApplicationDeployment, nodes: &NodeNames) -> Self { + fn of(deployment: &ApplicationDeployment, nodes: &NodeNames, reveal_secrets: bool) -> Self { let state = state_label(deployment); let observed = deployment.state.as_ref(); let failure = observed @@ -140,6 +156,7 @@ impl Details { route: single_line(route), failure, message, + parameters: parameters::rows(deployment, reveal_secrets), } } @@ -183,7 +200,7 @@ impl Details { .map(|(name, state)| format!("{name}: {state}")) .collect::>() .join(", "); - format!( + let mut block = format!( "Application deployment\nDOTCORE_APPLICATION_DEPLOYMENT_ID={}\nDOTCORE_APPLICATION_DEPLOYMENT_NAME={}\nDOTCORE_APPLICATION_DEPLOYMENT_STATE={}\nDOTCORE_APPLICATION_DEPLOYMENT_DESIRED_STATE={}\nDOTCORE_APPLICATION_DEPLOYMENT_READINESS={}\nDOTCORE_APPLICATION_DEPLOYMENT_STALE={}\nDOTCORE_APPLICATION_DEPLOYMENT_NODE={}\nDOTCORE_APPLICATION_DEPLOYMENT_EXECUTION_TARGET={}\nDOTCORE_APPLICATION_DEPLOYMENT_APPLICATION={}\nDOTCORE_APPLICATION_DEPLOYMENT_SERVICES={services}\nDOTCORE_APPLICATION_DEPLOYMENT_ENDPOINT={}\nDOTCORE_APPLICATION_DEPLOYMENT_ROUTE={}\nDOTCORE_APPLICATION_DEPLOYMENT_FAILURE={}\nDOTCORE_APPLICATION_DEPLOYMENT_MESSAGE={}\n", self.id, self.name, @@ -198,6 +215,11 @@ impl Details { self.route, self.failure, self.message, - ) + ); + block.push_str(&record_lines( + "DOTCORE_APPLICATION_DEPLOYMENT_PARAMETER", + &self.parameters, + )); + block } } diff --git a/backend/cli/src/lib.rs b/backend/cli/src/lib.rs index ed73039f..97a858cb 100644 --- a/backend/cli/src/lib.rs +++ b/backend/cli/src/lib.rs @@ -30,6 +30,7 @@ mod hosts; mod management; mod model; mod node; +mod parameters; mod paths; mod runtime_confirmation; mod server; diff --git a/backend/cli/src/management/deployment.rs b/backend/cli/src/management/deployment.rs index 28852609..521bf119 100644 --- a/backend/cli/src/management/deployment.rs +++ b/backend/cli/src/management/deployment.rs @@ -46,6 +46,10 @@ pub(crate) trait ManagedDeployment: Addressable + Clone { /// The word for the desired state the manager drives the deployment toward. fn desired_state_label(&self) -> &'static str; + + /// What tearing the deployment down removes and keeps, asked after its name in the + /// confirmation. + fn teardown_consequence(&self) -> String; } /// The lifecycle change an operator asks for. @@ -123,7 +127,7 @@ pub(crate) async fn teardown( confirm_teardown(format!( "Tear down {}? {}", single_line(deployment.name()), - T::GROUP.teardown_consequence() + deployment.teardown_consequence() ))?; } diff --git a/backend/cli/src/management/mod.rs b/backend/cli/src/management/mod.rs index 6aef29ac..b2a137f7 100644 --- a/backend/cli/src/management/mod.rs +++ b/backend/cli/src/management/mod.rs @@ -266,17 +266,6 @@ impl DeploymentGroup { Self::Application => "DOTCORE_APPLICATION_DEPLOYMENT", } } - - /// What a teardown removes, asked after the deployment name in the confirmation. - pub(crate) fn teardown_consequence(self) -> &'static str { - match self { - Self::Model => "This removes the deployment and its running workload resources.", - Self::Application => { - "This stops and removes its workloads and deletes its application data. The \ - catalog application is kept." - } - } - } } /// A deployment the operator can name by its id or its name. diff --git a/backend/cli/src/model/mod.rs b/backend/cli/src/model/mod.rs index bfb2007e..5422de4d 100644 --- a/backend/cli/src/model/mod.rs +++ b/backend/cli/src/model/mod.rs @@ -7,6 +7,7 @@ //! stdout, for example `DOTCORE_MODEL_DEPLOYMENT_STATE=Ready`. mod list; +mod parameters; mod projection; mod show; @@ -185,6 +186,10 @@ impl ManagedDeployment for ModelDeployment { fn desired_state_label(&self) -> &'static str { desired_state_label(self.desired_state()) } + + fn teardown_consequence(&self) -> String { + "This removes the deployment and its running workload resources.".to_owned() + } } impl LoggedDeployment for ModelDeployment { diff --git a/backend/cli/src/model/parameters.rs b/backend/cli/src/model/parameters.rs new file mode 100644 index 00000000..7d80b024 --- /dev/null +++ b/backend/cli/src/model/parameters.rs @@ -0,0 +1,82 @@ +//! The engine parameters of one model deployment, as rows an operator can read. + +use dotcore_proto::dotcore::{ + catalog::v1::ParameterVisibility, + model::v1::{ModelDeployment, UpstreamModelDesiredConfiguration, model_deployment_snapshot}, +}; + +use crate::parameters::{ParameterInput, ParameterRow, definition_default, retained, secret_row}; + +const UPSTREAM_API_KEY_NAME: &str = "API key"; +const UPSTREAM_API_KEY_ID: &str = "api-key"; + +/// The deployment's parameters in engine schema order. Hidden parameters, by the profile's +/// visibility override else the engine's, are left out as in the console; a parameter without a +/// retained value shows its default, and one with neither is left out. An upstream deployment +/// shows its API key, when it has one, as a secret row. +pub(super) fn rows(deployment: &ModelDeployment, reveal_secrets: bool) -> Vec { + match deployment + .snapshot + .as_ref() + .and_then(|snapshot| snapshot.source.as_ref()) + { + Some(model_deployment_snapshot::Source::Local(local)) => { + let Some(schema) = local.engine_parameter_schema.as_ref() else { + return Vec::new(); + }; + let values = local + .engine_parameter_values + .as_ref() + .map_or(&[][..], |values| &values.values); + let defaults = local + .profile_engine_parameter_defaults + .as_ref() + .map_or(&[][..], |values| &values.values); + + schema + .parameters + .iter() + .filter(|definition| { + let overridden = local + .profile_engine_parameter_visibility_overrides + .iter() + .find(|entry| entry.parameter_id == definition.parameter_id) + .map(|entry| entry.visibility()); + overridden.unwrap_or(definition.visibility()) != ParameterVisibility::Hidden + }) + .filter_map(|definition| { + ParameterInput { + definition, + value: retained(values, &definition.parameter_id), + default: retained(defaults, &definition.parameter_id) + .cloned() + .or_else(|| definition_default(definition)), + } + .row(reveal_secrets) + }) + .collect() + } + _ => upstream_api_key( + deployment.upstream_desired_configuration.as_ref(), + reveal_secrets, + ) + .into_iter() + .collect(), + } +} + +fn upstream_api_key( + configuration: Option<&UpstreamModelDesiredConfiguration>, + reveal_secrets: bool, +) -> Option { + let key = configuration + .map(|configuration| configuration.api_key.as_str()) + .filter(|key| !key.is_empty())?; + + Some(secret_row( + UPSTREAM_API_KEY_ID, + UPSTREAM_API_KEY_NAME, + key, + reveal_secrets, + )) +} diff --git a/backend/cli/src/model/show.rs b/backend/cli/src/model/show.rs index ceda6307..16173f91 100644 --- a/backend/cli/src/model/show.rs +++ b/backend/cli/src/model/show.rs @@ -8,7 +8,7 @@ use clap::Args; use dotcore_proto::dotcore::model::v1::{ModelDeployment, model_deployment_snapshot}; use super::{ - node_of, placement, + node_of, parameters, placement, projection::{ desired_state_label, model_display_name, presentation_name, state_label, state_overridden, }, @@ -16,6 +16,7 @@ use super::{ use crate::{ management::{ManagerArgs, NodeNames, find_deployment, single_line}, node::write_records, + parameters::{ParameterRow, record_lines}, terminal::{Message, TerminalCondition, controlling_terminal_condition, open_terminal}, }; @@ -26,6 +27,10 @@ pub struct ModelShowArgs { #[arg(value_name = "DEPLOYMENT")] reference: String, + /// Print secret parameter values instead of masking them. + #[arg(long)] + reveal_secrets: bool, + #[command(flatten)] manager: ManagerArgs, } @@ -35,7 +40,7 @@ pub(crate) async fn model_show(arguments: ModelShowArgs) -> anyhow::Result<()> { let manager = arguments.manager.connect().await?; let deployment = find_deployment::(&manager, &arguments.reference).await?; let nodes = manager.node_names().await; - let details = Details::of(&deployment, &nodes); + let details = Details::of(&deployment, &nodes, arguments.reveal_secrets); if controlling_terminal_condition() == TerminalCondition::Interactive { let mut terminal = open_terminal(); @@ -44,6 +49,16 @@ pub(crate) async fn model_show(arguments: ModelShowArgs) -> anyhow::Result<()> { details.name )))?; terminal.message(&Message::KeyValues(details.pairs()))?; + if !details.parameters.is_empty() { + terminal.message(&Message::Heading("Parameters".to_owned()))?; + terminal.message(&Message::KeyValues( + details + .parameters + .iter() + .map(|row| (row.name.clone(), row.value.clone())) + .collect(), + ))?; + } return Ok(()); } @@ -65,10 +80,11 @@ struct Details { progress: String, failure: String, message: String, + parameters: Vec, } impl Details { - fn of(deployment: &ModelDeployment, nodes: &NodeNames) -> Self { + fn of(deployment: &ModelDeployment, nodes: &NodeNames, reveal_secrets: bool) -> Self { let state = state_label(deployment); let observed = deployment.state.as_ref(); let (profile, engine) = match deployment @@ -125,6 +141,7 @@ impl Details { progress, failure, message, + parameters: parameters::rows(deployment, reveal_secrets), } } @@ -152,7 +169,7 @@ impl Details { } fn record_block(&self) -> String { - format!( + let mut block = format!( "Model deployment\nDOTCORE_MODEL_DEPLOYMENT_ID={}\nDOTCORE_MODEL_DEPLOYMENT_NAME={}\nDOTCORE_MODEL_DEPLOYMENT_STATE={}\nDOTCORE_MODEL_DEPLOYMENT_DESIRED_STATE={}\nDOTCORE_MODEL_DEPLOYMENT_NODE={}\nDOTCORE_MODEL_DEPLOYMENT_EXECUTION_TARGET={}\nDOTCORE_MODEL_DEPLOYMENT_MODEL={}\nDOTCORE_MODEL_DEPLOYMENT_PROFILE={}\nDOTCORE_MODEL_DEPLOYMENT_ENGINE={}\nDOTCORE_MODEL_DEPLOYMENT_PROGRESS={}\nDOTCORE_MODEL_DEPLOYMENT_FAILURE={}\nDOTCORE_MODEL_DEPLOYMENT_MESSAGE={}\n", self.id, self.name, @@ -166,6 +183,11 @@ impl Details { self.progress, self.failure, self.message, - ) + ); + block.push_str(&record_lines( + "DOTCORE_MODEL_DEPLOYMENT_PARAMETER", + &self.parameters, + )); + block } } diff --git a/backend/cli/src/parameters.rs b/backend/cli/src/parameters.rs new file mode 100644 index 00000000..34d1ae97 --- /dev/null +++ b/backend/cli/src/parameters.rs @@ -0,0 +1,144 @@ +//! Parameters of a deployment as rows an operator can read. +//! +//! Each row carries the parameter's display name and its value. Secret values are masked unless +//! the caller asks to reveal them. + +use dotcore_proto::dotcore::catalog::v1::{ + ParameterDefinition, ParameterValue, parameter_definition, parameter_value, +}; + +use crate::management::single_line; + +const SECRET_MASK: &str = "********"; +const EMPTY_TEXT: &str = "Empty"; + +/// One shown parameter. +pub(crate) struct ParameterRow { + /// Stable parameter identity. + pub(crate) id: String, + /// Human-readable parameter label. + pub(crate) name: String, + /// The value as shown. + pub(crate) value: String, +} + +/// What a deployment retains about one parameter, from which its row is built. +pub(crate) struct ParameterInput<'a> { + pub(crate) definition: &'a ParameterDefinition, + /// The retained value, if any. + pub(crate) value: Option<&'a parameter_value::Value>, + /// The value the deployment's owner defaults to, shown when nothing is retained. + pub(crate) default: Option, +} + +impl ParameterInput<'_> { + /// The row of this parameter. A parameter without a retained value shows its default, and one + /// with neither is left out. + pub(crate) fn row(&self, reveal_secrets: bool) -> Option { + let definition = self.definition; + let value = self.value.or(self.default.as_ref())?; + let is_secret = matches!( + definition.parameter_type, + Some(parameter_definition::ParameterType::Secret(_)) + ); + + let text = if is_secret && !reveal_secrets { + SECRET_MASK.to_owned() + } else { + match single_line(&display(value, definition)) { + empty if empty.is_empty() => EMPTY_TEXT.to_owned(), + text => text, + } + }; + Some(ParameterRow { + id: definition.parameter_id.clone(), + name: single_line(&definition.display_name), + value: text, + }) + } +} + +/// The row of a secret that has no parameter definition, masked unless revealed. +pub(crate) fn secret_row(id: &str, name: &str, secret: &str, reveal_secrets: bool) -> ParameterRow { + ParameterRow { + id: id.to_owned(), + name: name.to_owned(), + value: if reveal_secrets { + single_line(secret) + } else { + SECRET_MASK.to_owned() + }, + } +} + +/// The retained value of a parameter, if it has one. +pub(crate) fn retained<'a>( + values: &'a [ParameterValue], + parameter_id: &str, +) -> Option<&'a parameter_value::Value> { + values + .iter() + .find(|value| value.parameter_id == parameter_id) + .and_then(|value| value.value.as_ref()) +} + +/// The default the parameter's own definition declares; secrets have none. +pub(crate) fn definition_default( + definition: &ParameterDefinition, +) -> Option { + use parameter_definition::ParameterType; + + match definition.parameter_type.as_ref()? { + ParameterType::Boolean(schema) => schema.default_value.map(parameter_value::Value::Boolean), + ParameterType::UnsignedInteger(schema) => schema + .default_value + .map(parameter_value::Value::UnsignedInteger), + ParameterType::StringEnum(schema) => schema + .default_value + .clone() + .map(parameter_value::Value::String), + ParameterType::Text(schema) => schema + .default_value + .clone() + .map(parameter_value::Value::String), + ParameterType::Secret(_) => None, + } +} + +/// The record key suffix of a parameter: its id upper-cased, with every non-alphanumeric +/// character replaced by an underscore. +pub(crate) fn record_key(id: &str) -> String { + id.chars() + .map(|character| { + if character.is_ascii_alphanumeric() { + character.to_ascii_uppercase() + } else { + '_' + } + }) + .collect() +} + +/// One record line per row under the given key prefix. +pub(crate) fn record_lines(prefix: &str, rows: &[ParameterRow]) -> String { + rows.iter() + .map(|row| format!("{prefix}_{}={}\n", record_key(&row.id), row.value)) + .collect() +} + +fn display(value: ¶meter_value::Value, definition: &ParameterDefinition) -> String { + match value { + parameter_value::Value::Boolean(value) => value.to_string(), + parameter_value::Value::UnsignedInteger(value) => { + match definition.parameter_type.as_ref() { + Some(parameter_definition::ParameterType::UnsignedInteger(schema)) + if !schema.unit.is_empty() => + { + format!("{value} {}", schema.unit) + } + _ => value.to_string(), + } + } + parameter_value::Value::String(value) => value.clone(), + } +} diff --git a/backend/cli/src/workload/mod.rs b/backend/cli/src/workload/mod.rs index b4a5f87d..1a3c3187 100644 --- a/backend/cli/src/workload/mod.rs +++ b/backend/cli/src/workload/mod.rs @@ -31,8 +31,8 @@ use dotcore_node_hardware::vram::{ }; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterValue, EngineParameterValues, ModelOptionParameterValue, ModelOptionValues, - engine_parameter_value, model_option_parameter_value, + ModelOptionParameterValue, ModelOptionValues, ParameterValue, ParameterValues, + model_option_parameter_value, parameter_value, }, telemetry::v1::WorkloadResourceRecord, }; @@ -290,7 +290,7 @@ async fn launch(paths: &RootPathArgs, args: LaunchArgs) -> Result<()> { merge_parameter_overrides( configuration .engine_parameter_values - .get_or_insert_with(EngineParameterValues::default), + .get_or_insert_with(ParameterValues::default), args.engine_parameter .into_iter() .map(parse_parameter) @@ -831,10 +831,7 @@ fn accelerator_device_ids( ] } -fn merge_parameter_overrides( - defaults: &mut EngineParameterValues, - overrides: Vec, -) { +fn merge_parameter_overrides(defaults: &mut ParameterValues, overrides: Vec) { for parameter in overrides { match defaults .values @@ -926,7 +923,7 @@ fn merge_option_overrides( } } -fn parse_parameter(value: String) -> Result { +fn parse_parameter(value: String) -> Result { let (id, typed) = value .split_once('=') .ok_or_else(|| anyhow::anyhow!("engine parameter must be ID=TYPE:VALUE"))?; @@ -934,12 +931,12 @@ fn parse_parameter(value: String) -> Result { .split_once(':') .ok_or_else(|| anyhow::anyhow!("engine parameter must be ID=TYPE:VALUE"))?; let value = match kind { - "unsigned" => engine_parameter_value::Value::UnsignedInteger( + "unsigned" => parameter_value::Value::UnsignedInteger( raw.parse().context("invalid unsigned parameter")?, ), _ => bail!("unsupported engine parameter type: {kind}"), }; - Ok(EngineParameterValue { + Ok(ParameterValue { parameter_id: id.to_owned(), value: Some(value), }) @@ -972,8 +969,8 @@ mod tests { }; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterValue, EngineParameterValues, ModelOptionParameterValue, - ModelOptionValues, engine_parameter_value, model_option_parameter_value, + ModelOptionParameterValue, ModelOptionValues, ParameterValue, ParameterValues, + model_option_parameter_value, parameter_value, }, telemetry::v1::WorkloadResourceRecord, }; @@ -982,7 +979,7 @@ mod tests { use super::{ CommonArgs, DevSubcommand, HandleArgs, Runtime, Transport, WorkloadAction, accelerator_device_ids, accounting_payload, gpu_memory_payload, handle_root, - merge_option_overrides, merge_parameter_overrides, parse_option_overrides, + merge_option_overrides, merge_parameter_overrides, parse_option_overrides, parse_parameter, record_measurement, resolve_handle, vram_payload, write_log_content_to, }; use crate::RootPathArgs; @@ -1508,16 +1505,16 @@ mod tests { #[test] fn explicit_parameter_values_override_profile_defaults() { - let mut values = EngineParameterValues { + let mut values = ParameterValues { values: vec![unsigned_parameter("total-context", 30_000)], }; merge_parameter_overrides( &mut values, - vec![ - unsigned_parameter("total-context", 8_192), - unsigned_parameter("new-parameter", 7), - ], + ["total-context=unsigned:8192", "new-parameter=unsigned:7"] + .into_iter() + .map(|flag| parse_parameter(flag.to_owned()).expect("engine parameter flag")) + .collect(), ); assert_eq!( @@ -1566,10 +1563,10 @@ mod tests { } } - fn unsigned_parameter(id: &str, value: u64) -> EngineParameterValue { - EngineParameterValue { + fn unsigned_parameter(id: &str, value: u64) -> ParameterValue { + ParameterValue { parameter_id: id.to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(value)), + value: Some(parameter_value::Value::UnsignedInteger(value)), } } } diff --git a/backend/cli/tests/application_contract.rs b/backend/cli/tests/application_contract.rs index 9ee6348b..3ca6b2ef 100644 --- a/backend/cli/tests/application_contract.rs +++ b/backend/cli/tests/application_contract.rs @@ -2,14 +2,19 @@ use std::process::{Command, Output, Stdio}; +use dotcore_proto::dotcore::catalog::v1::{ + ApplicationGraph, BooleanParameterSchema, ParameterDefinition, ParameterValue, + SecretParameterSchema, StoragePlacement, StringEnumParameterSchema, TextParameterSchema, + UnsignedIntegerParameterSchema, parameter_definition::ParameterType, parameter_value::Value, +}; use dotcore_proto::{ dotcore::application::v1::{ ApplicationDeployment, ApplicationDeploymentDesiredState, ApplicationDeploymentFailure, ApplicationEndpoint, ApplicationLifecycleState, ApplicationReadinessState, ApplicationSelectedInferenceRoute, ApplicationServiceLifecycleState, - ApplicationServiceObservation, DeploymentLogEntry, DeploymentLogResumePosition, - DeploymentLogSource, DeploymentLogSourceEvent, DeploymentLogSourceState, - FollowDeploymentLogsResponse, ReadDeploymentLogsResponse, + ApplicationServiceObservation, ApplicationStorageAttachment, DeploymentLogEntry, + DeploymentLogResumePosition, DeploymentLogSource, DeploymentLogSourceEvent, + DeploymentLogSourceState, FollowDeploymentLogsResponse, ReadDeploymentLogsResponse, follow_deployment_logs_response::Event, }, google::protobuf::Timestamp, @@ -696,7 +701,7 @@ fn a_terminal_teardown_warns_about_data_and_removes_when_accepted() { .join(" "); assert!( screen.contains( - "Tear down Chat? This stops and removes its workloads and deletes its application data. The catalog application is kept." + "Tear down Chat? This stops and removes its workloads and deletes this deployment's data (data). It keeps the data shared by the app (models) and the catalog application." ), "{}", run.screen() @@ -731,6 +736,176 @@ fn a_terminal_teardown_cancels_by_default_with_exit_code_130() { ); } +#[test] +fn show_prints_parameters_in_catalog_order_with_secrets_masked() { + let manager = FakeManager::start(FakeState::with_applications(vec![ + deployment_with_parameters(), + ])); + + let output = run(&manager, &["show", "Chat"]); + + assert_eq!(text(&output.stderr), ""); + assert_eq!(output.status.code(), Some(0)); + let stdout = text(&output.stdout); + assert!(!stdout.contains("s3cretPassw0rd"), "{stdout}"); + assert_eq!(parameter_lines(&stdout), PARAMETER_LINES_MASKED); +} + +#[test] +fn show_reveals_secret_parameters_with_the_flag() { + let manager = FakeManager::start(FakeState::with_applications(vec![ + deployment_with_parameters(), + ])); + + let output = run(&manager, &["show", "Chat", "--reveal-secrets"]); + + assert_eq!(output.status.code(), Some(0)); + let revealed = PARAMETER_LINES_MASKED.map(|line| line.replace("********", "s3cretPassw0rd")); + assert_eq!(parameter_lines(&text(&output.stdout)), revealed); +} + +#[test] +fn help_describes_the_reveal_flag() { + let manager = FakeManager::start(FakeState::with_applications(Vec::new())); + + let output = run(&manager, &["show", "--help"]); + + let help = text(&output.stdout); + let line = help + .lines() + .find(|line| line.contains("--reveal-secrets")) + .unwrap_or_else(|| panic!("no flag in help\n{help}")); + assert_eq!( + line.split_whitespace().collect::>().join(" "), + "--reveal-secrets Print secret parameter values instead of masking them" + ); +} + +#[test] +fn a_terminal_shows_parameters_under_their_own_heading() { + let manager = FakeManager::start(FakeState::with_applications(vec![ + deployment_with_parameters(), + ])); + + let Some(run) = run_in_pty( + terminal_command(&manager, &["show", "Chat"]), + Session::new([]), + ) else { + return; + }; + + assert_eq!(run.code(), Some(0), "screen:\n{}", run.screen()); + let lines = run.screen().lines().map(str::trim).collect::>(); + let start = lines + .iter() + .position(|line| *line == "Parameters") + .unwrap_or_else(|| panic!("no Parameters heading\n{}", run.screen())); + let rows = lines[start + 1..] + .iter() + .map(|line| line.split_whitespace().collect::>().join(" ")) + .filter(|line| !line.is_empty()) + .collect::>(); + assert_eq!( + rows, + [ + "Web search true", + "Memory limit 2048 MB", + "Mode fast", + "Welcome note Empty", + "Admin password ********", + ] + ); +} + +const PARAMETER_LINES_MASKED: [&str; 5] = [ + "DOTCORE_APPLICATION_DEPLOYMENT_PARAMETER_WEB_SEARCH=true", + "DOTCORE_APPLICATION_DEPLOYMENT_PARAMETER_MEMORY_LIMIT=2048 MB", + "DOTCORE_APPLICATION_DEPLOYMENT_PARAMETER_MODE=fast", + "DOTCORE_APPLICATION_DEPLOYMENT_PARAMETER_WELCOME_NOTE=Empty", + "DOTCORE_APPLICATION_DEPLOYMENT_PARAMETER_ADMIN_PASSWORD=********", +]; + +/// The parameter lines of a record block. +fn parameter_lines(stdout: &str) -> Vec<&str> { + stdout + .lines() + .filter(|line| line.starts_with("DOTCORE_APPLICATION_DEPLOYMENT_PARAMETER_")) + .collect() +} + +/// A deployment with one parameter of each kind: a default boolean, a non-default integer with a +/// unit, an enum, an empty text and a secret. +fn deployment_with_parameters() -> ApplicationDeployment { + let definition = |id: &str, name: &str, parameter_type| ParameterDefinition { + parameter_id: id.to_owned(), + display_name: name.to_owned(), + parameter_type: Some(parameter_type), + ..Default::default() + }; + let value = |id: &str, value| ParameterValue { + parameter_id: id.to_owned(), + value: Some(value), + }; + + let mut deployment = local_application_deployment("app-1", "Chat", Ready, Present); + let snapshot = deployment.snapshot.as_mut().expect("seeded snapshot"); + snapshot.graph = Some(ApplicationGraph { + parameters: vec![ + definition( + "web-search", + "Web search", + ParameterType::Boolean(BooleanParameterSchema { + default_value: Some(true), + }), + ), + definition( + "memory-limit", + "Memory limit", + ParameterType::UnsignedInteger(UnsignedIntegerParameterSchema { + minimum: 512, + maximum: 8192, + unit: "MB".to_owned(), + default_value: Some(1024), + }), + ), + definition( + "mode", + "Mode", + ParameterType::StringEnum(StringEnumParameterSchema { + values: vec!["fast".to_owned(), "careful".to_owned()], + default_value: Some("careful".to_owned()), + }), + ), + definition( + "welcome-note", + "Welcome note", + ParameterType::Text(TextParameterSchema { + minimum_length: 0, + maximum_length: 100, + default_value: None, + }), + ), + definition( + "admin-password", + "Admin password", + ParameterType::Secret(SecretParameterSchema { + generated_length: 32, + minimum_length: 8, + }), + ), + ], + ..Default::default() + }); + snapshot.parameter_values = vec![ + value("web-search", Value::Boolean(true)), + value("memory-limit", Value::UnsignedInteger(2048)), + value("mode", Value::String("fast".to_owned())), + value("welcome-note", Value::String(String::new())), + value("admin-password", Value::String("s3cretPassw0rd".to_owned())), + ]; + deployment +} + /// Seeds one deployment in each state the commands present differently. fn mixed_deployments() -> Vec { let mut ready = local_application_deployment("app-1", "Chat", Ready, Present); @@ -747,6 +922,17 @@ fn mixed_deployments() -> Vec { inference_route_id: "route-1".to_owned(), route_name: "gemma".to_owned(), }); + ready.storage_attachments = [ + ("data", StoragePlacement::Deployment), + ("models", StoragePlacement::Application), + ] + .map(|(binding, placement)| ApplicationStorageAttachment { + service_name: "web".to_owned(), + binding_name: binding.to_owned(), + storage_name: format!("web-{binding}"), + placement: placement as i32, + }) + .to_vec(); let mut starting = local_application_deployment("app-2", "Notes", Starting, Present); let state = starting.state.as_mut().expect("seeded state"); diff --git a/backend/cli/tests/model_contract.rs b/backend/cli/tests/model_contract.rs index cb54d0f3..a0fcf261 100644 --- a/backend/cli/tests/model_contract.rs +++ b/backend/cli/tests/model_contract.rs @@ -3,12 +3,21 @@ use std::process::{Command, Output, Stdio}; use dotcore_proto::{ - dotcore::model::v1::{ - DeploymentLogEntry, DeploymentLogResumePosition, DeploymentLogSource, - DeploymentLogSourceEvent, DeploymentLogSourceState, FollowDeploymentLogsResponse, - ModelDeployment, ModelDeploymentDesiredState, ModelDeploymentFailure, - ModelDeploymentProgress, ModelDeploymentStatus, ReadDeploymentLogsResponse, - follow_deployment_logs_response::Event, + dotcore::{ + catalog::v1::{ + BooleanParameterSchema, EngineParameterSchema, ParameterDefinition, ParameterValue, + ParameterValues, ParameterVisibility, ParameterVisibilityOverride, + SecretParameterSchema, UnsignedIntegerParameterSchema, + parameter_definition::ParameterType, parameter_value::Value, + }, + model::v1::{ + DeploymentLogEntry, DeploymentLogResumePosition, DeploymentLogSource, + DeploymentLogSourceEvent, DeploymentLogSourceState, FollowDeploymentLogsResponse, + ModelDeployment, ModelDeploymentDesiredState, ModelDeploymentFailure, + ModelDeploymentProgress, ModelDeploymentSnapshot, ModelDeploymentStatus, + ReadDeploymentLogsResponse, ResolvedSizing, UpstreamModelDesiredConfiguration, + follow_deployment_logs_response::Event, model_deployment_snapshot::Source, + }, }, google::protobuf::Timestamp, }; @@ -777,6 +786,226 @@ fn a_terminal_shows_a_failure_as_a_verdict_with_next_steps() { } } +#[test] +fn show_prints_visible_parameters_in_schema_order_with_secrets_masked() { + let manager = FakeManager::start(FakeState::with_models(vec![deployment_with_parameters()])); + + let output = run(&manager, &["show", "Gemma"]); + + assert_eq!(text(&output.stderr), ""); + assert_eq!(output.status.code(), Some(0)); + let stdout = text(&output.stdout); + assert!(!stdout.contains("s3cretKey"), "{stdout}"); + assert_eq!(parameter_lines(&stdout), PARAMETER_LINES_MASKED); +} + +#[test] +fn show_reveals_secret_parameters_with_the_flag() { + let manager = FakeManager::start(FakeState::with_models(vec![deployment_with_parameters()])); + + let output = run(&manager, &["show", "Gemma", "--reveal-secrets"]); + + assert_eq!(output.status.code(), Some(0)); + let revealed = PARAMETER_LINES_MASKED.map(|line| line.replace("********", "s3cretKey")); + assert_eq!(parameter_lines(&text(&output.stdout)), revealed); +} + +#[test] +fn show_prints_the_upstream_api_key_masked_unless_revealed() { + let mut upstream = local_model_deployment("model-9", "Remote", Ready, Present); + upstream.snapshot = Some(ModelDeploymentSnapshot { + source: Some(Source::Upstream(Default::default())), + }); + upstream.upstream_desired_configuration = Some(UpstreamModelDesiredConfiguration { + api_key: "sk-remote".to_owned(), + ..Default::default() + }); + let manager = FakeManager::start(FakeState::with_models(vec![upstream])); + + let masked = text(&run(&manager, &["show", "Remote"]).stdout); + let revealed = text(&run(&manager, &["show", "Remote", "--reveal-secrets"]).stdout); + + assert!(!masked.contains("sk-remote"), "{masked}"); + assert_eq!( + parameter_lines(&masked), + ["DOTCORE_MODEL_DEPLOYMENT_PARAMETER_API_KEY=********"] + ); + assert_eq!( + parameter_lines(&revealed), + ["DOTCORE_MODEL_DEPLOYMENT_PARAMETER_API_KEY=sk-remote"] + ); +} + +#[test] +fn help_describes_the_reveal_flag() { + let manager = FakeManager::start(FakeState::with_models(Vec::new())); + + let output = run(&manager, &["show", "--help"]); + + let help = text(&output.stdout); + let line = help + .lines() + .find(|line| line.contains("--reveal-secrets")) + .unwrap_or_else(|| panic!("no flag in help\n{help}")); + assert_eq!( + line.split_whitespace().collect::>().join(" "), + "--reveal-secrets Print secret parameter values instead of masking them" + ); +} + +#[test] +fn a_terminal_shows_parameters_under_their_own_heading() { + let manager = FakeManager::start(FakeState::with_models(vec![deployment_with_parameters()])); + + let Some(run) = run_in_pty( + terminal_command(&manager, &["show", "Gemma"]), + Session::new([]), + ) else { + return; + }; + + assert_eq!(run.code(), Some(0), "screen:\n{}", run.screen()); + let lines = run.screen().lines().map(str::trim).collect::>(); + let start = lines + .iter() + .position(|line| *line == "Parameters") + .unwrap_or_else(|| panic!("no Parameters heading\n{}", run.screen())); + let rows = lines[start + 1..] + .iter() + .map(|line| line.split_whitespace().collect::>().join(" ")) + .filter(|line| !line.is_empty()) + .collect::>(); + assert_eq!( + rows, + [ + "Flash attention true", + "Parallel requests 4", + "Expert cap 24576 MiB", + "Tuned flag false", + "API key ********", + ] + ); +} + +const PARAMETER_LINES_MASKED: [&str; 5] = [ + "DOTCORE_MODEL_DEPLOYMENT_PARAMETER_FLASH_ATTENTION=true", + "DOTCORE_MODEL_DEPLOYMENT_PARAMETER_PARALLEL_REQUESTS=4", + "DOTCORE_MODEL_DEPLOYMENT_PARAMETER_EXPERT_CAP=24576 MiB", + "DOTCORE_MODEL_DEPLOYMENT_PARAMETER_TUNED_FLAG=false", + "DOTCORE_MODEL_DEPLOYMENT_PARAMETER_API_KEY=********", +]; + +/// The parameter lines of a record block. +fn parameter_lines(stdout: &str) -> Vec<&str> { + stdout + .lines() + .filter(|line| line.starts_with("DOTCORE_MODEL_DEPLOYMENT_PARAMETER_")) + .collect() +} + +/// A local deployment whose schema has a default flag, a plain integer, a sized integer, a flag +/// the profile makes visible, a secret and two hidden parameters (one hidden by the +/// engine, one by the profile). +fn deployment_with_parameters() -> ModelDeployment { + let definition = |id: &str, name: &str, visibility, parameter_type| ParameterDefinition { + parameter_id: id.to_owned(), + display_name: name.to_owned(), + visibility: visibility as i32, + parameter_type: Some(parameter_type), + ..Default::default() + }; + let integer = |default_value| { + ParameterType::UnsignedInteger(UnsignedIntegerParameterSchema { + minimum: 1, + maximum: 100_000, + unit: String::new(), + default_value, + }) + }; + let flag = |default_value| ParameterType::Boolean(BooleanParameterSchema { default_value }); + let value = |id: &str, value| ParameterValue { + parameter_id: id.to_owned(), + value: Some(value), + }; + use ParameterVisibility::{Advanced, Basic, Hidden}; + + let mut deployment = local_model_deployment("model-1", "Gemma", Ready, Present); + let Some(Source::Local(local)) = deployment + .snapshot + .as_mut() + .and_then(|snapshot| snapshot.source.as_mut()) + else { + unreachable!("seeded local snapshot"); + }; + let mut expert_cap = definition("expert-cap", "Expert cap", Advanced, integer(None)); + if let Some(ParameterType::UnsignedInteger(schema)) = expert_cap.parameter_type.as_mut() { + schema.unit = "MiB".to_owned(); + } + local.engine_parameter_schema = Some(EngineParameterSchema { + parameters: vec![ + definition( + "flash-attention", + "Flash attention", + Basic, + flag(Some(false)), + ), + definition( + "parallel-requests", + "Parallel requests", + Basic, + integer(Some(1)), + ), + expert_cap, + definition("engine-hidden", "Engine hidden", Hidden, integer(Some(7))), + definition("profile-hidden", "Profile hidden", Basic, integer(Some(8))), + definition("tuned-flag", "Tuned flag", Hidden, flag(Some(true))), + definition( + "api-key", + "API key", + Basic, + ParameterType::Secret(SecretParameterSchema { + generated_length: 32, + minimum_length: 8, + }), + ), + ], + ..Default::default() + }); + local.engine_parameter_values = Some(ParameterValues { + values: vec![ + value("flash-attention", Value::Boolean(true)), + value("parallel-requests", Value::UnsignedInteger(4)), + value("expert-cap", Value::UnsignedInteger(24576)), + value("engine-hidden", Value::UnsignedInteger(7)), + value("profile-hidden", Value::UnsignedInteger(8)), + value("tuned-flag", Value::Boolean(false)), + value("api-key", Value::String("s3cretKey".to_owned())), + ], + }); + local.profile_engine_parameter_defaults = Some(ParameterValues { + values: vec![ + value("flash-attention", Value::Boolean(true)), + value("parallel-requests", Value::UnsignedInteger(1)), + value("tuned-flag", Value::Boolean(true)), + ], + }); + local.profile_engine_parameter_visibility_overrides = vec![ + ParameterVisibilityOverride { + parameter_id: "profile-hidden".to_owned(), + visibility: Hidden as i32, + }, + ParameterVisibilityOverride { + parameter_id: "tuned-flag".to_owned(), + visibility: Basic as i32, + }, + ]; + local.sizing = Some(ResolvedSizing { + derived_engine_parameter_ids: vec!["expert-cap".to_owned()], + ..Default::default() + }); + deployment +} + /// Seeds one deployment in each state the commands present differently. fn mixed_deployments() -> Vec { let mut downloading = local_model_deployment("model-2", "Qwen", Downloading, Present); diff --git a/backend/cli/tests/support/simulated_manager.rs b/backend/cli/tests/support/simulated_manager.rs index a9217942..0eca483b 100644 --- a/backend/cli/tests/support/simulated_manager.rs +++ b/backend/cli/tests/support/simulated_manager.rs @@ -115,6 +115,7 @@ impl SimulatedManager { expected_inference_route_generation: 0, application_deployment_name: Some(name.to_owned()), parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .unwrap_or_else(|status| { @@ -325,7 +326,7 @@ graph: - { operatingSystem: "linux", architecture: "amd64" } - { operatingSystem: "linux", architecture: "arm64" } ports: [{ portName: "http", portNumber: 8080 }] - storageBindings: [{ bindingName: "data", mountPath: "/data", kind: "STORAGE_KIND_DATA", scope: "STORAGE_SCOPE_DEPLOYMENT" }] + storageBindings: [{ bindingName: "data", mountPath: "/data", kind: "STORAGE_KIND_DATA", placements: ["STORAGE_PLACEMENT_DEPLOYMENT"] }] readiness: { httpGet: { portName: "http", path: "/health" } } primaryEndpoint: serviceName: "web" diff --git a/backend/core/Cargo.toml b/backend/core/Cargo.toml index 8452514c..8346db6c 100644 --- a/backend/core/Cargo.toml +++ b/backend/core/Cargo.toml @@ -12,6 +12,7 @@ rust-version = "1.99" [dependencies] anyhow.workspace = true base64.workspace = true +bcrypt = "0.19.3" dotcore-artifact.workspace = true dotcore-catalog.workspace = true dotcore-domain.workspace = true diff --git a/backend/core/src/application/binding.rs b/backend/core/src/application/binding.rs new file mode 100644 index 00000000..7398bbcd --- /dev/null +++ b/backend/core/src/application/binding.rs @@ -0,0 +1,57 @@ +//! Whether an application configuration or secret binding reaches its service. +//! +//! A binding may be enabled by a boolean parameter of its graph: the service receives it only +//! when that parameter's resolved value is true, and otherwise the variable is not set at all. +//! Creation, launch and the retained declaration checks all decide through this one rule. + +use dotcore_proto::dotcore::catalog::v1::{ParameterValue, parameter_value::Value}; + +/// Returns whether a binding enabled by `enabled_by_parameter_id` reaches its service, given a +/// deployment's resolved parameter values. A binding without an enabling parameter always does. +/// Returns `None` when the enabling parameter has no boolean value. +pub(super) fn binding_enabled( + enabled_by_parameter_id: Option<&str>, + parameter_values: &[ParameterValue], +) -> Option { + let Some(parameter_id) = enabled_by_parameter_id else { + return Some(true); + }; + match parameter_values + .iter() + .find(|value| value.parameter_id == parameter_id)? + .value + { + Some(Value::Boolean(enabled)) => Some(enabled), + _ => None, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn binding_enabled_follows_its_boolean_parameter() { + let values = [ + boolean("login", true), + boolean("signup", false), + ParameterValue { + parameter_id: "mode".to_owned(), + value: Some(Value::String("auto".to_owned())), + }, + ]; + + assert_eq!(binding_enabled(None, &values), Some(true)); + assert_eq!(binding_enabled(Some("login"), &values), Some(true)); + assert_eq!(binding_enabled(Some("signup"), &values), Some(false)); + assert_eq!(binding_enabled(Some("mode"), &values), None); + assert_eq!(binding_enabled(Some("missing"), &values), None); + } + + fn boolean(parameter_id: &str, value: bool) -> ParameterValue { + ParameterValue { + parameter_id: parameter_id.to_owned(), + value: Some(Value::Boolean(value)), + } + } +} diff --git a/backend/core/src/application/controller/errors.rs b/backend/core/src/application/controller/errors.rs index 68fe8fe7..5cd59588 100644 --- a/backend/core/src/application/controller/errors.rs +++ b/backend/core/src/application/controller/errors.rs @@ -23,6 +23,13 @@ pub enum ApplicationLifecycleError { reason: ParameterRejection, }, + #[error("Application storage binding {binding_name:?} of service {service_name:?} {reason}")] + InvalidStorageChoice { + service_name: String, + binding_name: String, + reason: StorageChoiceRejection, + }, + #[error("Application deployment was not found")] NotFound, @@ -92,6 +99,19 @@ pub enum ParameterRejection { InvalidValue(anyhow::Error), } +/// Why a deployment creation request's storage choice was rejected. +#[derive(Debug, thiserror::Error)] +pub enum StorageChoiceRejection { + #[error("is not declared by the application")] + Unknown, + + #[error("is chosen more than once")] + Duplicate, + + #[error("does not allow placement {0}")] + NotAllowed(String), +} + /// What differs between a deployment's retained target facts and the current site configuration. #[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] pub enum RetainedTargetMismatch { diff --git a/backend/core/src/application/controller/launch.rs b/backend/core/src/application/controller/launch.rs index bbdf1f04..096e84dc 100644 --- a/backend/core/src/application/controller/launch.rs +++ b/backend/core/src/application/controller/launch.rs @@ -16,12 +16,13 @@ use dotcore_domain::{SiteNodeHost, SiteNodeId}; use dotcore_proto::dotcore::{ application::v1::{ ApplicationConfigurationAttachment, ApplicationDeploymentDeclaration, - ApplicationStorageAttachment, + ApplicationSecretAttachment, ApplicationStorageAttachment, ApplicationStorageChoice, }, catalog::v1::{ - Application, ApplicationDependencyCondition, ApplicationInferenceRouteProjection, - ParameterValue, StorageScope, application_configuration_binding, - application_deployment_input, application_service_readiness, + Application, ApplicationDependencyCondition, ApplicationGraph, + ApplicationInferenceRouteProjection, ApplicationSecretForm, ParameterValue, + StoragePlacement, application_configuration_binding, application_deployment_input, + application_service_readiness, }, }; use dotcore_runtime::{ @@ -33,8 +34,9 @@ use tracing::debug; use super::{ ApplicationConfigurationSources, ApplicationLifecycleError, InferenceRouteResolver, - ResolvedInferenceRoute, configuration, configuration_with_source, + ResolvedInferenceRoute, StorageChoiceRejection, configuration, configuration_with_source, }; +use crate::application::{binding::binding_enabled, secret_form::bcrypt_hash}; use crate::storage::{deployment_storage, workload_storage}; pub(super) fn workload_group_launch( @@ -71,8 +73,8 @@ pub(super) fn deployment_site_node_id( }) } -/// Returns the data storage scoped to one application deployment alone: its launch attaches -/// deployment-scoped bindings under this key and its teardown removes it. +/// Returns the data storage placed with one application deployment alone: its launch attaches +/// deployment-placed bindings under this key and its teardown removes it. pub(super) fn deployment_data(desired: &ApplicationDeploymentDeclaration) -> WorkloadStorageKey { deployment_storage(WorkloadStorageKind::Data, &deployment_storage_key(desired)) } @@ -87,12 +89,11 @@ fn published_port( .ok_or_else(|| configuration("Application deployment published host port is invalid")) } -/// Returns the key of storage scoped to one application deployment. +/// Returns the key of storage placed with one application deployment. fn deployment_storage_key(desired: &ApplicationDeploymentDeclaration) -> String { desired.application_deployment_id.clone() } -#[derive(Debug)] struct LaunchPlan { deployment_id: String, site_node_id: SiteNodeId, @@ -113,10 +114,10 @@ impl LaunchPlan { ) -> Result { let declaration = ValidatedLaunchDeclaration::extract(desired)?; let attachments = AttachmentIndexes::build(desired)?; - let storage_key = |scope| match scope { - StorageScope::Deployment => Some(deployment_storage_key(desired)), - StorageScope::Application => Some(declaration.application_id.to_owned()), - StorageScope::EngineArtifact | StorageScope::Unspecified => None, + let storage_key = |placement| match placement { + StoragePlacement::Deployment => Some(deployment_storage_key(desired)), + StoragePlacement::Application => Some(declaration.application_id.to_owned()), + StoragePlacement::EngineArtifact | StoragePlacement::Unspecified => None, }; let ready_dependency_names = declaration .services @@ -139,6 +140,7 @@ impl LaunchPlan { service, image, &attachments, + declaration.parameter_values, &storage_key, declaration.route_id.is_some(), ready_dependency_names.contains(service.service_name.as_str()), @@ -227,6 +229,7 @@ impl LaunchPlan { struct ValidatedLaunchDeclaration<'a> { application_id: &'a str, services: &'a [dotcore_proto::dotcore::catalog::v1::ApplicationService], + parameter_values: &'a [ParameterValue], service_images: BTreeMap<&'a str, String>, execution_target_id: String, accelerator_device_ids: Vec, @@ -306,6 +309,7 @@ impl<'a> ValidatedLaunchDeclaration<'a> { Ok(Self { application_id: &snapshot.application_id, services: &graph.services, + parameter_values: &snapshot.parameter_values, service_images, execution_target_id: placement.execution_target_id.clone(), accelerator_device_ids: snapshot.accelerator_device_ids.clone(), @@ -325,10 +329,12 @@ struct LaunchPrimaryEndpoint { port_name: String, } -#[derive(Debug)] +/// Retained attachment values keyed by service and binding name. It holds secret values, so it +/// has no `Debug` form. struct AttachmentIndexes { - storage: BTreeMap<(String, String), String>, + storage: BTreeMap<(String, String), (String, StoragePlacement)>, configuration: BTreeMap<(String, String), String>, + secret: BTreeMap<(String, String), String>, } impl AttachmentIndexes { @@ -339,12 +345,14 @@ impl AttachmentIndexes { .storage_attachments .iter() .map(|attachment| { + let placement = StoragePlacement::try_from(attachment.placement) + .map_err(|_| configuration("Application storage placement is invalid"))?; Ok(( ( attachment.service_name.clone(), attachment.binding_name.clone(), ), - attachment.storage_name.clone(), + (attachment.storage_name.clone(), placement), )) }) .collect::, ApplicationLifecycleError>>()?; @@ -361,19 +369,33 @@ impl AttachmentIndexes { )) }) .collect::, ApplicationLifecycleError>>()?; + let secret = desired + .secret_attachments + .iter() + .map(|attachment| { + ( + ( + attachment.service_name.clone(), + attachment.binding_name.clone(), + ), + attachment.value.clone(), + ) + }) + .collect::>(); if storage.len() != desired.storage_attachments.len() || configuration_values.len() != desired.configuration_attachments.len() + || secret.len() != desired.secret_attachments.len() { return Err(configuration("Application attachment is duplicated")); } Ok(Self { storage, configuration: configuration_values, + secret, }) } } -#[derive(Debug)] struct WorkloadUnitPlan { image: String, command: Vec, @@ -391,30 +413,45 @@ impl WorkloadUnitPlan { service: &dotcore_proto::dotcore::catalog::v1::ApplicationService, image: &str, attachments: &AttachmentIndexes, - storage_key: &dyn Fn(StorageScope) -> Option, + parameter_values: &[ParameterValue], + storage_key: &dyn Fn(StoragePlacement) -> Option, route_linked: bool, has_ready_dependents: bool, ) -> Result { - let mut environment: Vec<(String, LaunchEnvironmentValue)> = service - .configuration_bindings - .iter() - .map(|binding| { - let value = if route_linked && is_route_name_binding(binding)? { - LaunchEnvironmentValue::RouteName - } else { - LaunchEnvironmentValue::Fixed( - attachments - .configuration - .get(&(service.service_name.clone(), binding.binding_name.clone())) - .cloned() - .ok_or_else(|| { - configuration("Application configuration attachment is missing") - })?, - ) - }; - Ok((binding.environment_variable.clone(), value)) - }) - .collect::>()?; + let mut environment = Vec::new(); + for binding in &service.configuration_bindings { + if !enabled(binding.enabled_by_parameter_id.as_deref(), parameter_values)? { + continue; + } + let value = if route_linked && is_route_name_binding(binding)? { + LaunchEnvironmentValue::RouteName + } else { + LaunchEnvironmentValue::Fixed( + attachments + .configuration + .get(&(service.service_name.clone(), binding.binding_name.clone())) + .cloned() + .ok_or_else(|| { + configuration("Application configuration attachment is missing") + })?, + ) + }; + environment.push((binding.environment_variable.clone(), value)); + } + for binding in &service.secret_bindings { + if !enabled(binding.enabled_by_parameter_id.as_deref(), parameter_values)? { + continue; + } + let value = attachments + .secret + .get(&(service.service_name.clone(), binding.binding_name.clone())) + .cloned() + .ok_or_else(|| configuration("Application secret attachment is missing"))?; + environment.push(( + binding.environment_variable.clone(), + LaunchEnvironmentValue::Fixed(value), + )); + } let host_aliases = service .configuration_bindings .iter() @@ -433,7 +470,7 @@ impl WorkloadUnitPlan { .storage .get(&(service.service_name.clone(), binding.binding_name.clone())) .cloned() - .map(|name| (name, binding)) + .map(|(name, placement)| (name, binding, placement)) .ok_or_else(|| configuration("Application storage attachment is missing")) }) .collect::, _>>()?; @@ -543,7 +580,6 @@ impl WorkloadUnitPlan { } } -#[derive(Debug)] enum LaunchEnvironmentValue { Fixed(String), @@ -570,8 +606,14 @@ fn canonical_launch_generation( publication: &WorkloadPublicationRequest, ) -> u64 { let mut value = 0xcbf29ce484222325_u64; + // A unit's `Debug` form leaves out environment values, so they are hashed explicitly. + let environment = units + .values() + .flat_map(|unit| &unit.environment) + .map(|(name, value)| format!("{name}={value}\n")) + .collect::(); for byte in format!( - "{execution_target:?}:{execution_target_id}:{required_platform:?}:{units:?}:{publication:?}" + "{execution_target:?}:{execution_target_id}:{required_platform:?}:{units:?}:{publication:?}:{environment}" ) .bytes() { @@ -606,7 +648,8 @@ fn site_host_alias( /// /// Route-name bindings get no attachment when a route is linked, since launch projects the current /// route name; without a link they retain their catalog literal fallback. Parameter bindings -/// retain the canonical text of their resolved `parameter_values` entry. +/// retain the canonical text of their resolved `parameter_values` entry. A binding switched off by +/// its enabling parameter gets no attachment. pub(super) fn configuration_attachments( application: &Application, resolve: impl Fn(&str) -> Option, @@ -617,48 +660,40 @@ pub(super) fn configuration_attachments( .graph .as_ref() .ok_or_else(|| configuration("Application graph is missing"))?; - if graph - .services - .iter() - .any(|service| !service.secret_bindings.is_empty()) - { - return Err(ApplicationLifecycleError::FailedPrecondition); + let mut attachments = Vec::new(); + for service in &graph.services { + for binding in &service.configuration_bindings { + let route_name = is_route_name_binding(binding)?; + if !enabled(binding.enabled_by_parameter_id.as_deref(), parameter_values)? + || (route_linked && route_name) + { + continue; + } + let value = configuration_attachment_value(binding, &resolve, parameter_values)?; + if !valid_configuration_value(&value) { + return Err(configuration( + "Application configuration value cannot be represented safely", + )); + } + attachments.push(ApplicationConfigurationAttachment { + service_name: service.service_name.clone(), + binding_name: binding.binding_name.clone(), + environment_variable: binding.environment_variable.clone(), + value, + }); + } } - let resolve = &resolve; - graph - .services - .iter() - .flat_map(|service| { - service - .configuration_bindings - .iter() - .filter_map(move |binding| match is_route_name_binding(binding) { - Ok(true) if route_linked => None, - Ok(_) => { - let value = match configuration_attachment_value( - binding, - resolve, - parameter_values, - ) { - Ok(value) if valid_configuration_value(&value) => value, - Ok(_) => { - return Some(Err(configuration( - "Application configuration value cannot be represented safely", - ))); - } - Err(error) => return Some(Err(error)), - }; - Some(Ok(ApplicationConfigurationAttachment { - service_name: service.service_name.clone(), - binding_name: binding.binding_name.clone(), - environment_variable: binding.environment_variable.clone(), - value, - })) - } - Err(error) => Some(Err(error)), - }) - }) - .collect() + Ok(attachments) +} + +/// Returns whether a binding reaches its service, rejecting a retained declaration whose enabling +/// parameter has no boolean value. +fn enabled( + enabled_by_parameter_id: Option<&str>, + parameter_values: &[ParameterValue], +) -> Result { + binding_enabled(enabled_by_parameter_id, parameter_values) + .ok_or_else(|| configuration("Application binding switch value is missing")) } /// Returns whether a binding projects the linked route name, rejecting unknown projections. @@ -695,30 +730,115 @@ fn configuration_attachment_value( } } +/// Resolves the retained secret attachments of a new deployment: each enabled secret binding +/// receives its parameter's resolved value in the binding's form, and a binding switched off by +/// its enabling parameter gets none. A bcrypt form hashes, so callers run it off the async +/// executor, and only once per deployment. +pub(super) fn secret_attachments( + graph: &ApplicationGraph, + parameter_values: &[ParameterValue], +) -> Result, ApplicationLifecycleError> { + let mut attachments = Vec::new(); + for service in &graph.services { + for binding in &service.secret_bindings { + if enabled(binding.enabled_by_parameter_id.as_deref(), parameter_values)? { + attachments.push(secret_attachment(service, binding, parameter_values)?); + } + } + } + Ok(attachments) +} + +/// Resolves one enabled secret binding's attachment in the binding's form. +fn secret_attachment( + service: &dotcore_proto::dotcore::catalog::v1::ApplicationService, + binding: &dotcore_proto::dotcore::catalog::v1::ApplicationSecretBinding, + parameter_values: &[ParameterValue], +) -> Result { + let value = canonical_value_text(parameter_values, &binding.parameter_id) + .ok_or_else(|| configuration("Application secret parameter value is missing"))?; + let value = match ApplicationSecretForm::try_from(binding.form) { + Ok(ApplicationSecretForm::Plain) => value, + Ok(ApplicationSecretForm::Bcrypt) => bcrypt_hash(&value).map_err(|error| { + configuration_with_source("Application secret form cannot be computed", error) + })?, + Ok(ApplicationSecretForm::Unspecified) | Err(_) => { + return Err(configuration("Application secret binding form is invalid")); + } + }; + if !valid_configuration_value(&value) { + return Err(configuration( + "Application secret value cannot be represented safely", + )); + } + Ok(ApplicationSecretAttachment { + service_name: service.service_name.clone(), + binding_name: binding.binding_name.clone(), + environment_variable: binding.environment_variable.clone(), + value, + }) +} + +/// Resolves one storage attachment per storage binding of a new deployment, each placed where the +/// operator's `choices` say or at the binding's first allowed placement. pub(super) fn storage_attachments( application: &Application, + choices: &[ApplicationStorageChoice], ) -> Result, ApplicationLifecycleError> { let graph = application .graph .as_ref() .ok_or_else(|| configuration("Application graph is missing"))?; + let mut chosen = BTreeMap::new(); + for choice in choices { + let reject = |reason| ApplicationLifecycleError::InvalidStorageChoice { + service_name: choice.service_name.clone(), + binding_name: choice.binding_name.clone(), + reason, + }; + let binding = graph + .services + .iter() + .filter(|service| service.service_name == choice.service_name) + .flat_map(|service| &service.storage_bindings) + .find(|binding| binding.binding_name == choice.binding_name) + .ok_or_else(|| reject(StorageChoiceRejection::Unknown))?; + if !binding.placements.contains(&choice.placement) { + let placement = StoragePlacement::try_from(choice.placement).map_or_else( + |_| choice.placement.to_string(), + |placement| placement.as_str_name().to_owned(), + ); + return Err(reject(StorageChoiceRejection::NotAllowed(placement))); + } + let key = (choice.service_name.as_str(), choice.binding_name.as_str()); + if chosen.insert(key, choice.placement).is_some() { + return Err(reject(StorageChoiceRejection::Duplicate)); + } + } + graph .services .iter() .flat_map(|service| { + let chosen = &chosen; service.storage_bindings.iter().map(move |binding| { let storage_name = format!("{}-{}", service.service_name, binding.binding_name); - if valid_id(&storage_name) { - Ok(ApplicationStorageAttachment { - service_name: service.service_name.clone(), - binding_name: binding.binding_name.clone(), - storage_name, - }) - } else { - Err(configuration( + if !valid_id(&storage_name) { + return Err(configuration( "Application storage attachment name cannot be represented safely", - )) + )); } + let placement = chosen + .get(&(service.service_name.as_str(), binding.binding_name.as_str())) + .or(binding.placements.first()) + .copied() + .ok_or_else(|| configuration("Application storage binding has no placement"))?; + Ok(ApplicationStorageAttachment { + service_name: service.service_name.clone(), + binding_name: binding.binding_name.clone(), + storage_name, + placement, + }) }) }) .collect() diff --git a/backend/core/src/application/controller/logs.rs b/backend/core/src/application/controller/logs.rs index a9e5f77f..0cf0d6a2 100644 --- a/backend/core/src/application/controller/logs.rs +++ b/backend/core/src/application/controller/logs.rs @@ -4,12 +4,14 @@ //! its process and over the private deployment-log service otherwise. The node serves the output //! of each preparation step it ran for the deployment's services, resolves its own group from the //! installed assignment and its pinned launch identities, exactly as it launches it, and redacts -//! the launch environment values from both before any entry leaves it. +//! the launch environment values and the secret parameter values from both before any entry leaves +//! it, so a secret is redacted in its plain form as well as in the form its service receives. use std::{future::Future, net::Ipv4Addr, sync::Arc}; use anyhow::anyhow; use dotcore_artifact::PreparationStep; +use dotcore_catalog::typed_parameters::{canonical_value_text, is_secret}; use dotcore_proto::dotcore::{ application::v1::{ApplicationDeploymentDeclaration, ApplicationDeploymentDesiredState}, deployment::private::v1::DeploymentOwnerKind, @@ -221,7 +223,7 @@ impl NodeApplicationLogs { steps: split.steps.clone(), known_steps: known_steps.clone(), workload: NodeGroupSource::NotCreated(selection), - protected: ProtectedLogValues::new(attachment_values(declaration)), + protected: protected_values(declaration, attachment_values(declaration)), }; let Some(workload_names) = &split.workload else { return Ok(not_created( @@ -486,7 +488,8 @@ fn local_group( .clone(); let group = workload_group_launch(deployment, publication_host, inference_route_resolver) .map_err(map_lifecycle_error)?; - let protected = ProtectedLogValues::new( + let protected = protected_values( + deployment, group .units .values() @@ -569,8 +572,8 @@ fn step_sources(services: &[RuntimeLogSource]) -> Vec { .collect() } -/// Returns the configuration values the deployment's launch receives, which step lines must -/// not reveal either. +/// Returns the configuration and secret values the deployment's launch receives, which step +/// lines must not reveal either. fn attachment_values( deployment: &ApplicationDeploymentDeclaration, ) -> impl Iterator { @@ -578,6 +581,40 @@ fn attachment_values( .configuration_attachments .iter() .map(|attachment| attachment.value.clone()) + .chain( + deployment + .secret_attachments + .iter() + .map(|attachment| attachment.value.clone()), + ) +} + +/// Protects `launch_values` and the deployment's secret parameter values. A secret bound in a +/// derived form, such as a hash, never appears among the launch values as typed, yet stays +/// protected. +fn protected_values( + deployment: &ApplicationDeploymentDeclaration, + launch_values: impl IntoIterator, +) -> ProtectedLogValues { + let mut protected = ProtectedLogValues::new(launch_values); + protected.extend(ProtectedLogValues::new(secret_parameter_values(deployment))); + protected +} + +/// Returns the retained values of the deployment's secret parameters. +fn secret_parameter_values( + deployment: &ApplicationDeploymentDeclaration, +) -> impl Iterator { + deployment.snapshot.iter().flat_map(|snapshot| { + snapshot + .graph + .iter() + .flat_map(|graph| &graph.parameters) + .filter(|definition| is_secret(definition)) + .filter_map(|definition| { + canonical_value_text(&snapshot.parameter_values, &definition.parameter_id) + }) + }) } /// Selects the requested sources among `declared`, or all of them. diff --git a/backend/core/src/application/controller/mod.rs b/backend/core/src/application/controller/mod.rs index eca41141..bdffe26b 100644 --- a/backend/core/src/application/controller/mod.rs +++ b/backend/core/src/application/controller/mod.rs @@ -67,7 +67,8 @@ use crate::application::repository::{ CurrentApplicationRepository, }; pub use errors::{ - ApplicationLifecycleError, ParameterRejection, RetainedTargetMismatch, TargetIneligibility, + ApplicationLifecycleError, ParameterRejection, RetainedTargetMismatch, StorageChoiceRejection, + TargetIneligibility, }; use errors::{configuration, configuration_with_source, outcome_uncertain_with_source}; pub use logs::ApplicationDeploymentLogController; @@ -208,6 +209,7 @@ pub struct ApplicationControllerConfiguration { pub configuration_sources: ApplicationConfigurationSources, pub inference_route_resolver: Arc, pub target_capabilities: Arc, + pub secret_generator: Arc, } /// Apps-controller lifecycle authority on the site manager: the owner of application deployment @@ -237,12 +239,14 @@ where configuration_sources, inference_route_resolver, target_capabilities, + secret_generator, } = controller_configuration; let owner = Arc::new(ApplicationOwner { catalog, configuration_sources, inference_route_resolver, target_capabilities, + secret_generator, desired: Arc::new(desired), current: Arc::new(current), operation: AsyncMutex::new(()), diff --git a/backend/core/src/application/controller/node.rs b/backend/core/src/application/controller/node.rs index e7596de7..25664ee5 100644 --- a/backend/core/src/application/controller/node.rs +++ b/backend/core/src/application/controller/node.rs @@ -922,14 +922,15 @@ mod tests { v1::{ ApplicationConfigurationAttachment, ApplicationDeploymentDeclaration, ApplicationDeploymentDesiredState, ApplicationLifecycleState, - ApplicationReadinessState, ApplicationServiceAcceleratorMemory, - ApplicationStorageAttachment, CurrentApplicationDeployment, - ResolvedApplicationDeploymentSnapshot, ResolvedApplicationServiceSelection, + ApplicationReadinessState, ApplicationSecretAttachment, + ApplicationServiceAcceleratorMemory, ApplicationStorageAttachment, + CurrentApplicationDeployment, ResolvedApplicationDeploymentSnapshot, + ResolvedApplicationServiceSelection, }, }, catalog::v1::{ - ApplicationHttpReadiness, ApplicationServiceReadiness, StorageBinding, StorageKind, - StorageScope, application_service_readiness, + ApplicationHttpReadiness, ApplicationServiceReadiness, ParameterValue, StorageBinding, + StorageKind, StoragePlacement, application_service_readiness, parameter_value, }, deployment::private::v1::{ DeploymentLogSourceOpening, DeploymentLogTarget, DeploymentLogUnavailableReason, @@ -985,7 +986,7 @@ mod tests { application::{ controller::test_support::{ application_execution_target, application_with_route_input, deployment, - resolved_snapshot, start_steps, + resolved_snapshot, secret_parameter, start_steps, }, repository::CurrentApplicationRepository as _, }, @@ -1009,6 +1010,7 @@ mod tests { "sha256:1111111111111111111111111111111111111111111111111111111111111111"; const REBUILT_IMAGE: &str = "sha256:2222222222222222222222222222222222222222222222222222222222222222"; + const PASSWORD_HASH: &str = "$2b$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhWy"; #[tokio::test] async fn node_realizes_its_valid_assignments_and_reports_each_typed_status() { @@ -2480,6 +2482,27 @@ mod tests { environment_variable: "TOKEN".to_owned(), value: "secret-token".to_owned(), }); + // The service receives the password as a hash; the retained password itself is never + // part of the launch, yet stays protected. + let snapshot = preparing.snapshot.as_mut().expect("snapshot"); + snapshot + .graph + .as_mut() + .expect("graph") + .parameters + .push(secret_parameter("password")); + snapshot.parameter_values.push(ParameterValue { + parameter_id: "password".to_owned(), + value: Some(parameter_value::Value::String("secret-password".to_owned())), + }); + preparing + .secret_attachments + .push(ApplicationSecretAttachment { + service_name: "web".to_owned(), + binding_name: "password".to_owned(), + environment_variable: "PASSWORD_HASH".to_owned(), + value: PASSWORD_HASH.to_owned(), + }); fixture.install(1, vec![assignment(preparing, None)]); // Never reconciled, so the deployment's group does not exist yet. let logs = fixture.realizer().log_owner(); @@ -2495,7 +2518,7 @@ mod tests { build.append( time(1), RuntimeLogStreamKind::Stderr, - b"building with secret-token", + format!("building with secret-token, secret-password and {PASSWORD_HASH}").as_bytes(), ); preparation .writer( @@ -2555,7 +2578,11 @@ mod tests { )); assert_eq!( all.protected, - ProtectedLogValues::new(["secret-token".to_owned()]) + ProtectedLogValues::new([ + "secret-token".to_owned(), + PASSWORD_HASH.to_owned(), + "secret-password".to_owned(), + ]) ); let build_entry = all.sources[0] .result @@ -2565,7 +2592,7 @@ mod tests { .clone(); assert_eq!( filter_entry(build_entry, &all.protected).content, - b"building with [REDACTED]" + b"building with [REDACTED], [REDACTED] and [REDACTED]" ); assert_eq!( filtered @@ -3052,13 +3079,14 @@ mod tests { binding_name: "data".to_owned(), mount_path: "/data".to_owned(), kind: StorageKind::Data as i32, - scope: StorageScope::Deployment as i32, + placements: vec![StoragePlacement::Deployment as i32], environment: Vec::new(), }]; declaration.storage_attachments = vec![ApplicationStorageAttachment { service_name: service.service_name.clone(), binding_name: "data".to_owned(), storage_name: "web-data".to_owned(), + placement: StoragePlacement::Deployment as i32, }]; declaration } @@ -3158,7 +3186,7 @@ mod tests { .expect("application status") } - /// Returns the data the launch of one deployment attaches under its deployment scope. + /// Returns the data the launch of one deployment attaches under its deployment placement. fn deployment_data(deployment_id: &str) -> WorkloadStorageKey { WorkloadStorageKey { kind: WorkloadStorageKind::Data, diff --git a/backend/core/src/application/controller/operations.rs b/backend/core/src/application/controller/operations.rs index 2388c48d..987a96ed 100644 --- a/backend/core/src/application/controller/operations.rs +++ b/backend/core/src/application/controller/operations.rs @@ -6,7 +6,7 @@ use dotcore_domain::{SiteNodeId, real_time}; use dotcore_proto::dotcore::{ application::v1::{ ApplicationDeploymentDeclaration, ApplicationDeploymentDesiredState, - ApplicationLifecycleState, CreateApplicationDeploymentRequest, + ApplicationLifecycleState, ApplicationSecretAttachment, CreateApplicationDeploymentRequest, CurrentApplicationDeployment, ResolvedApplicationDeploymentSnapshot, ResolvedApplicationServiceSelection, }, @@ -55,6 +55,7 @@ where &site, &declaration_snapshot, )?; + declaration.secret_attachments = secret_attachments(&declaration).await?; declaration.lifecycle_generation = owner.reserve_lifecycle_generation(&declaration.application_deployment_id)?; persist_create(owner, &declaration, declaration_snapshot.storage_revision)?; @@ -526,10 +527,33 @@ fn resolve_application( .ok_or(ApplicationLifecycleError::NotFound) } +/// Resolves a new declaration's secret attachments on a blocking thread, since a bcrypt form +/// spends a deliberate amount of CPU time per hash. +async fn secret_attachments( + declaration: &ApplicationDeploymentDeclaration, +) -> Result, ApplicationLifecycleError> { + let snapshot = declaration + .snapshot + .as_ref() + .ok_or(ApplicationLifecycleError::FailedPrecondition)?; + let graph = snapshot + .graph + .clone() + .ok_or(ApplicationLifecycleError::FailedPrecondition)?; + let values = snapshot.parameter_values.clone(); + tokio::task::spawn_blocking(move || super::launch::secret_attachments(&graph, &values)) + .await + .map_err(|error| { + configuration_with_source("Application secret resolution stopped", error) + })? +} + /// Builds the declaration of a new deployment on `placement`, with its immutable snapshot for -/// the selected `platform`, the parameter values resolved from the request, and a published host -/// port no other retained deployment holds, even one still tearing down. Local-development launch -/// identities are left empty for the hosting node to pin once on its own image store. +/// the selected `platform`, the parameter values resolved from the request with missing secrets +/// generated, and a published host port no other retained deployment holds, even one still +/// tearing down. Its secret attachments are left for the caller to derive once, off the async +/// executor. Local-development launch identities are left empty for the hosting node to pin once +/// on its own image store. fn declaration( owner: &Arc>, placement: &ExactPlacement, @@ -549,6 +573,7 @@ where .as_ref() .ok_or(ApplicationLifecycleError::FailedPrecondition)?, &request.parameter_values, + owner.secret_generator.as_ref(), )?; // The link is checked for identity and generation only; route availability never gates creation. if let Some(inference_route_id) = request.inference_route_id.as_deref() { @@ -589,7 +614,8 @@ where ¶meter_values, request.inference_route_id.is_some(), )?; - let storage_attachments = super::launch::storage_attachments(&application)?; + let storage_attachments = + super::launch::storage_attachments(&application, &request.storage_choices)?; Ok(ApplicationDeploymentDeclaration { application_deployment_id: deployment_id, @@ -606,6 +632,7 @@ where desired_state: ApplicationDeploymentDesiredState::Present as i32, storage_attachments, configuration_attachments, + secret_attachments: Vec::new(), inference_route_id: request.inference_route_id, lifecycle_generation: 0, node_resolved_launch_identities: Vec::new(), diff --git a/backend/core/src/application/controller/owner.rs b/backend/core/src/application/controller/owner.rs index 76465466..076894b8 100644 --- a/backend/core/src/application/controller/owner.rs +++ b/backend/core/src/application/controller/owner.rs @@ -48,6 +48,7 @@ pub(super) struct ApplicationOwner { pub(super) configuration_sources: ApplicationConfigurationSources, pub(super) inference_route_resolver: Arc, pub(super) target_capabilities: Arc, + pub(super) secret_generator: Arc, pub(super) desired: Arc, pub(super) current: Arc, diff --git a/backend/core/src/application/controller/parameters.rs b/backend/core/src/application/controller/parameters.rs index 21a5017e..51067a20 100644 --- a/backend/core/src/application/controller/parameters.rs +++ b/backend/core/src/application/controller/parameters.rs @@ -1,21 +1,26 @@ //! Application parameter resolution at deployment creation. //! //! Merges operator-supplied values over the defaults an application declares, rejecting values -//! the application does not declare or does not accept. The result is the complete set a -//! deployment retains for its whole life, in the application's declaration order. +//! the application does not declare or does not accept, and generates a random value for each +//! secret the operator leaves out. The result is the complete set a deployment retains for its +//! whole life, in the application's declaration order. use std::collections::BTreeSet; use dotcore_catalog::typed_parameters::{check_value, default_value}; -use dotcore_proto::dotcore::catalog::v1::{ApplicationGraph, ParameterValue}; +use dotcore_proto::dotcore::catalog::v1::{ + ApplicationGraph, ParameterValue, parameter_definition::ParameterType, parameter_value::Value, +}; use super::errors::{ApplicationLifecycleError, ParameterRejection, configuration}; +use crate::SecretGenerator; /// Resolves the complete parameter values of a new deployment from the operator's `supplied` -/// overrides and the graph's declared defaults. +/// overrides, the graph's declared defaults, and a generated value for each secret left out. pub(super) fn resolve_parameter_values( graph: &ApplicationGraph, supplied: &[ParameterValue], + secrets: &dyn SecretGenerator, ) -> Result, ApplicationLifecycleError> { let mut supplied_ids = BTreeSet::new(); for value in supplied { @@ -29,7 +34,6 @@ pub(super) fn resolve_parameter_values( let definition = graph .parameters .iter() - .filter_map(|parameter| parameter.definition.as_ref()) .find(|definition| definition.parameter_id == value.parameter_id) .ok_or_else(|| reject(ParameterRejection::Unknown))?; let inner = value @@ -40,26 +44,32 @@ pub(super) fn resolve_parameter_values( .map_err(|source| reject(ParameterRejection::InvalidValue(source)))?; } - graph - .parameters - .iter() - .map(|parameter| { - let definition = parameter - .definition - .as_ref() - .ok_or_else(|| configuration("Application parameter definition is missing"))?; - let value = match supplied - .iter() - .find(|value| value.parameter_id == definition.parameter_id) - { - Some(supplied) => supplied.value.clone(), - None => default_value(definition), + let mut resolved = Vec::with_capacity(graph.parameters.len()); + for definition in &graph.parameters { + let supplied = supplied + .iter() + .find(|value| value.parameter_id == definition.parameter_id) + .and_then(|value| value.value.clone()); + let value = match (supplied, definition.parameter_type.as_ref()) { + (Some(value), _) => value, + (None, Some(ParameterType::Secret(schema))) => { + let value = + secrets + .generate(schema.generated_length as usize) + .map_err(|source| { + ApplicationLifecycleError::Configuration( + source.context("Application secret parameter generation failed"), + ) + })?; + Value::String(value) } - .ok_or_else(|| configuration("Application parameter default is missing"))?; - Ok(ParameterValue { - parameter_id: definition.parameter_id.clone(), - value: Some(value), - }) - }) - .collect() + (None, _) => default_value(definition) + .ok_or_else(|| configuration("Application parameter default is missing"))?, + }; + resolved.push(ParameterValue { + parameter_id: definition.parameter_id.clone(), + value: Some(value), + }); + } + Ok(resolved) } diff --git a/backend/core/src/application/controller/test_support.rs b/backend/core/src/application/controller/test_support.rs index e2c76ba4..d01a28f5 100644 --- a/backend/core/src/application/controller/test_support.rs +++ b/backend/core/src/application/controller/test_support.rs @@ -69,6 +69,7 @@ use dotcore_site::{ use tokio::sync::Notify; use super::*; +pub(super) use crate::test_support::{sequential_secret, test_secret_generator}; #[derive(Default)] pub(super) struct FailingApplicationPreparer { @@ -861,6 +862,7 @@ pub(super) fn manager_controller( target_capabilities, configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("manager apps controller") @@ -1149,6 +1151,7 @@ where target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, artifacts, ) @@ -1174,6 +1177,7 @@ where target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver, + secret_generator: test_secret_generator(), }, ) .expect("Apps controller") @@ -1193,6 +1197,7 @@ where target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller") @@ -2799,6 +2804,7 @@ pub(super) fn route_configuration_binding( ), ), }), + enabled_by_parameter_id: None, } } @@ -2813,6 +2819,7 @@ pub(super) fn deployment( desired_state: desired_state as i32, storage_attachments: Vec::new(), configuration_attachments: Vec::new(), + secret_attachments: Vec::new(), inference_route_id: None, lifecycle_generation: 1, node_resolved_launch_identities: Vec::new(), @@ -2865,6 +2872,7 @@ pub(super) fn controller_for_application_and_target_source( target_capabilities, configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller") @@ -2948,40 +2956,36 @@ pub(super) fn application_with_route_input() -> Application { /// an environment variable of its web service. pub(super) fn application_with_parameters() -> Application { use dotcore_proto::dotcore::catalog::v1::{ - ApplicationParameter, ParameterDefinition, StringEnumParameterSchema, + ParameterDefinition, ParameterVisibility, StringEnumParameterSchema, UnsignedIntegerParameterSchema, parameter_definition::ParameterType, }; let mut application = application(); let graph = application.graph.as_mut().expect("graph"); graph.parameters = vec![ - ApplicationParameter { - definition: Some(ParameterDefinition { - parameter_id: "mode".to_owned(), - display_name: "Mode".to_owned(), - description: String::new(), - parameter_type: Some(ParameterType::StringEnum(StringEnumParameterSchema { - values: vec!["auto".to_owned(), "highvram".to_owned()], - default_value: Some("auto".to_owned()), - })), - }), - advanced: true, + ParameterDefinition { + parameter_id: "mode".to_owned(), + display_name: "Mode".to_owned(), + description: String::new(), + parameter_type: Some(ParameterType::StringEnum(StringEnumParameterSchema { + values: vec!["auto".to_owned(), "highvram".to_owned()], + default_value: Some("auto".to_owned()), + })), + visibility: ParameterVisibility::Advanced as i32, }, - ApplicationParameter { - definition: Some(ParameterDefinition { - parameter_id: "workers".to_owned(), - display_name: "Workers".to_owned(), - description: String::new(), - parameter_type: Some(ParameterType::UnsignedInteger( - UnsignedIntegerParameterSchema { - minimum: 1, - maximum: 8, - unit: String::new(), - default_value: Some(2), - }, - )), - }), - advanced: false, + ParameterDefinition { + parameter_id: "workers".to_owned(), + display_name: "Workers".to_owned(), + description: String::new(), + parameter_type: Some(ParameterType::UnsignedInteger( + UnsignedIntegerParameterSchema { + minimum: 1, + maximum: 8, + unit: String::new(), + default_value: Some(2), + }, + )), + visibility: ParameterVisibility::Basic as i32, }, ]; graph.services[0].configuration_bindings = ["mode", "workers"] @@ -2993,11 +2997,152 @@ pub(super) fn application_with_parameters() -> Application { id.to_owned(), )), input: None, + enabled_by_parameter_id: None, }) .collect(); application } +/// The test application whose web service keeps `data` with the deployment only and lets the +/// operator place `models` with the application (the default) or with the deployment. +pub(super) fn application_with_storage_choices() -> Application { + use dotcore_proto::dotcore::catalog::v1::{StorageBinding, StorageKind, StoragePlacement}; + + let mut application = application(); + let binding = |name: &str, placements: &[StoragePlacement]| StorageBinding { + binding_name: name.to_owned(), + mount_path: format!("/{name}"), + kind: StorageKind::Data as i32, + placements: placements + .iter() + .map(|&placement| placement as i32) + .collect(), + environment: Vec::new(), + }; + application.graph.as_mut().expect("graph").services[0].storage_bindings = vec![ + binding("data", &[StoragePlacement::Deployment]), + binding( + "models", + &[StoragePlacement::Application, StoragePlacement::Deployment], + ), + ]; + application +} + +/// The test application with a text `username` parameter bound to `USERNAME` and a secret +/// `password` parameter bound in plain form to `PASSWORD` and in bcrypt form to `PASSWORD_HASH`. +pub(super) fn application_with_secret_parameters() -> Application { + use dotcore_proto::dotcore::catalog::v1::{ + ApplicationSecretBinding, ApplicationSecretForm, ParameterDefinition, ParameterVisibility, + TextParameterSchema, parameter_definition::ParameterType, + }; + + let mut application = application(); + let graph = application.graph.as_mut().expect("graph"); + graph.parameters = vec![ + ParameterDefinition { + parameter_id: "username".to_owned(), + display_name: "Username".to_owned(), + description: String::new(), + parameter_type: Some(ParameterType::Text(TextParameterSchema { + minimum_length: 1, + maximum_length: 64, + default_value: Some("admin".to_owned()), + })), + visibility: ParameterVisibility::Basic as i32, + }, + secret_parameter("password"), + ]; + let web = &mut graph.services[0]; + web.configuration_bindings = vec![ApplicationConfigurationBinding { + binding_name: "username".to_owned(), + environment_variable: "USERNAME".to_owned(), + source: Some(application_configuration_binding::Source::ParameterId( + "username".to_owned(), + )), + input: None, + enabled_by_parameter_id: None, + }]; + web.secret_bindings = vec![ + ApplicationSecretBinding { + binding_name: "password".to_owned(), + environment_variable: "PASSWORD".to_owned(), + parameter_id: "password".to_owned(), + form: ApplicationSecretForm::Plain as i32, + enabled_by_parameter_id: None, + }, + ApplicationSecretBinding { + binding_name: "password-hash".to_owned(), + environment_variable: "PASSWORD_HASH".to_owned(), + parameter_id: "password".to_owned(), + form: ApplicationSecretForm::Bcrypt as i32, + enabled_by_parameter_id: None, + }, + ]; + application +} + +/// The secret-parameter test application with a boolean `login` parameter, off by default, bound +/// to `LOGIN` and enabling the `username` and both `password` bindings. +pub(super) fn application_with_login_switch() -> Application { + use dotcore_proto::dotcore::catalog::v1::{ + BooleanParameterSchema, ParameterDefinition, ParameterVisibility, + parameter_definition::ParameterType, + }; + + let mut application = application_with_secret_parameters(); + let graph = application.graph.as_mut().expect("graph"); + graph.parameters.push(ParameterDefinition { + parameter_id: "login".to_owned(), + display_name: "Login".to_owned(), + description: String::new(), + parameter_type: Some(ParameterType::Boolean(BooleanParameterSchema { + default_value: Some(false), + })), + visibility: ParameterVisibility::Basic as i32, + }); + let web = &mut graph.services[0]; + let switch = Some("login".to_owned()); + for binding in &mut web.configuration_bindings { + binding.enabled_by_parameter_id = switch.clone(); + } + for binding in &mut web.secret_bindings { + binding.enabled_by_parameter_id = switch.clone(); + } + web.configuration_bindings + .push(ApplicationConfigurationBinding { + binding_name: "login".to_owned(), + environment_variable: "LOGIN".to_owned(), + source: Some(application_configuration_binding::Source::ParameterId( + "login".to_owned(), + )), + input: None, + enabled_by_parameter_id: None, + }); + application +} + +/// A basic secret parameter generated at 32 characters and accepted from 12. +pub(super) fn secret_parameter( + id: &str, +) -> dotcore_proto::dotcore::catalog::v1::ParameterDefinition { + use dotcore_proto::dotcore::catalog::v1::{ + ParameterDefinition, ParameterVisibility, SecretParameterSchema, + parameter_definition::ParameterType, + }; + + ParameterDefinition { + parameter_id: id.to_owned(), + display_name: id.to_owned(), + description: String::new(), + parameter_type: Some(ParameterType::Secret(SecretParameterSchema { + generated_length: 32, + minimum_length: 12, + })), + visibility: ParameterVisibility::Basic as i32, + } +} + /// The test application with its web service bound to the hosting node's inference base URL. pub(super) fn base_url_application() -> Application { let mut application = application(); @@ -3012,6 +3157,7 @@ pub(super) fn base_url_application() -> Application { ), ), input: None, + enabled_by_parameter_id: None, }); application } diff --git a/backend/core/src/application/controller/tests.rs b/backend/core/src/application/controller/tests.rs index b62f6f29..51d8be99 100644 --- a/backend/core/src/application/controller/tests.rs +++ b/backend/core/src/application/controller/tests.rs @@ -19,7 +19,9 @@ use dotcore_domain::{ }, timing::{ReconcileCadence, ReconcileOutcome, Timing}, }; -use dotcore_proto::dotcore::application::v1::ApplicationConfigurationAttachment; +use dotcore_proto::dotcore::application::v1::{ + ApplicationConfigurationAttachment, ApplicationSecretAttachment, +}; use dotcore_proto::dotcore::{ application::private::v1::{ApplicationLifecycleObservation, ApplicationStatusStage}, application::v1::{ @@ -27,17 +29,18 @@ use dotcore_proto::dotcore::{ ApplicationDeploymentState, ApplicationEndpoint, ApplicationLifecycleState, ApplicationReadinessState, ApplicationSelectedInferenceRoute, ApplicationServiceAcceleratorMemory, ApplicationServiceLifecycleState, - ApplicationServiceObservation, ApplicationStorageAttachment, + ApplicationServiceObservation, ApplicationStorageAttachment, ApplicationStorageChoice, CreateApplicationDeploymentRequest, CurrentApplicationDeployment, UpdateApplicationDeploymentMetadataRequest, application_deployment_service_server::ApplicationDeploymentService as ApplicationServiceContract, }, catalog::v1::{ ApplicationConfigurationBinding, ApplicationDependencyCondition, ApplicationHttpReadiness, - ApplicationInferenceRouteProjection, ApplicationSecretBinding, ApplicationService, - ApplicationServiceDependency, ApplicationServicePort, ApplicationServiceReadiness, - ParameterValue, StorageBinding, StorageEnvironmentVariable, StorageKind, StorageScope, - application_configuration_binding, application_service_readiness, parameter_value, + ApplicationInferenceRouteProjection, ApplicationSecretBinding, ApplicationSecretForm, + ApplicationService, ApplicationServiceDependency, ApplicationServicePort, + ApplicationServiceReadiness, ParameterValue, StorageBinding, StorageEnvironmentVariable, + StorageKind, StoragePlacement, application_configuration_binding, + application_service_readiness, parameter_value, }, deployment::v1::{ AcceleratorMemoryDeviceUsage, AcceleratorMemoryObservation, AcceleratorMemoryUnavailable, @@ -66,7 +69,7 @@ use tokio::{io::AsyncWriteExt, sync::Notify}; use tonic::Request; use super::identity::allocate_deployment_id; -use super::launch::configuration_attachments; +use super::launch::{configuration_attachments, secret_attachments}; use super::launch::{deployment_data, workload_group_launch}; use super::observation::{exact_owned_observation, stale_owned_group}; use super::readiness::{ @@ -115,6 +118,7 @@ async fn application_artifact_verification_failure_prevents_runtime_creation_wit target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, artifacts, ) @@ -1461,7 +1465,7 @@ async fn present_recovery_replaces_stale_groups_only_after_resume_changes_desire } #[test] -fn launch_keys_each_storage_binding_by_its_scope() { +fn launch_keys_each_storage_attachment_by_its_retained_placement() { let mut deployment = deployment(ApplicationDeploymentDesiredState::Present); let web = &mut deployment .snapshot @@ -1476,27 +1480,34 @@ fn launch_keys_each_storage_binding_by_its_scope() { binding_name: "data".to_owned(), mount_path: "/data".to_owned(), kind: StorageKind::Data as i32, - scope: StorageScope::Deployment as i32, + placements: vec![StoragePlacement::Deployment as i32], environment: Vec::new(), }, StorageBinding { binding_name: "models".to_owned(), mount_path: "/data/models".to_owned(), kind: StorageKind::Data as i32, - scope: StorageScope::Application as i32, + placements: vec![ + StoragePlacement::Application as i32, + StoragePlacement::Deployment as i32, + ], environment: vec![StorageEnvironmentVariable { name: "MODELS_DIR".to_owned(), relative_path: "checkpoints".to_owned(), }], }, ]; - deployment.storage_attachments = ["data", "models"] - .map(|binding| ApplicationStorageAttachment { - service_name: "web".to_owned(), - binding_name: binding.to_owned(), - storage_name: format!("web-{binding}"), - }) - .to_vec(); + deployment.storage_attachments = [ + ("data", StoragePlacement::Deployment), + ("models", StoragePlacement::Application), + ] + .map(|(binding, placement)| ApplicationStorageAttachment { + service_name: "web".to_owned(), + binding_name: binding.to_owned(), + storage_name: format!("web-{binding}"), + placement: placement as i32, + }) + .to_vec(); let launch = workload_group_launch( &deployment, @@ -1533,8 +1544,27 @@ fn launch_keys_each_storage_binding_by_its_scope() { web.environment.get("MODELS_DIR").map(String::as_str), Some("/data/models/checkpoints") ); - // Teardown removes exactly the deployment-scoped storage the launch attached. + // Teardown removes exactly the deployment-placed storage the launch attached. assert_eq!(web.storage[0].storage_key(), deployment_data(&deployment)); + + // The same binding placed with the deployment lands in the deployment's own folder. + deployment.storage_attachments[1].placement = StoragePlacement::Deployment as i32; + let launch = workload_group_launch( + &deployment, + Ipv4Addr::LOCALHOST, + test_inference_route_resolver().as_ref(), + ) + .expect("launch"); + let models = &launch.units[&WorkloadUnitName("web".to_owned())].storage[1]; + assert_eq!( + ( + models.namespace.as_str(), + models.key.as_str(), + models.expires_when_unused + ), + ("deployment", "open-webui-1", false) + ); + assert_eq!(models.storage_key(), deployment_data(&deployment)); } #[test] @@ -1552,7 +1582,7 @@ fn master_catalog_comfyui_launch_attaches_deployment_data_and_shared_models() { .expect("comfyui application"); let mut deployment = deployment(ApplicationDeploymentDesiredState::Present); deployment.application_deployment_id = "comfyui-1".to_owned(); - deployment.snapshot = Some(resolved_snapshot(comfyui)); + deployment.snapshot = Some(resolved_snapshot(comfyui.clone())); // The launch needs an image it can pin; the storage bindings under test stay the catalog's. deployment .snapshot @@ -1572,13 +1602,8 @@ fn master_catalog_comfyui_launch_attaches_deployment_data_and_shared_models() { environment_variable: "COMFYUI_VRAM_MODE".to_owned(), value: "auto".to_owned(), }]; - deployment.storage_attachments = ["data", "models"] - .map(|binding| ApplicationStorageAttachment { - service_name: "comfyui".to_owned(), - binding_name: binding.to_owned(), - storage_name: format!("comfyui-{binding}"), - }) - .to_vec(); + deployment.storage_attachments = + super::launch::storage_attachments(&comfyui, &[]).expect("attachments"); let launch = workload_group_launch( &deployment, @@ -2730,6 +2755,7 @@ async fn unknown_group_after_restart_is_rehydrated_before_present_or_absent_reco service_name: "web".to_owned(), binding_name: "data".to_owned(), storage_name: "web-data".to_owned(), + placement: StoragePlacement::Deployment as i32, }]; desired .upsert(deployment.clone(), 0) @@ -2764,6 +2790,7 @@ async fn unknown_group_after_restart_is_rehydrated_before_present_or_absent_reco target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -2813,6 +2840,7 @@ async fn cold_absent_reconciliation_passes_stable_identity_to_runtime_cleanup() target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -3007,6 +3035,7 @@ async fn teardown_startup_allows_target_drift_and_retains_missing_runtime_for_re target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("teardown controller without configured runtime"); @@ -3088,6 +3117,7 @@ async fn teardown_retries_exact_network_only_residue_and_finalizes_records() { target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -3475,6 +3505,7 @@ async fn teardown_deletes_application_data_and_retries_a_failed_removal_while_pa service_name: "web".to_owned(), binding_name: "data".to_owned(), storage_name: "web-data".to_owned(), + placement: StoragePlacement::Deployment as i32, }]; desired .upsert(deployment.clone(), 0) @@ -3620,6 +3651,7 @@ async fn present_completes_in_process_network_only_residue_after_failed_removal( target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -3716,6 +3748,7 @@ fn configuration_attachments_preserve_empty_literals_and_resolve_named_values() String::new(), )), input: None, + enabled_by_parameter_id: None, }, ApplicationConfigurationBinding { binding_name: "inference".to_owned(), @@ -3726,6 +3759,7 @@ fn configuration_attachments_preserve_empty_literals_and_resolve_named_values() ), ), input: None, + enabled_by_parameter_id: None, }, ]; let sources = |name: &str| { @@ -3834,6 +3868,7 @@ async fn created_deployment_retains_the_base_url_of_its_hosting_node() { 6421, ), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("manager apps controller"); @@ -3846,6 +3881,7 @@ async fn created_deployment_retains_the_base_url_of_its_hosting_node() { expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("node-hosted deployment"); @@ -3885,6 +3921,7 @@ async fn created_deployments_hold_distinct_published_ports_through_teardown_and_ ), ), input: None, + enabled_by_parameter_id: None, }); let controller = ApplicationController::new( Arc::new( @@ -3904,6 +3941,7 @@ async fn created_deployments_hold_distinct_published_ports_through_teardown_and_ 6421, ), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("manager apps controller"); @@ -3914,6 +3952,7 @@ async fn created_deployments_hold_distinct_published_ports_through_teardown_and_ expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }; let declared = |desired: &SiteRepository, deployment_id: &str| { desired @@ -4004,6 +4043,7 @@ async fn created_deployment_snapshot_retains_the_application_notes() { 6421, ), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("manager apps controller"); @@ -4016,6 +4056,7 @@ async fn created_deployment_snapshot_retains_the_application_notes() { expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("deployment"); @@ -4071,6 +4112,7 @@ fn launch_pins_only_site_host_names_from_site_host_sources() { environment_variable: environment_variable.to_owned(), source: Some(source), input: None, + enabled_by_parameter_id: None, }); } deployment.configuration_attachments = vec![ @@ -4108,19 +4150,34 @@ fn launch_pins_only_site_host_names_from_site_host_sources() { } #[test] -fn configuration_attachments_reject_secret_bindings() { +fn secret_bindings_attach_apart_from_configuration() { let mut application = application(); - application.graph.as_mut().expect("graph").services[0] + let graph = application.graph.as_mut().expect("graph"); + graph.parameters.push(secret_parameter("password")); + graph.services[0] .secret_bindings .push(ApplicationSecretBinding { - binding_name: "secret".to_owned(), + binding_name: "password".to_owned(), environment_variable: "SECRET_VALUE".to_owned(), + parameter_id: "password".to_owned(), + form: ApplicationSecretForm::Plain as i32, + enabled_by_parameter_id: None, }); + let values = [string_parameter("password", "typed-password")]; - assert!(matches!( - configuration_attachments(&application, |_| None, &[], false), - Err(ApplicationLifecycleError::FailedPrecondition) - )); + assert_eq!( + configuration_attachments(&application, |_| None, &values, false).expect("configuration"), + [] + ); + assert_eq!( + secret_attachments(application.graph.as_ref().expect("graph"), &values).expect("secrets"), + [ApplicationSecretAttachment { + service_name: "web".to_owned(), + binding_name: "password".to_owned(), + environment_variable: "SECRET_VALUE".to_owned(), + value: "typed-password".to_owned(), + }] + ); } #[test] @@ -4137,6 +4194,7 @@ fn route_name_binding_retains_its_literal_only_without_a_route_link() { ), ), input: None, + enabled_by_parameter_id: None, }); let sources = |name: &str| { (name == "configured-inference").then(|| "http://192.0.2.10:6470/v1".to_owned()) @@ -4186,6 +4244,7 @@ fn workload_launch_uses_persisted_configuration_attachments() { ), ), input: None, + enabled_by_parameter_id: None, }]; deployment.configuration_attachments = vec![ApplicationConfigurationAttachment { service_name: "web".to_owned(), @@ -4227,6 +4286,7 @@ fn changed_configuration_attachments_reject_stale_workload_observations() { ), ), input: None, + enabled_by_parameter_id: None, }]; persisted.configuration_attachments = vec![ApplicationConfigurationAttachment { service_name: "web".to_owned(), @@ -4363,6 +4423,7 @@ async fn deployment_creation_persists_the_route_link_without_a_route_name_attach target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -4375,6 +4436,7 @@ async fn deployment_creation_persists_the_route_link_without_a_route_name_attach expected_inference_route_generation: 7, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("deployment creation"); @@ -4439,6 +4501,7 @@ async fn route_input_uses_its_literal_without_a_route_link() { inference_route_resolver: test_inference_route_resolver_with(Err( InferenceRouteResolutionError::Stale, )), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -4451,6 +4514,7 @@ async fn route_input_uses_its_literal_without_a_route_link() { expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("optional route default"); @@ -4506,6 +4570,7 @@ async fn invalid_route_binding_identity_or_generation_rejects_without_mutation() target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -4523,6 +4588,7 @@ async fn invalid_route_binding_identity_or_generation_rejects_without_mutation() expected_inference_route_generation, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await; assert!(matches!( @@ -4582,6 +4648,7 @@ async fn route_binding_without_projection_rejects_before_resolution_or_mutation( inference_route_resolver: Arc::new(CountingCreateInferenceRouteResolver { calls: route_resolution_calls.clone(), }), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -4594,6 +4661,7 @@ async fn route_binding_without_projection_rejects_before_resolution_or_mutation( expected_inference_route_generation: 7, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await; @@ -4651,6 +4719,7 @@ async fn route_binding_failures_reject_before_site_store_or_runtime_mutation() { target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver_with(Err(resolution)), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -4663,6 +4732,7 @@ async fn route_binding_failures_reject_before_site_store_or_runtime_mutation() { expected_inference_route_generation: 7, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await; @@ -4869,6 +4939,7 @@ fn route_link_launches_with_current_name_and_retained_base_url() { ), ), input: None, + enabled_by_parameter_id: None, }, ]; selected.configuration_attachments = vec![ApplicationConfigurationAttachment { @@ -4970,11 +5041,13 @@ async fn malformed_persisted_launch_declarations_fail_before_resolution_or_runti service_name: "web".to_owned(), binding_name: "data".to_owned(), storage_name: "data".to_owned(), + placement: StoragePlacement::Deployment as i32, }, ApplicationStorageAttachment { service_name: "web".to_owned(), binding_name: "data".to_owned(), storage_name: "data-two".to_owned(), + placement: StoragePlacement::Deployment as i32, }, ]; }), @@ -5008,7 +5081,7 @@ async fn malformed_persisted_launch_declarations_fail_before_resolution_or_runti binding_name: "missing".to_owned(), mount_path: "/missing".to_owned(), kind: StorageKind::Data as i32, - scope: StorageScope::Deployment as i32, + placements: vec![StoragePlacement::Deployment as i32], environment: Vec::new(), }); }), @@ -5266,6 +5339,7 @@ async fn reconciliation_replaces_changed_completed_non_route_projection() { ), ), input: None, + enabled_by_parameter_id: None, }]; original_declaration.configuration_attachments = vec![ApplicationConfigurationAttachment { service_name: "web".to_owned(), @@ -5434,14 +5508,14 @@ fn canonical_generation_ignores_valid_ordering_and_changes_for_every_runtime_pro binding_name: "data".to_owned(), mount_path: "/data".to_owned(), kind: StorageKind::Data as i32, - scope: StorageScope::Deployment as i32, + placements: vec![StoragePlacement::Deployment as i32], environment: Vec::new(), }, StorageBinding { binding_name: "cache".to_owned(), mount_path: "/cache".to_owned(), kind: StorageKind::Data as i32, - scope: StorageScope::Deployment as i32, + placements: vec![StoragePlacement::Deployment as i32], environment: Vec::new(), }, ]; @@ -5473,11 +5547,13 @@ fn canonical_generation_ignores_valid_ordering_and_changes_for_every_runtime_pro service_name: "web".to_owned(), binding_name: "data".to_owned(), storage_name: "data".to_owned(), + placement: StoragePlacement::Deployment as i32, }, ApplicationStorageAttachment { service_name: "web".to_owned(), binding_name: "cache".to_owned(), storage_name: "cache".to_owned(), + placement: StoragePlacement::Deployment as i32, }, ]; baseline.configuration_attachments = vec![ @@ -5980,6 +6056,7 @@ async fn invalid_persisted_oci_artifact_never_mutates_the_runtime() { target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -6029,6 +6106,7 @@ async fn unavailable_configuration_source_rejects_before_desired_or_runtime_muta target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -6042,6 +6120,7 @@ async fn unavailable_configuration_source_rejects_before_desired_or_runtime_muta expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }, ) .await; @@ -6109,6 +6188,7 @@ async fn controller_rejects_retained_declarations_without_an_eligible_target() { target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) { Ok(_) => panic!("declarations for an undeclared target must be rejected"), @@ -6156,6 +6236,7 @@ async fn create_rejects_site_node_identity_outside_the_identifier_rule() { target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -6172,6 +6253,7 @@ async fn create_rejects_site_node_identity_outside_the_identifier_rule() { expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await; assert!( @@ -6197,6 +6279,7 @@ async fn create_rejects_site_node_identity_outside_the_identifier_rule() { expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("deployment creation"); @@ -6479,6 +6562,7 @@ async fn creation_failure_is_local_redacted_logged_retried_and_does_not_block_la 6470, ), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -6550,6 +6634,7 @@ async fn creation_failure_is_local_redacted_logged_retried_and_does_not_block_la expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("unrelated deployment failure does not block create"); @@ -6739,6 +6824,7 @@ async fn create_accepts_a_fresh_node_target_and_rejects_ineligible_pairs_with_th expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) }; @@ -6831,6 +6917,7 @@ async fn node_deployments_follow_node_status_and_stay_pending_while_disconnected expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("node deployment"); @@ -6980,6 +7067,7 @@ async fn node_deployments_stay_stale_until_the_current_node_process_lists_them() expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("node deployment") @@ -7047,6 +7135,7 @@ async fn unit_memory_is_read_from_current_node_status_and_unavailable_otherwise( expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("node deployment") @@ -7297,6 +7386,7 @@ fn controller_restart_rejects_a_retained_deployment_whose_target_is_undeclared() target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ); @@ -7699,6 +7789,7 @@ async fn reconcile_after_target_change( target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator: test_secret_generator(), }, ) .expect("Apps controller"); @@ -7957,11 +8048,437 @@ async fn invalid_parameter_values_reject_creation_without_mutation() { std::fs::remove_dir_all(directory).expect("test directory cleanup"); } +#[tokio::test] +async fn deployment_creation_retains_each_chosen_or_default_storage_placement() { + let directory = test_directory(); + let (controller, desired) = test_controller_for( + &directory, + application_with_storage_choices(), + test_secret_generator(), + ); + let placements = |deployment: &ApplicationDeploymentDeclaration| { + deployment + .storage_attachments + .iter() + .map(|attachment| { + ( + attachment.binding_name.clone(), + StoragePlacement::try_from(attachment.placement).expect("placement"), + ) + }) + .collect::>() + }; + + controller + .create_deployment(parameter_request(Vec::new())) + .await + .expect("deployment with default placements"); + controller + .create_deployment(CreateApplicationDeploymentRequest { + storage_choices: vec![storage_choice("models", StoragePlacement::Deployment)], + ..parameter_request(Vec::new()) + }) + .await + .expect("deployment with a chosen placement"); + + let deployments = desired.snapshot().expect("declarations").deployments; + assert_eq!( + deployments.iter().map(placements).collect::>(), + [ + vec![ + ("data".to_owned(), StoragePlacement::Deployment), + ("models".to_owned(), StoragePlacement::Application), + ], + vec![ + ("data".to_owned(), StoragePlacement::Deployment), + ("models".to_owned(), StoragePlacement::Deployment), + ], + ] + ); + drop(controller); + fs::remove_dir_all(directory).expect("test directory cleanup"); +} + +#[tokio::test] +async fn invalid_storage_choices_reject_creation_without_mutation() { + let directory = test_directory(); + let (controller, desired) = test_controller_for( + &directory, + application_with_storage_choices(), + test_secret_generator(), + ); + let cases = [ + ( + vec![storage_choice("data", StoragePlacement::Application)], + "Application storage binding \"data\" of service \"web\" does not allow placement STORAGE_PLACEMENT_APPLICATION", + ), + ( + vec![storage_choice("models", StoragePlacement::Unspecified)], + "Application storage binding \"models\" of service \"web\" does not allow placement STORAGE_PLACEMENT_UNSPECIFIED", + ), + ( + vec![storage_choice("cache", StoragePlacement::Deployment)], + "Application storage binding \"cache\" of service \"web\" is not declared by the application", + ), + ( + vec![ApplicationStorageChoice { + service_name: "worker".to_owned(), + ..storage_choice("models", StoragePlacement::Deployment) + }], + "Application storage binding \"models\" of service \"worker\" is not declared by the application", + ), + ( + vec![ + storage_choice("models", StoragePlacement::Deployment), + storage_choice("models", StoragePlacement::Application), + ], + "Application storage binding \"models\" of service \"web\" is chosen more than once", + ), + ]; + + for (choices, expected) in cases { + let error = controller + .create_deployment(CreateApplicationDeploymentRequest { + storage_choices: choices, + ..parameter_request(Vec::new()) + }) + .await + .expect_err("invalid storage choice"); + assert!(matches!( + error, + ApplicationLifecycleError::InvalidStorageChoice { .. } + )); + assert_eq!(error.to_string(), expected); + } + assert!( + desired + .snapshot() + .expect("declaration snapshot") + .deployments + .is_empty() + ); + drop(controller); + fs::remove_dir_all(directory).expect("test directory cleanup"); +} + +#[tokio::test] +async fn deployment_creation_generates_missing_secrets_once_and_keeps_typed_ones() { + let directory = test_directory(); + let (controller, desired) = + secret_parameter_test_controller(&directory, test_secret_generator()); + + let generated = controller + .create_deployment(parameter_request(Vec::new())) + .await + .expect("deployment with a generated secret"); + let typed = controller + .create_deployment(parameter_request(vec![ + string_parameter("username", "operator"), + string_parameter("password", "typed-password-1"), + ])) + .await + .expect("deployment with a typed secret"); + + let generated_password = sequential_secret(1, 32); + assert_eq!(generated_password, "secret1xxxxxxxxxxxxxxxxxxxxxxxxx"); + let snapshot = generated.snapshot.as_ref().expect("snapshot"); + assert_eq!( + snapshot.parameter_values, + [ + string_parameter("username", "admin"), + string_parameter("password", &generated_password), + ] + ); + let typed_snapshot = typed.snapshot.as_ref().expect("snapshot"); + assert_eq!( + typed_snapshot.parameter_values, + [ + string_parameter("username", "operator"), + string_parameter("password", "typed-password-1"), + ] + ); + + let declarations = desired + .snapshot() + .expect("persisted declarations") + .deployments; + let declaration = declarations + .iter() + .find(|declaration| { + declaration.application_deployment_id == generated.application_deployment_id + }) + .expect("generated declaration") + .clone(); + // The bcrypt form is a hash computed at create that verifies against the generated value. + let password_hash = declaration.secret_attachments[1].value.clone(); + assert!(password_hash.starts_with("$2b$10$")); + assert!(bcrypt::verify(&generated_password, &password_hash).expect("verifiable hash")); + assert_eq!( + declaration.secret_attachments, + [ + ApplicationSecretAttachment { + service_name: "web".to_owned(), + binding_name: "password".to_owned(), + environment_variable: "PASSWORD".to_owned(), + value: generated_password.clone(), + }, + ApplicationSecretAttachment { + service_name: "web".to_owned(), + binding_name: "password-hash".to_owned(), + environment_variable: "PASSWORD_HASH".to_owned(), + value: password_hash.clone(), + }, + ] + ); + assert_eq!( + declaration + .configuration_attachments + .iter() + .map(|attachment| ( + attachment.environment_variable.as_str(), + attachment.value.as_str() + )) + .collect::>(), + [("USERNAME", "admin")] + ); + let launch = |declaration: &ApplicationDeploymentDeclaration| { + workload_group_launch( + declaration, + Ipv4Addr::LOCALHOST, + test_inference_route_resolver().as_ref(), + ) + .expect("workload launch") + }; + let first_launch = launch(&declaration); + assert_eq!( + first_launch.units[&WorkloadUnitName("web".to_owned())].environment, + BTreeMap::from([ + ("PASSWORD".to_owned(), generated_password.clone()), + ("PASSWORD_HASH".to_owned(), password_hash.clone()), + ("USERNAME".to_owned(), "admin".to_owned()), + ]) + ); + assert_eq!(launch(&declaration), first_launch); + // Launch types never show a secret in their `Debug` form. + let launch_debug = format!("{first_launch:?}"); + assert!(!launch_debug.contains(&generated_password)); + assert!(!launch_debug.contains(&password_hash)); + drop(controller); + + // A restarted control plane reads the retained value and hash back, and never generates or + // hashes again. + let path = directory.join("site.yaml"); + let mut persisted_site = site(); + persisted_site.application_deployments = declarations.clone(); + fs::write( + &path, + dotcore_proto::yaml::to_string(&persisted_site).expect("site YAML"), + ) + .expect("write site YAML"); + let reloaded = ApplicationDeclarationRepository::snapshot( + &SiteRepository::load(&path, crate::test_support::site_declaration_rules()) + .expect("reloaded site repository"), + ) + .expect("validated declarations after reload"); + assert_eq!(reloaded.deployments, declarations); + let reloaded_declaration = reloaded + .deployments + .iter() + .find(|declaration| { + declaration.application_deployment_id == generated.application_deployment_id + }) + .expect("reloaded declaration"); + let reloaded_launch = launch(reloaded_declaration); + assert_eq!(reloaded_launch, first_launch); + assert_eq!( + reloaded_launch.correlation.reconcile_generation, + first_launch.correlation.reconcile_generation + ); + fs::remove_dir_all(directory).expect("test directory cleanup"); +} + +#[tokio::test] +async fn bindings_enabled_by_a_switch_reach_the_service_only_when_it_is_on() { + let directory = test_directory(); + let (controller, desired) = test_controller_for( + &directory, + application_with_login_switch(), + test_secret_generator(), + ); + let environment = |declaration: &ApplicationDeploymentDeclaration| { + workload_group_launch( + declaration, + Ipv4Addr::LOCALHOST, + test_inference_route_resolver().as_ref(), + ) + .expect("workload launch") + .units[&WorkloadUnitName("web".to_owned())] + .environment + .clone() + }; + let declaration = |deployment_id: &str| { + desired + .snapshot() + .expect("persisted declarations") + .deployments + .into_iter() + .find(|declaration| declaration.application_deployment_id == deployment_id) + .expect("persisted declaration") + }; + + // Off: the switch's own binding is unchanged, the switched bindings get no attachment and no + // hash, and the generated secret is still kept. + let off = controller + .create_deployment(parameter_request(Vec::new())) + .await + .expect("deployment with login off"); + let off = declaration(&off.application_deployment_id); + let off_password = sequential_secret(1, 32); + assert_eq!( + off.snapshot.as_ref().expect("snapshot").parameter_values, + [ + string_parameter("username", "admin"), + string_parameter("password", &off_password), + boolean_parameter("login", false), + ] + ); + assert_eq!( + off.configuration_attachments + .iter() + .map(|attachment| ( + attachment.environment_variable.as_str(), + attachment.value.as_str() + )) + .collect::>(), + [("LOGIN", "false")] + ); + assert_eq!(off.secret_attachments, []); + assert_eq!( + environment(&off), + BTreeMap::from([("LOGIN".to_owned(), "false".to_owned())]) + ); + + // On: every binding reaches the service, the secret in each of its forms. + let on = controller + .create_deployment(parameter_request(vec![boolean_parameter("login", true)])) + .await + .expect("deployment with login on"); + let on = declaration(&on.application_deployment_id); + let on_password = sequential_secret(2, 32); + let password_hash = on + .secret_attachments + .iter() + .find(|attachment| attachment.binding_name == "password-hash") + .expect("bcrypt attachment") + .value + .clone(); + assert!(bcrypt::verify(&on_password, &password_hash).expect("verifiable hash")); + assert_eq!( + environment(&on), + BTreeMap::from([ + ("LOGIN".to_owned(), "true".to_owned()), + ("PASSWORD".to_owned(), on_password.clone()), + ("PASSWORD_HASH".to_owned(), password_hash), + ("USERNAME".to_owned(), "admin".to_owned()), + ]) + ); + drop(controller); + fs::remove_dir_all(directory).expect("test directory cleanup"); +} + +#[tokio::test] +async fn invalid_secret_and_text_values_reject_creation_without_quoting_them() { + let directory = test_directory(); + let (controller, desired) = + secret_parameter_test_controller(&directory, test_secret_generator()); + let cases = [ + ( + string_parameter("password", ""), + "Application parameter \"password\" has an invalid value: it is shorter than the minimum length of 12", + ), + ( + string_parameter("password", "short-pass"), + "Application parameter \"password\" has an invalid value: it is shorter than the minimum length of 12", + ), + ( + string_parameter("password", "typed=password-1"), + "Application parameter \"password\" has an invalid value: it contains `=` or a control character", + ), + ( + string_parameter("password", "typed-password\n1"), + "Application parameter \"password\" has an invalid value: it contains `=` or a control character", + ), + ( + string_parameter("username", ""), + "Application parameter \"username\" has an invalid value: it is shorter than the minimum length of 1", + ), + ( + string_parameter("username", &"u".repeat(65)), + "Application parameter \"username\" has an invalid value: it is longer than the maximum length of 64", + ), + ]; + + for (value, expected) in cases { + let error = controller + .create_deployment(parameter_request(vec![value.clone()])) + .await + .expect_err("invalid parameter value"); + assert!(matches!( + error, + ApplicationLifecycleError::InvalidParameter { .. } + )); + assert_eq!(error.to_string(), expected); + if let Some(parameter_value::Value::String(value)) = &value.value + && !value.is_empty() + { + assert!(!format!("{error:?}").contains(value.as_str())); + } + } + assert!( + desired + .snapshot() + .expect("declaration snapshot") + .deployments + .is_empty() + ); + drop(controller); + std::fs::remove_dir_all(directory).expect("test directory cleanup"); +} + +fn secret_parameter_test_controller( + directory: &Path, + secret_generator: Arc, +) -> ( + TestController, + TestApplicationDeclarationRepository, +) { + test_controller_for( + directory, + application_with_secret_parameters(), + secret_generator, + ) +} + fn parameter_test_controller( directory: &Path, ) -> ( TestController, TestApplicationDeclarationRepository, +) { + test_controller_for( + directory, + application_with_parameters(), + test_secret_generator(), + ) +} + +fn test_controller_for( + directory: &Path, + application: dotcore_proto::dotcore::catalog::v1::Application, + secret_generator: Arc, +) -> ( + TestController, + TestApplicationDeclarationRepository, ) { let desired = TestApplicationDeclarationRepository::open(directory, "site-one") .expect("declaration repository"); @@ -7979,7 +8496,7 @@ fn parameter_test_controller( let controller = TestController::new( Arc::new( CatalogBuilder::new() - .application("test", application_with_parameters()) + .application("test", application) .build() .expect("Catalog"), ), @@ -7990,6 +8507,7 @@ fn parameter_test_controller( target_capabilities: test_target_capability_source(), configuration_sources: ApplicationConfigurationSources::default(), inference_route_resolver: test_inference_route_resolver(), + secret_generator, }, ) .expect("Apps controller"); @@ -8004,6 +8522,22 @@ fn parameter_request(parameter_values: Vec) -> CreateApplication expected_inference_route_generation: 0, application_deployment_name: None, parameter_values, + storage_choices: Vec::new(), + } +} + +fn storage_choice(binding_name: &str, placement: StoragePlacement) -> ApplicationStorageChoice { + ApplicationStorageChoice { + service_name: "web".to_owned(), + binding_name: binding_name.to_owned(), + placement: placement as i32, + } +} + +fn boolean_parameter(id: &str, value: bool) -> ParameterValue { + ParameterValue { + parameter_id: id.to_owned(), + value: Some(parameter_value::Value::Boolean(value)), } } diff --git a/backend/core/src/application/mod.rs b/backend/core/src/application/mod.rs index 6b5be8b2..dffb25f5 100644 --- a/backend/core/src/application/mod.rs +++ b/backend/core/src/application/mod.rs @@ -1,8 +1,10 @@ //! Apps-controller deployment lifecycle, persistence, and management API boundaries. +mod binding; mod controller; mod published_port; pub mod repository; +mod secret_form; pub mod service; pub(crate) mod workloads; @@ -15,5 +17,5 @@ pub use controller::{ ApplicationConfigurationSources, ApplicationController, ApplicationControllerConfiguration, ApplicationDeploymentLogController, ApplicationLifecycleController, ApplicationLifecycleError, InferenceRouteResolutionError, InferenceRouteResolver, ParameterRejection, - ResolvedInferenceRoute, RetainedTargetMismatch, TargetIneligibility, + ResolvedInferenceRoute, RetainedTargetMismatch, StorageChoiceRejection, TargetIneligibility, }; diff --git a/backend/core/src/application/repository.rs b/backend/core/src/application/repository.rs index 09818443..c0bd9d30 100644 --- a/backend/core/src/application/repository.rs +++ b/backend/core/src/application/repository.rs @@ -1,6 +1,6 @@ //! Owner-facing repositories for application declarations and current deployment state. -use std::collections::BTreeSet; +use std::collections::{BTreeMap, BTreeSet}; use dotcore_catalog::{ OciArtifactPolicy, typed_parameters::canonical_value_text, valid_configuration_value, @@ -17,11 +17,11 @@ use dotcore_proto::dotcore::{ application::v1::{ ApplicationConfigurationAttachment, ApplicationDeploymentDeclaration, ApplicationDeploymentDesiredState, ApplicationLifecycleState, ApplicationReadinessState, - ApplicationServiceLifecycleState, ApplicationStorageAttachment, - CurrentApplicationDeployment, ResolvedApplicationDeploymentSnapshot, - ResolvedApplicationServiceSelection, + ApplicationSecretAttachment, ApplicationServiceLifecycleState, + ApplicationStorageAttachment, CurrentApplicationDeployment, + ResolvedApplicationDeploymentSnapshot, ResolvedApplicationServiceSelection, }, - catalog::v1::application_configuration_binding, + catalog::v1::{ApplicationSecretForm, StoragePlacement, application_configuration_binding}, site::store::v1::SiteCurrentState, }; use dotcore_site::{ @@ -31,7 +31,9 @@ use dotcore_site::{ store::{CurrentStateStore, StoreError, StoredCurrentState, reserve_lifecycle_generation}, }; -use super::published_port::valid_published_port; +use super::{ + binding::binding_enabled, published_port::valid_published_port, secret_form::is_bcrypt_hash, +}; /// An isolated read view of apps-controller declarations. #[derive(Clone, Debug, PartialEq)] @@ -589,6 +591,7 @@ pub(crate) fn validate_declaration( validate_storage_attachments(deployment)?; validate_inference_route_binding(deployment)?; validate_configuration_attachments(deployment)?; + validate_secret_attachments(deployment)?; Ok(()) } @@ -627,7 +630,7 @@ fn validate_resolved_snapshot( .services .iter() .map(|service| (service.service_name.as_str(), service)) - .collect::>(); + .collect::>(); let mut selected = BTreeSet::new(); for selection in &snapshot.service_selections { let service = services @@ -719,34 +722,44 @@ fn validate_storage_attachments( .graph .as_ref() .expect("Application snapshot was validated"); - let expected = graph + let allowed = graph .services .iter() .flat_map(|service| { - service - .storage_bindings - .iter() - .map(move |binding| (service.service_name.as_str(), binding.binding_name.as_str())) + service.storage_bindings.iter().map(move |binding| { + ( + (service.service_name.as_str(), binding.binding_name.as_str()), + binding.placements.as_slice(), + ) + }) }) - .collect::>(); + .collect::>(); let mut actual = BTreeSet::new(); let mut storage_names = BTreeSet::new(); for ApplicationStorageAttachment { service_name, binding_name, storage_name, + placement, } in &deployment.storage_attachments { + let key = (service_name.as_str(), binding_name.as_str()); + // Every retained attachment names a known binding, so equal counts mean every binding has one. + let allowed_placement = *placement != StoragePlacement::Unspecified as i32 + && allowed + .get(&key) + .is_some_and(|placements| placements.contains(placement)); if !valid_id(service_name) || !valid_id(binding_name) || !valid_id(storage_name) - || !actual.insert((service_name.as_str(), binding_name.as_str())) + || !allowed_placement + || !actual.insert(key) || !storage_names.insert(storage_name.as_str()) { return Err(ApplicationRepositoryError::InvalidState); } } - if actual != expected { + if actual.len() != allowed.len() { return Err(ApplicationRepositoryError::InvalidState); } Ok(()) @@ -763,23 +776,24 @@ fn validate_configuration_attachments( .graph .as_ref() .expect("Application snapshot was validated"); - let expected = graph - .services - .iter() - .flat_map(|service| { - service - .configuration_bindings - .iter() - .filter_map(move |binding| { - (deployment.inference_route_id.is_none() - || !is_inference_route_projection(binding)) - .then_some(( - (service.service_name.as_str(), binding.binding_name.as_str()), - binding.environment_variable.as_str(), - )) - }) - }) - .collect::>(); + let mut expected = BTreeSet::new(); + for service in &graph.services { + for binding in &service.configuration_bindings { + let route_projected = + deployment.inference_route_id.is_some() && is_inference_route_projection(binding); + let enabled = binding_enabled( + binding.enabled_by_parameter_id.as_deref(), + &snapshot.parameter_values, + ) + .ok_or(ApplicationRepositoryError::InvalidState)?; + if enabled && !route_projected { + expected.insert(( + (service.service_name.as_str(), binding.binding_name.as_str()), + binding.environment_variable.as_str(), + )); + } + } + } let mut actual = BTreeSet::new(); let mut environment_destinations = BTreeSet::new(); for ApplicationConfigurationAttachment { @@ -814,6 +828,10 @@ fn validate_configuration_attachments( else { continue; }; + let key = (service.service_name.as_str(), binding.binding_name.as_str()); + if !expected.contains(&(key, binding.environment_variable.as_str())) { + continue; + } let attachment = deployment .configuration_attachments .iter() @@ -830,6 +848,68 @@ fn validate_configuration_attachments( Ok(()) } +/// Checks that every enabled secret binding has exactly one attachment carrying its parameter's +/// retained value in the binding's form, and that a disabled one has none. A bcrypt attachment is the authoritative hash computed at create: it must be well formed, and +/// is never verified against the value here. +fn validate_secret_attachments( + deployment: &ApplicationDeploymentDeclaration, +) -> Result<(), ApplicationRepositoryError> { + let snapshot = deployment + .snapshot + .as_ref() + .expect("Application declaration snapshot was validated"); + let graph = snapshot + .graph + .as_ref() + .expect("Application snapshot was validated"); + let mut expected = BTreeMap::new(); + for service in &graph.services { + for binding in &service.secret_bindings { + let enabled = binding_enabled( + binding.enabled_by_parameter_id.as_deref(), + &snapshot.parameter_values, + ) + .ok_or(ApplicationRepositoryError::InvalidState)?; + if !enabled { + continue; + } + let value = canonical_value_text(&snapshot.parameter_values, &binding.parameter_id) + .ok_or(ApplicationRepositoryError::InvalidState)?; + let form = ApplicationSecretForm::try_from(binding.form) + .map_err(|_| ApplicationRepositoryError::InvalidState)?; + expected.insert( + (service.service_name.as_str(), binding.binding_name.as_str()), + (binding.environment_variable.as_str(), form, value), + ); + } + } + let mut attached = BTreeSet::new(); + for ApplicationSecretAttachment { + service_name, + binding_name, + environment_variable, + value: attachment, + } in &deployment.secret_attachments + { + let key = (service_name.as_str(), binding_name.as_str()); + let carries_its_form = expected.get(&key).is_some_and(|(variable, form, value)| { + variable == environment_variable + && match form { + ApplicationSecretForm::Plain => attachment == value, + ApplicationSecretForm::Bcrypt => is_bcrypt_hash(attachment), + ApplicationSecretForm::Unspecified => false, + } + }); + if !carries_its_form || !valid_configuration_value(attachment) || !attached.insert(key) { + return Err(ApplicationRepositoryError::InvalidState); + } + } + if attached.len() != expected.len() { + return Err(ApplicationRepositoryError::InvalidState); + } + Ok(()) +} + fn validate_inference_route_binding( deployment: &ApplicationDeploymentDeclaration, ) -> Result<(), ApplicationRepositoryError> { @@ -1047,15 +1127,18 @@ mod tests { use dotcore_proto::dotcore::{ application::v1::{ ApplicationConfigurationAttachment, ApplicationDeploymentDeclaration, - ApplicationDeploymentDesiredState, ResolvedApplicationDeploymentSnapshot, + ApplicationDeploymentDesiredState, ApplicationSecretAttachment, + ApplicationStorageAttachment, ResolvedApplicationDeploymentSnapshot, ResolvedApplicationServiceSelection, }, catalog::v1::{ Application, ApplicationConfigurationBinding, ApplicationEndpointProtocol, - ApplicationGraph, ApplicationParameter, ApplicationPrimaryEndpoint, ApplicationService, - ApplicationServicePort, ParameterDefinition, ParameterValue, StringEnumParameterSchema, - application_configuration_binding, parameter_definition::ParameterType, - parameter_value, + ApplicationGraph, ApplicationPrimaryEndpoint, ApplicationSecretBinding, + ApplicationSecretForm, ApplicationService, ApplicationServicePort, + BooleanParameterSchema, ParameterDefinition, ParameterValue, ParameterVisibility, + SecretParameterSchema, StorageBinding, StorageKind, StoragePlacement, + StringEnumParameterSchema, TextParameterSchema, application_configuration_binding, + parameter_definition::ParameterType, parameter_value, }, deployment::v1::{ArtifactPreparationPolicy, NodeResolvedLaunchIdentity}, }; @@ -1234,17 +1317,15 @@ mod tests { let mut deployment = valid_deployment(); let snapshot = deployment.snapshot.as_mut().expect("snapshot"); let graph = snapshot.graph.as_mut().expect("graph"); - graph.parameters = vec![ApplicationParameter { - definition: Some(ParameterDefinition { - parameter_id: "mode".to_owned(), - display_name: "Mode".to_owned(), - description: String::new(), - parameter_type: Some(ParameterType::StringEnum(StringEnumParameterSchema { - values: vec!["auto".to_owned(), "highvram".to_owned()], - default_value: Some("auto".to_owned()), - })), - }), - advanced: true, + graph.parameters = vec![ParameterDefinition { + parameter_id: "mode".to_owned(), + display_name: "Mode".to_owned(), + description: String::new(), + parameter_type: Some(ParameterType::StringEnum(StringEnumParameterSchema { + values: vec!["auto".to_owned(), "highvram".to_owned()], + default_value: Some("auto".to_owned()), + })), + visibility: ParameterVisibility::Advanced as i32, }]; graph.services[0].configuration_bindings = vec![ApplicationConfigurationBinding { binding_name: "mode".to_owned(), @@ -1253,6 +1334,7 @@ mod tests { "mode".to_owned(), )), input: None, + enabled_by_parameter_id: None, }]; snapshot.parameter_values = vec![ParameterValue { parameter_id: "mode".to_owned(), @@ -1274,6 +1356,236 @@ mod tests { )); } + #[test] + fn storage_attachment_retains_one_placement_its_binding_allows() { + let with_attachments = |attachments: &[(&str, StoragePlacement)]| { + let mut deployment = valid_deployment(); + let graph = deployment + .snapshot + .as_mut() + .and_then(|snapshot| snapshot.graph.as_mut()) + .expect("graph"); + graph.services[0].storage_bindings = ["data", "models"] + .map(|name| StorageBinding { + binding_name: name.to_owned(), + mount_path: format!("/{name}"), + kind: StorageKind::Data as i32, + placements: vec![ + StoragePlacement::Application as i32, + StoragePlacement::Deployment as i32, + ], + environment: Vec::new(), + }) + .to_vec(); + graph.services[0].storage_bindings[0].placements = + vec![StoragePlacement::Deployment as i32]; + deployment.storage_attachments = attachments + .iter() + .map(|(name, placement)| ApplicationStorageAttachment { + service_name: "service".to_owned(), + binding_name: (*name).to_owned(), + storage_name: format!("service-{name}"), + placement: *placement as i32, + }) + .collect(); + deployment + }; + + for valid in [ + with_attachments(&[ + ("data", StoragePlacement::Deployment), + ("models", StoragePlacement::Application), + ]), + with_attachments(&[ + ("data", StoragePlacement::Deployment), + ("models", StoragePlacement::Deployment), + ]), + ] { + assert!(validate_declarations(&[valid]).is_ok()); + } + for invalid in [ + with_attachments(&[ + ("data", StoragePlacement::Application), + ("models", StoragePlacement::Application), + ]), + with_attachments(&[ + ("data", StoragePlacement::Deployment), + ("models", StoragePlacement::Unspecified), + ]), + with_attachments(&[("data", StoragePlacement::Deployment)]), + with_attachments(&[ + ("data", StoragePlacement::Deployment), + ("models", StoragePlacement::Application), + ("models", StoragePlacement::Deployment), + ]), + with_attachments(&[ + ("data", StoragePlacement::Deployment), + ("models", StoragePlacement::Application), + ("other", StoragePlacement::Deployment), + ]), + ] { + assert!(matches!( + validate_declarations(&[invalid]), + Err(ApplicationRepositoryError::InvalidState) + )); + } + } + + #[test] + fn secret_attachment_carries_its_form_unless_its_binding_is_disabled() { + let with_secret = |attachment_value: &str| { + let mut deployment = valid_deployment(); + let snapshot = deployment.snapshot.as_mut().expect("snapshot"); + let graph = snapshot.graph.as_mut().expect("graph"); + graph.parameters = vec![ + ParameterDefinition { + parameter_id: "username".to_owned(), + display_name: "Username".to_owned(), + description: String::new(), + parameter_type: Some(ParameterType::Text(TextParameterSchema { + minimum_length: 1, + maximum_length: 64, + default_value: Some("admin".to_owned()), + })), + visibility: ParameterVisibility::Basic as i32, + }, + ParameterDefinition { + parameter_id: "password".to_owned(), + display_name: "Password".to_owned(), + description: String::new(), + parameter_type: Some(ParameterType::Secret(SecretParameterSchema { + generated_length: 32, + minimum_length: 12, + })), + visibility: ParameterVisibility::Basic as i32, + }, + ]; + graph.services[0].configuration_bindings = vec![ApplicationConfigurationBinding { + binding_name: "username".to_owned(), + environment_variable: "USERNAME".to_owned(), + source: Some(application_configuration_binding::Source::ParameterId( + "username".to_owned(), + )), + input: None, + enabled_by_parameter_id: None, + }]; + graph.services[0].secret_bindings = vec![ApplicationSecretBinding { + binding_name: "password".to_owned(), + environment_variable: "PASSWORD".to_owned(), + parameter_id: "password".to_owned(), + form: ApplicationSecretForm::Plain as i32, + enabled_by_parameter_id: None, + }]; + snapshot.parameter_values = vec![ + ParameterValue { + parameter_id: "username".to_owned(), + value: Some(parameter_value::Value::String("admin".to_owned())), + }, + ParameterValue { + parameter_id: "password".to_owned(), + value: Some(parameter_value::Value::String( + "retained-password".to_owned(), + )), + }, + ]; + deployment.configuration_attachments = vec![ApplicationConfigurationAttachment { + service_name: "service".to_owned(), + binding_name: "username".to_owned(), + environment_variable: "USERNAME".to_owned(), + value: "admin".to_owned(), + }]; + deployment.secret_attachments = vec![ApplicationSecretAttachment { + service_name: "service".to_owned(), + binding_name: "password".to_owned(), + environment_variable: "PASSWORD".to_owned(), + value: attachment_value.to_owned(), + }]; + deployment + }; + + assert!(validate_declarations(&[with_secret("retained-password")]).is_ok()); + for invalid in [with_secret("other-password-1"), { + let mut missing = with_secret("retained-password"); + missing.secret_attachments.clear(); + missing + }] { + assert!(matches!( + validate_declarations(&[invalid]), + Err(ApplicationRepositoryError::InvalidState) + )); + } + + // A bcrypt attachment only needs to be a well-formed hash: this one was computed from + // another value and still reads back, since reads never verify it. + let with_hash = |attachment_value: &str| { + let mut deployment = with_secret(attachment_value); + deployment + .snapshot + .as_mut() + .and_then(|snapshot| snapshot.graph.as_mut()) + .expect("graph") + .services[0] + .secret_bindings[0] + .form = ApplicationSecretForm::Bcrypt as i32; + deployment + }; + let foreign_hash = "$2b$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhWy"; + assert!(!bcrypt::verify("retained-password", foreign_hash).expect("verifiable hash")); + assert!(validate_declarations(&[with_hash(foreign_hash)]).is_ok()); + for invalid in [ + "retained-password", + "$2b$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhW", + ] { + assert!(matches!( + validate_declarations(&[with_hash(invalid)]), + Err(ApplicationRepositoryError::InvalidState) + )); + } + + // Bindings switched off by their boolean parameter have no attachment; the secret value + // is still retained. + let with_switch = |login: Option, attached: bool| { + let mut deployment = with_secret("retained-password"); + let snapshot = deployment.snapshot.as_mut().expect("snapshot"); + let graph = snapshot.graph.as_mut().expect("graph"); + graph.parameters.push(ParameterDefinition { + parameter_id: "login".to_owned(), + display_name: "Login".to_owned(), + description: String::new(), + parameter_type: Some(ParameterType::Boolean(BooleanParameterSchema { + default_value: Some(false), + })), + visibility: ParameterVisibility::Basic as i32, + }); + graph.services[0].configuration_bindings[0].enabled_by_parameter_id = + Some("login".to_owned()); + graph.services[0].secret_bindings[0].enabled_by_parameter_id = Some("login".to_owned()); + if let Some(login) = login { + snapshot.parameter_values.push(ParameterValue { + parameter_id: "login".to_owned(), + value: Some(parameter_value::Value::Boolean(login)), + }); + } + if !attached { + deployment.configuration_attachments.clear(); + deployment.secret_attachments.clear(); + } + deployment + }; + assert!(validate_declarations(&[with_switch(Some(true), true)]).is_ok()); + assert!(validate_declarations(&[with_switch(Some(false), false)]).is_ok()); + for invalid in [ + with_switch(Some(true), false), + with_switch(Some(false), true), + with_switch(None, false), + ] { + assert!(matches!( + validate_declarations(&[invalid]), + Err(ApplicationRepositoryError::InvalidState) + )); + } + } + #[test] fn resolved_snapshot_validation_rejects_incomplete_or_ambiguous_selection_contracts() { let valid = valid_deployment(); diff --git a/backend/core/src/application/secret_form.rs b/backend/core/src/application/secret_form.rs new file mode 100644 index 00000000..1fbca3d8 --- /dev/null +++ b/backend/core/src/application/secret_form.rs @@ -0,0 +1,79 @@ +//! Bcrypt form of a secret parameter's value, for services that take a password hash. +//! +//! The hash is computed once, when a deployment is created, with a fresh random salt, and is +//! retained as the authoritative form; it is never recomputed, so relaunches see the same +//! environment. Reading a retained hash back only checks that it is well formed and never verifies +//! it against the value, which would cost a full hash on every read. + +use std::str::FromStr; + +use bcrypt::HashParts; + +/// Work factor of a new hash; the one services such as n8n use for their own hashes. +const COST: u32 = 10; + +/// Version prefix of every hash this form computes. +const VERSION_PREFIX: &str = "$2b$"; + +/// Lowest and highest work factors a well-formed hash may carry. +const VALID_COSTS: std::ops::RangeInclusive = 4..=31; + +/// Failure to compute the bcrypt form of a value. +#[derive(Debug, thiserror::Error)] +pub(super) enum BcryptFormError { + #[error("Bcrypt hashing failed")] + Hash(#[source] bcrypt::BcryptError), + + #[error("Bcrypt hash does not verify against its value")] + Mismatch, +} + +/// Returns a bcrypt hash of `value` with a fresh random salt, verified against `value` once. +pub(super) fn bcrypt_hash(value: &str) -> Result { + let hash = bcrypt::hash(value, COST).map_err(BcryptFormError::Hash)?; + match bcrypt::verify(value, &hash).map_err(BcryptFormError::Hash)? { + true => Ok(hash), + false => Err(BcryptFormError::Mismatch), + } +} + +/// Reports whether `text` is a well-formed bcrypt hash in this form's version, without verifying +/// it against any value. +pub(super) fn is_bcrypt_hash(text: &str) -> bool { + text.starts_with(VERSION_PREFIX) + && HashParts::from_str(text).is_ok_and(|parts| VALID_COSTS.contains(&parts.get_cost())) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn bcrypt_hash_verifies_against_its_value_with_a_fresh_salt() { + let first = bcrypt_hash("retained-password").expect("first hash"); + let second = bcrypt_hash("retained-password").expect("second hash"); + + assert!(first.starts_with("$2b$10$")); + assert!(is_bcrypt_hash(&first)); + assert!(bcrypt::verify("retained-password", &first).expect("verify")); + assert!(!bcrypt::verify("other-password", &first).expect("verify")); + assert_ne!(first, second); + } + + #[test] + fn well_formed_hashes_are_recognized_without_their_value() { + let valid = "$2b$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhWy"; + + assert!(is_bcrypt_hash(valid)); + for invalid in [ + "", + "retained-password", + "$2a$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhWy", + "$2b$99$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhWy", + "$2b$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhW", + "$2b$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lh!y", + ] { + assert!(!is_bcrypt_hash(invalid), "{invalid:?} is not a hash"); + } + } +} diff --git a/backend/core/src/application/service.rs b/backend/core/src/application/service.rs index 377428e4..aaa0712a 100644 --- a/backend/core/src/application/service.rs +++ b/backend/core/src/application/service.rs @@ -504,9 +504,10 @@ impl TonicStatusExt for ApplicationLifecycleError { | Self::TargetPlatformMismatch { .. } | Self::UnsupportedServicePlatform { .. } | Self::RetainedTargetIncompatible(_) => Status::failed_precondition(self.to_string()), - Self::InvalidArgument | Self::InvalidName | Self::InvalidParameter { .. } => { - Status::invalid_argument(self.to_string()) - } + Self::InvalidArgument + | Self::InvalidName + | Self::InvalidParameter { .. } + | Self::InvalidStorageChoice { .. } => Status::invalid_argument(self.to_string()), Self::NotFound => Status::not_found(self.to_string()), Self::PublishedPortsExhausted => Status::resource_exhausted(self.to_string()), Self::InferenceRoute(error) => error.to_tonic_status(), diff --git a/backend/core/src/composition/local_node.rs b/backend/core/src/composition/local_node.rs index 832f5eb4..e3d8b8bb 100644 --- a/backend/core/src/composition/local_node.rs +++ b/backend/core/src/composition/local_node.rs @@ -662,6 +662,7 @@ impl LocalNodeHarness { expected_inference_route_generation: route.map_or(0, |route| route.generation), application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }), ) .await @@ -1164,7 +1165,7 @@ graph: input: inferenceRoute: projection: "APPLICATION_INFERENCE_ROUTE_PROJECTION_ROUTE_NAME" - storageBindings: [{ bindingName: "data", mountPath: "/data", kind: "STORAGE_KIND_DATA", scope: "STORAGE_SCOPE_DEPLOYMENT" }] + storageBindings: [{ bindingName: "data", mountPath: "/data", kind: "STORAGE_KIND_DATA", placements: ["STORAGE_PLACEMENT_DEPLOYMENT"] }] readiness: { httpGet: { portName: "http", path: "/health" } } primaryEndpoint: serviceName: "web" diff --git a/backend/core/src/composition/mod.rs b/backend/core/src/composition/mod.rs index 0a955a0e..d3513d0a 100644 --- a/backend/core/src/composition/mod.rs +++ b/backend/core/src/composition/mod.rs @@ -370,6 +370,7 @@ where ), inference_route_resolver: inference_route_resolver(routing.route_manager.clone()), target_capabilities, + secret_generator: Arc::new(crate::RandomSecretGenerator), }, log_admission, )?; diff --git a/backend/core/src/deployment_logs/mod.rs b/backend/core/src/deployment_logs/mod.rs index 60cf7cec..53eb24c6 100644 --- a/backend/core/src/deployment_logs/mod.rs +++ b/backend/core/src/deployment_logs/mod.rs @@ -102,11 +102,6 @@ pub(crate) struct OwnerLogSources { } impl OwnerLogSources { - /// Results of a deployment whose launch holds no protected values. - pub(crate) fn unprotected(sources: Vec) -> Self { - Self::protected(sources, ProtectedLogValues::default()) - } - /// Results whose entries and diagnostics the node redacts `protected` from. pub(crate) fn protected(sources: Vec, protected: ProtectedLogValues) -> Self { Self { diff --git a/backend/core/src/development/mod.rs b/backend/core/src/development/mod.rs index 4a41c1d6..28c254c7 100644 --- a/backend/core/src/development/mod.rs +++ b/backend/core/src/development/mod.rs @@ -36,7 +36,7 @@ use dotcore_domain::{ }; use dotcore_node_hardware::hardware::SystemHardwareSnapshotCollector; use dotcore_proto::dotcore::{ - catalog::v1::{EngineParameterValues, engine_artifact::Kind as EngineArtifactKind}, + catalog::v1::{ParameterValues, engine_artifact::Kind as EngineArtifactKind}, engine::v1::EngineKind, model::v1::{ LocalModelDeploymentConfiguration, PortableModelLaunchPlan, @@ -66,8 +66,8 @@ use tokio_util::sync::CancellationToken; use crate::model::{ engine::{self, EngineIntegrationRegistry}, resolution::{ - apply_profile_defaults, resolve_local_launch, resolve_local_values_on_target, - select_engine_artifact, + EngineSecrets, apply_profile_defaults, resolve_local_launch, + resolve_local_values_on_target, select_engine_artifact, }, resolved::ResolvedModelLaunch, }; @@ -373,6 +373,7 @@ impl DirectWorkloadFacade { &configured_target, &target_inventory, engine_identity.as_ref(), + EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .map_err(DirectWorkloadLaunchError::resolution)?; self.pin_nix_engine(&mut snapshot, &cancellation).await?; @@ -1386,7 +1387,7 @@ pub struct DevelopmentLaunchFacts { pub model_artifact_id: String, pub auxiliary_artifact_ids: Vec, pub resolved_options: Vec, - pub engine_parameter_values: EngineParameterValues, + pub engine_parameter_values: ParameterValues, pub engine_release_id: String, pub engine_artifact_id: String, pub engine_version: String, diff --git a/backend/core/src/development/tests.rs b/backend/core/src/development/tests.rs index c2f63c5d..404640b7 100644 --- a/backend/core/src/development/tests.rs +++ b/backend/core/src/development/tests.rs @@ -1508,6 +1508,7 @@ async fn facade_pins_a_tag_only_engine_image_and_prepares_and_launches_by_its_im reference: reference.to_owned(), image_id: IMAGE_ID.to_owned(), }), + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("pinned launch"); let resolved = ResolvedModelLaunch::from_local_snapshot(&pinned).expect("resolved model"); @@ -1613,6 +1614,7 @@ async fn facade_pins_prepares_and_launches_a_docker_delivered_engine_of_its_own_ &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("resolved launch"); pinned @@ -1721,6 +1723,7 @@ fn seed_selected_artifacts( &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("resolved launch"); let resolved = ResolvedModelLaunch::from_snapshot(&ModelDeploymentSnapshot { diff --git a/backend/core/src/dispatch/model.rs b/backend/core/src/dispatch/model.rs index 2d880ab7..98c831da 100644 --- a/backend/core/src/dispatch/model.rs +++ b/backend/core/src/dispatch/model.rs @@ -21,6 +21,7 @@ use super::{ Admission, dispatch_admitted, registry::{KeyedRegistry, KeyedRegistryError}, }; +use crate::management_logs::ProtectedLogValues; /// One observation plus the execution target whose adapter returned it. #[derive(Clone, Debug, PartialEq, Eq)] @@ -205,7 +206,12 @@ impl RuntimeTargetMap { }); if let Err(error) = capability { let result = Err(error); - record_runtime_result(&span, &result, "create_workload"); + record_runtime_result( + &span, + &result, + "create_workload", + &ProtectedLogValues::default(), + ); return result; } let _permit = self @@ -221,11 +227,14 @@ impl RuntimeTargetMap { .await } + /// Creates a workload under an admitted lifecycle operation; a failure's log event never + /// shows the `protected` values its runtime error may quote. pub(crate) async fn create_workload_admitted( &self, token: &LifecycleOperationToken, execution_target_id: &str, workload: &WorkloadLaunch, + protected: &ProtectedLogValues, ) -> Result { let adapter = self.adapter(execution_target_id)?; workload @@ -239,7 +248,7 @@ impl RuntimeTargetMap { Admission::Operation(token), commit_fencing_error, || { - dispatch_adapter_action( + dispatch_protected_action( execution_target_id, info_span!( "runtime_create_workload", @@ -251,17 +260,21 @@ impl RuntimeTargetMap { ), "create_workload", adapter.create_workload(workload), + protected, ) }, ) .await } + /// Starts a workload under an admitted lifecycle operation; a failure's log event never + /// shows the `protected` values its runtime error may quote. pub(crate) async fn start_workload_admitted( &self, token: &LifecycleOperationToken, execution_target_id: &str, workload_id: &WorkloadId, + protected: &ProtectedLogValues, ) -> Result { let adapter = self.adapter(execution_target_id)?; dispatch_admitted( @@ -269,7 +282,7 @@ impl RuntimeTargetMap { Admission::Operation(token), commit_fencing_error, || { - dispatch_adapter_action( + dispatch_protected_action( execution_target_id, info_span!( "runtime_start_workload", @@ -280,6 +293,7 @@ impl RuntimeTargetMap { ), "start_workload", adapter.start_workload(workload_id), + protected, ) }, ) @@ -721,6 +735,25 @@ async fn dispatch_adapter_action( span: Span, operation: &'static str, action: RuntimeFuture<'_, T>, +) -> Result { + dispatch_protected_action( + execution_target_id, + span, + operation, + action, + &ProtectedLogValues::default(), + ) + .await +} + +/// Runs an adapter action like [`dispatch_adapter_action`], redacting `protected` from the +/// logged failure. +async fn dispatch_protected_action( + execution_target_id: &str, + span: Span, + operation: &'static str, + action: RuntimeFuture<'_, T>, + protected: &ProtectedLogValues, ) -> Result { let result = action @@ -730,7 +763,7 @@ async fn dispatch_adapter_action( execution_target_id: execution_target_id.into(), source, }); - record_runtime_result(&span, &result, operation); + record_runtime_result(&span, &result, operation, protected); result } @@ -752,6 +785,7 @@ fn record_runtime_result( span: &Span, result: &Result, operation: &'static str, + protected: &ProtectedLogValues, ) { match result { Ok(_) => { @@ -768,7 +802,7 @@ fn record_runtime_result( Err(error) => { span.record("outcome", "failed"); span.record("error_kind", "adapter"); - error!(parent: span, event_kind = "model_runtime_failure", operation, error_kind = "adapter", error = ?error, "Runtime operation failed"); + error!(parent: span, event_kind = "model_runtime_failure", operation, error_kind = "adapter", error = %protected.redact(&format!("{error:?}")), "Runtime operation failed"); } } } diff --git a/backend/core/src/lib.rs b/backend/core/src/lib.rs index afe67141..1476c5a3 100644 --- a/backend/core/src/lib.rs +++ b/backend/core/src/lib.rs @@ -36,8 +36,11 @@ pub(crate) mod node_memory; pub(crate) mod resource_watch; pub(crate) mod route_manager; pub(crate) mod router; +mod secret_generator; pub(crate) mod storage; +pub use secret_generator::{RandomSecretGenerator, SecretGenerator}; + pub(crate) mod inference; #[cfg(test)] diff --git a/backend/core/src/management_logs.rs b/backend/core/src/management_logs.rs index 8f89e875..e755c19b 100644 --- a/backend/core/src/management_logs.rs +++ b/backend/core/src/management_logs.rs @@ -168,31 +168,35 @@ impl DeploymentLogFollowQuery { } /// Request-time values that must be removed from deployment-log output. +/// +/// Values are distinct and kept longest first, so a value that contains a shorter one, such as a +/// password containing the username, is replaced whole before the shorter one could split it. #[derive(Clone, Debug, Default, PartialEq, Eq)] pub struct ProtectedLogValues(Vec>); impl ProtectedLogValues { pub fn new(values: impl IntoIterator) -> Self { - Self( - values - .into_iter() - .filter(|value| !value.is_empty()) - .map(String::into_bytes) - .collect(), - ) + let mut protected = Self::default(); + protected.insert(values.into_iter().map(String::into_bytes)); + protected } + /// Returns the values longest first. pub fn values(&self) -> &[Vec] { &self.0 } /// Adds the values of `other` that are not protected yet. pub(crate) fn extend(&mut self, other: ProtectedLogValues) { - for value in other.0 { - if !self.0.contains(&value) { - self.0.push(value); - } - } + self.insert(other.0); + } + + fn insert(&mut self, values: impl IntoIterator>) { + self.0 + .extend(values.into_iter().filter(|value| !value.is_empty())); + self.0 + .sort_by(|left, right| right.len().cmp(&left.len()).then_with(|| left.cmp(right))); + self.0.dedup(); } /// Returns `text` with every protected value replaced by the redaction marker, for free text @@ -206,20 +210,9 @@ impl ProtectedLogValues { } } -/// Names whose presence marks an environment variable or launch flag as carrying a secret value. -const SENSITIVE_NAMES: [&str; 5] = ["TOKEN", "SECRET", "PASSWORD", "API_KEY", "CREDENTIAL"]; - /// Stands in for the value of a sensitive launch flag in recorded custom arguments. const MASKED_ARGUMENT_VALUE: &str = ""; -/// Whether an environment variable name contains a sensitive name anywhere. -pub(crate) fn is_sensitive_environment_name(name: &str) -> bool { - let name = name.to_ascii_uppercase(); - SENSITIVE_NAMES - .iter() - .any(|sensitive| name.contains(sensitive)) -} - /// Masks the value of every long flag whose name contains a sensitive name as whole words, in the /// `--flag value` and `--flag=value` forms, keeping the flag and the token order. Best effort: /// short aliases and secrets without a sensitive flag name are not detected. @@ -239,7 +232,7 @@ pub(crate) fn mask_sensitive_arguments(tokens: &[String]) -> Vec { Some((name, _)) => (name, true), None => (flag, false), }; - if !is_sensitive_flag_name(name) { + if !dotcore_catalog::is_sensitive_name(name) { masked.push(token.clone()); } else if has_value { masked.push(format!("--{name}={MASKED_ARGUMENT_VALUE}")); @@ -251,22 +244,6 @@ pub(crate) fn mask_sensitive_arguments(tokens: &[String]) -> Vec { masked } -fn is_sensitive_flag_name(name: &str) -> bool { - let words = name - .split(['-', '_']) - .map(str::to_ascii_uppercase) - .collect::>(); - SENSITIVE_NAMES.iter().any(|sensitive| { - let sensitive_words = sensitive.split('_').collect::>(); - words.windows(sensitive_words.len()).any(|window| { - window - .iter() - .map(String::as_str) - .eq(sensitive_words.iter().copied()) - }) - }) -} - /// Public-safe context attached to one internal deployment-log diagnostic. #[derive(Clone, Debug)] pub(crate) struct DeploymentLogDiagnosticContext { @@ -433,13 +410,13 @@ pub(crate) fn filter_entry( mut entry: RuntimeLogEntry, protected: &ProtectedLogValues, ) -> RuntimeLogEntry { - let truncated_by_runtime = entry.truncated; + // The trailing part goes first, so a shorter value it contains can't split it. + if entry.truncated { + redact_trailing_partial_value(&mut entry.content, protected); + } for value in protected.values() { entry.content = filter_value(&entry.content, value, &mut entry.truncated); } - if truncated_by_runtime { - redact_trailing_partial_value(&mut entry.content, protected); - } if entry.content.len() > MAX_LOG_ENTRY_CONTENT_BYTES { entry.content.truncate(MAX_LOG_ENTRY_CONTENT_BYTES); entry.truncated = true; @@ -1043,13 +1020,6 @@ mod tests { } } - #[test] - fn environment_names_match_sensitive_names_anywhere_in_the_name() { - assert!(is_sensitive_environment_name("hf_token")); - assert!(is_sensitive_environment_name("MY_API_KEY_FILE")); - assert!(!is_sensitive_environment_name("CUDA_VISIBLE_DEVICES")); - } - #[test] fn unset_page_size_takes_the_default_and_bounded_sizes_pass_through() { for (requested, expected) in [ @@ -1237,6 +1207,41 @@ mod tests { ); } + #[test] + fn a_shorter_protected_value_never_splits_a_longer_one() { + // The shorter value comes first, as a username listed before the password would. + let protected = ProtectedLogValues::new([ + "admin".to_owned(), + "admin-Sup3rPass".to_owned(), + "admin".to_owned(), + ]); + assert_eq!( + protected.values(), + [b"admin-Sup3rPass".to_vec(), b"admin".to_vec()] + ); + + let filtered = filter_entry( + entry(source("web"), 1, b"login admin admin-Sup3rPass"), + &protected, + ); + assert_eq!(filtered.content, b"login [REDACTED] [REDACTED]"); + assert_eq!( + protected.redact("rejected admin-Sup3rPass for admin"), + "rejected [REDACTED] for [REDACTED]" + ); + + let mut cut = entry(source("web"), 2, b"login admin admin-Sup"); + cut.truncated = true; + assert_eq!( + filter_entry(cut, &protected).content, + b"login [REDACTED] [REDACTED]" + ); + + let mut extended = ProtectedLogValues::new(["1".to_owned()]); + extended.extend(ProtectedLogValues::new(["key-1-secret".to_owned()])); + assert_eq!(extended.redact("key-1-secret"), "[REDACTED]"); + } + #[test] fn empty_protected_values_still_bound_entry_content() { let filtered = filter_entry( diff --git a/backend/core/src/model/compatibility.rs b/backend/core/src/model/compatibility.rs index 8dd0122b..94463710 100644 --- a/backend/core/src/model/compatibility.rs +++ b/backend/core/src/model/compatibility.rs @@ -44,7 +44,7 @@ use crate::{ advisory::local_deployment_advisories, engine::{EngineIntegrationRegistry, engine_artifact_kinds, unsigned_parameter_value}, resolution::{ - ModelLaunchResolutionError, apply_profile_defaults, + EngineSecrets, ModelLaunchResolutionError, apply_profile_defaults, resolve_local_launch_on_selected_target, }, selection::{EngineArtifactSelectionError, engine_artifact_kind}, @@ -261,6 +261,7 @@ pub(crate) fn evaluate_profile( target, selected, None, + EngineSecrets::Withhold, ) }); let snapshot = match resolved { diff --git a/backend/core/src/model/controller/logs.rs b/backend/core/src/model/controller/logs.rs index 33c53600..cfe8e4fb 100644 --- a/backend/core/src/model/controller/logs.rs +++ b/backend/core/src/model/controller/logs.rs @@ -6,8 +6,9 @@ //! serves the output of each preparation step it ran for the deployment, then //! reads or tails the runtime logs of the workload launch rebuilt from its //! installed view and prepared artifacts, addressed by the launch's container -//! correlation. The configured upstream API keys and base URLs are also -//! returned so the caller can redact them from returned log lines. +//! correlation. The configured upstream API keys and base URLs, and the +//! secret engine parameter values of local deployments, are also returned so +//! the caller can redact them from returned log lines. use std::{ future::Future, @@ -129,7 +130,10 @@ where .as_ref() .map(|endpoint| endpoint.base_url.clone()), ) - }); + }) + .chain(crate::model::resolved::secret_engine_parameter_values( + deployment, + )); Ok(ProtectedLogValues::new(values)) } } @@ -215,6 +219,28 @@ impl NodeModelLogs { } } + /// The secret engine parameter values of `deployment_id` in the installed view, which its + /// log lines never show. None without a view, since nothing was realized yet, or when the + /// view does not assign it; any other failure to read the view fails the read. + fn protected_values( + &self, + deployment_id: &str, + ) -> Result { + let assigned = match self.input.model_deployments() { + Ok(assigned) => assigned, + // Without an installed view this incarnation has realized nothing, so no line yet + // carries a value. + Err(SiteRepositoryError::NoAcceptedView) => Vec::new(), + Err(error) => return Err(DeploymentLogError::configuration(error)), + }; + Ok(ProtectedLogValues::new( + assigned + .iter() + .filter(|deployment| deployment.model_deployment_id == deployment_id) + .flat_map(crate::model::resolved::secret_engine_parameter_values), + )) + } + /// Whether `deployment_id` is assigned but withheld from the realizer: its engine is not /// pinned on this node yet, or its launch does not resolve. fn is_withheld(&self, deployment_id: &str) -> Result { @@ -244,8 +270,9 @@ impl DeploymentLogOwner for NodeModelLogs { deployment_id: &'a str, query: DeploymentLogReadQuery, ) -> DeploymentLogOwnerFuture<'a, OwnerLogSources> { - // Node model deployments are never upstream, so no launch value is protected, in step - // lines either. + // Node model deployments are never upstream, so the protected values are their secret + // engine parameter values, in step lines too. A read that can't learn them fails rather + // than serve lines unprotected. Box::pin(async move { let selection = self.select(deployment_id, query.source_names()); // Only a selected workload source is resolved, so steps stay readable whatever its @@ -255,6 +282,7 @@ impl DeploymentLogOwner for NodeModelLogs { .as_deref() .map(|names| self.resolve(deployment_id, names)) .transpose()?; + let protected = self.protected_values(deployment_id)?; let mut histories = self .preparation .read(deployment_id, &selection.steps, query.history()) @@ -270,7 +298,7 @@ impl DeploymentLogOwner for NodeModelLogs { }], }); } - Ok(OwnerLogSources::unprotected(histories)) + Ok(OwnerLogSources::protected(histories, protected)) }) } @@ -291,6 +319,7 @@ impl DeploymentLogOwner for NodeModelLogs { .as_deref() .map(|names| self.resolve(deployment_id, names)) .transpose()?; + let protected = self.protected_values(deployment_id)?; let mut streams = self.preparation.follow( deployment_id, &selection.steps, @@ -319,7 +348,7 @@ impl DeploymentLogOwner for NodeModelLogs { }], }); } - Ok(OwnerLogSources::unprotected(streams) + Ok(OwnerLogSources::protected(streams, protected) .with_late(late) .with_changes(changes)) }) @@ -584,7 +613,8 @@ mod tests { dispatch::model::RuntimeTargetMapBuilder, model::controller::test_support::{ TestCurrent, TestDesired, ready_oci_preparer, test_desired_deployment, - test_simulated_runtime_target, test_site, valid_upstream_desired_deployment, + test_simulated_runtime_target, test_site, test_strata_desired_deployment_with_api_key, + valid_upstream_desired_deployment, }, }; @@ -642,6 +672,64 @@ mod tests { assert!(!source_history.candidate_budget_truncated); } + #[tokio::test] + async fn node_model_logs_protect_the_secret_engine_values_of_the_deployment() { + let directory = tempfile::tempdir().expect("preparation log directory"); + let calls = Arc::new(Mutex::new(Vec::new())); + let desired = TestDesired::new(calls.clone()); + desired + .state + .lock() + .expect("desired") + .model_deployments + .push(test_strata_desired_deployment_with_api_key( + "fixture-model", + "node-engine-secret-key", + )); + let mut runtimes = RuntimeTargetMapBuilder::new(); + runtimes + .insert( + "target-one", + LogRuntime { + calls: Arc::new(Mutex::new(Vec::new())), + }, + ) + .expect("log runtime"); + let logs = NodeModelLogs::new( + desired, + runtimes.build().expect("Runtime map"), + ArtifactManager::new("/test-cache".into(), Some(ready_oci_preparer())) + .expect("artifact paths"), + Arc::new(AtomicBool::new(true)), + PreparationLogs::on_system_clock(directory.path()), + crate::deployment_logs::WorkloadLifecycle::new(), + ); + let history = DeploymentLogReadQuery::new( + Vec::new(), + RuntimeLogQuery::new(NonZeroU16::new(2).expect("limit"), None).expect("query"), + ); + + let read = logs + .read_logs("deployment", history) + .await + .expect("history"); + let other = logs + .read_logs( + "other-deployment", + DeploymentLogReadQuery::new( + Vec::new(), + RuntimeLogQuery::new(NonZeroU16::new(2).expect("limit"), None).expect("query"), + ), + ) + .await; + + assert_eq!( + read.protected.values(), + [b"node-engine-secret-key".to_vec()] + ); + assert!(matches!(other, Err(DeploymentLogError::NotFound))); + } + #[tokio::test] async fn node_model_logs_serve_installed_assignments_once_the_realizer_started() { let directory = tempfile::tempdir().expect("preparation log directory"); @@ -1104,7 +1192,10 @@ mod tests { .lock() .expect("desired") .model_deployments - .push(upstream); + .extend([ + upstream, + test_strata_desired_deployment_with_api_key("fixture-model", "local-engine-key"), + ]); let controller = InferenceController::new( desired.clone(), TestCurrent::new(calls), @@ -1128,19 +1219,24 @@ mod tests { .protected_log_values(id) .await .expect("fresh protected values"); + let local = controller + .protected_log_values(ModelDeploymentId("deployment".to_owned())) + .await + .expect("local protected values"); + // Protected values are kept longest first. assert_eq!( first.values(), [ - b"first-secret".to_vec(), - b"https://private.example/v1".to_vec() + b"https://private.example/v1".to_vec(), + b"first-secret".to_vec() ] ); assert_eq!( replacement.values(), [ - b"replacement-secret".to_vec(), - b"https://private.example/v1".to_vec() + b"https://private.example/v1".to_vec(), + b"replacement-secret".to_vec() ] ); assert!( @@ -1149,6 +1245,7 @@ mod tests { .iter() .all(|value| value != b"private-upstream") ); + assert_eq!(local.values(), [b"local-engine-key".to_vec()]); } fn entry_texts(entries: &[RuntimeLogEntry]) -> Vec { diff --git a/backend/core/src/model/controller/reconciliation/mod.rs b/backend/core/src/model/controller/reconciliation/mod.rs index a3cca676..686a17d4 100644 --- a/backend/core/src/model/controller/reconciliation/mod.rs +++ b/backend/core/src/model/controller/reconciliation/mod.rs @@ -1490,6 +1490,9 @@ where artifacts_replaced, "model reconciliation selected runtime workload action", ); + // Runtime errors may quote the launch, so neither failure text nor logs show them + // unredacted. + let protected = protected_startup_values(desired, &launch.spec.environment); match self .transition_workload( desired, @@ -1517,12 +1520,8 @@ where } Ok(None) => {} Err(error) if is_definitive_realization_failure(&error) => { - self.record_realization_failure( - desired, - &error, - &protected_startup_values(&launch.spec.environment), - ) - .await?; + self.record_realization_failure(desired, &error, &protected) + .await?; self.serving_targets_publisher .withdraw_deployment(&ModelDeploymentId( desired.model_deployment_id.clone(), @@ -1531,7 +1530,7 @@ where event_kind = "model_realization_failed", deployment_id = %desired.model_deployment_id, error_kind = "definitive", - reason = %diagnostic_chain(&error), + reason = %protected.redact(&diagnostic_chain(&error)), "model workload realization failed definitively; recorded Failed and stopped retrying", ); continue; @@ -1541,7 +1540,7 @@ where event_kind = "model_realization_retry", deployment_id = %desired.model_deployment_id, error_kind = "transient", - reason = %diagnostic_chain(&error), + reason = %protected.redact(&diagnostic_chain(&error)), "model workload realization failed transiently; keeping in-progress and retrying", ); pending = true; @@ -2203,6 +2202,7 @@ where activation_state: Option<&dotcore_proto::dotcore::model::v1::ModelDeploymentState>, ) -> Result, ModelLifecycleError> { let generation = launch.correlation.reconcile_generation; + let protected = protected_startup_values(desired, &launch.spec.environment); let operation = self.capture_operation( desired, LifecycleOperationAttempt::Runtime { @@ -2258,6 +2258,7 @@ where })? .execution_target_id, launch, + &protected, ) .await?; self.announce_started_workload(desired); @@ -2303,6 +2304,7 @@ where })? .execution_target_id, launch, + &protected, ) .await?; self.announce_started_workload(desired); @@ -2353,6 +2355,7 @@ where &operation, &execution_target_id, &workload.workload_id, + &protected, ) .await?; self.announce_started_workload(desired); @@ -2388,6 +2391,7 @@ where &operation, &execution_target_id, &workload.workload_id, + &protected, ) .await?; self.announce_started_workload(desired); @@ -2764,7 +2768,7 @@ where ) .expect("startup log limit is within runtime bounds"); let query = dotcore_runtime::RuntimeWorkloadLogQuery::new(source, history); - let protected = protected_startup_values(&launch.spec.environment); + let protected = protected_startup_values(desired, &launch.spec.environment); match real_time::bound( Duration::from_secs(2), self.runtimes.read_workload_logs( @@ -3409,21 +3413,27 @@ fn startup_output( } } +/// Values startup output and failure text never show: the deployment's secret engine parameter +/// values and the literal values of sensitive environment variables. fn protected_startup_values( + desired: &dotcore_proto::dotcore::model::v1::ModelDeploymentDeclaration, environment: &BTreeMap, ) -> crate::management_logs::ProtectedLogValues { // Token values are runtime-resolved paths, hosts, and ports, never owner secrets, so each // literal segment of a sensitive value is protected on its own. let values = environment .iter() - .filter(|(key, _)| crate::management_logs::is_sensitive_environment_name(key)) + .filter(|(key, _)| dotcore_catalog::is_sensitive_name(key)) .flat_map(|(_, value)| value.segments()) .filter_map(|segment| match segment { dotcore_runtime::WorkloadValueSegment::Literal(literal) if !literal.is_empty() => { Some(literal.clone()) } _ => None, - }); + }) + .chain(crate::model::resolved::secret_engine_parameter_values( + desired, + )); crate::management_logs::ProtectedLogValues::new(values) } diff --git a/backend/core/src/model/controller/reconciliation/tests/mod.rs b/backend/core/src/model/controller/reconciliation/tests/mod.rs index ba682c0a..39542b86 100644 --- a/backend/core/src/model/controller/reconciliation/tests/mod.rs +++ b/backend/core/src/model/controller/reconciliation/tests/mod.rs @@ -32,8 +32,8 @@ use dotcore_proto::dotcore::{ site::v1::{ModelRuntimeBinding, execution_target, model_runtime_binding}, }; use dotcore_runtime::{ - LoopbackEndpoint, ObservedWorkload, RuntimeBackend, RuntimeBackendId, RuntimeFuture, - WorkloadCorrelation, WorkloadId, WorkloadLaunch, WorkloadObservation, + LoopbackEndpoint, ObservedWorkload, RuntimeBackend, RuntimeBackendError, RuntimeBackendId, + RuntimeFuture, WorkloadCorrelation, WorkloadId, WorkloadLaunch, WorkloadObservation, }; use dotcore_runtime_backends::simulated::{ InMemoryRuntimeBackend, SimulatedRuntimeConfig, SimulatedRuntimeTestSupport, @@ -103,7 +103,11 @@ fn startup_log_protection_selects_sensitive_environment_values() { ), ]); assert_eq!( - super::protected_startup_values(&environment).values(), + super::protected_startup_values( + &test_desired_deployment("fixture-model", ModelDeploymentDesiredState::Present), + &environment + ) + .values(), &[b"sk-test-private".to_vec()] ); } @@ -122,7 +126,11 @@ fn startup_log_protection_selects_literal_parts_of_symbolic_sensitive_values() { ), ]); assert_eq!( - super::protected_startup_values(&environment).values(), + super::protected_startup_values( + &test_desired_deployment("fixture-model", ModelDeploymentDesiredState::Present), + &environment + ) + .values(), &[b"sk-secret@".to_vec()] ); } @@ -559,7 +567,7 @@ async fn protected_launch_values_never_leave_the_node_in_a_realization_failure() WorkloadValue::literal("fixture-model".to_owned()), ), ]); - let protected: ProtectedLogValues = super::protected_startup_values(&environment); + let protected: ProtectedLogValues = super::protected_startup_values(&deployment, &environment); let error = ModelLifecycleError::configuration(format!( "engine rejected HF_TOKEN={SECRET} for fixture-model" )); @@ -2976,6 +2984,320 @@ async fn loading_stopped_workload_records_startup_failure_without_restart() { assert!(resumed_state.failure.is_none()); } +#[tokio::test] +async fn a_secret_engine_value_reaches_no_log_failure_text_telemetry_or_command() { + use prost::Message as _; + + const SENTINEL: &str = "sentinel-api-key-6f1c2b9d"; + let contains_sentinel = |bytes: &[u8]| { + bytes + .windows(SENTINEL.len()) + .any(|window| window == SENTINEL.as_bytes()) + }; + let calls = Arc::new(Mutex::new(Vec::new())); + let desired = TestDesired::new(calls.clone()); + let deployment = + crate::model::controller::test_support::test_strata_desired_deployment_with_api_key( + "model", SENTINEL, + ); + let identity = crate::model::controller::test_support::expected_identity( + &test_site(), + &deployment, + dotcore_runtime_backends::simulated::BACKEND_ID, + ); + desired + .state + .lock() + .expect("desired state") + .model_deployments + .push(deployment.clone()); + let current = TestCurrent::new(calls.clone()); + let mut starting = current_deployment("deployment"); + starting.state = Some(crate::model::controller::state::realization_prepared_state( + None, + )); + current.upsert(starting, None).expect("current state"); + // The engine echoes its key on its way out, as a misbehaving engine could. + let startup_logs = Arc::new(Mutex::new(StartupLogBehavior::History { + expected_workload_id: identity.workload_id.clone(), + history: dotcore_runtime::RuntimeLogHistory { + entries: vec![dotcore_runtime::RuntimeLogEntry { + source: dotcore_runtime::RuntimeLogSource { + name: "model".to_owned(), + label: "Model".to_owned(), + }, + timestamp: std::time::SystemTime::UNIX_EPOCH, + stream: dotcore_runtime::RuntimeLogStreamKind::Stderr, + content: format!("refusing api key {SENTINEL}").into_bytes(), + truncated: false, + }], + has_older_output: false, + candidate_budget_truncated: false, + }, + })); + let mut builder = RuntimeTargetMapBuilder::new(); + builder + .insert( + "target-one", + PausedRuntime { + calls: calls.clone(), + observations: Arc::new(Mutex::new(vec![vec![WorkloadObservation::Stopped { + workload: identity.clone(), + exit_status: Some(17), + }]])), + start_observation: Some(WorkloadObservation::Running(identity.clone())), + startup_logs, + backend_id: RuntimeBackendId( + dotcore_runtime_backends::simulated::BACKEND_ID.to_owned(), + ), + }, + ) + .expect("Runtime adapter"); + let artifacts = ArtifactManager::from_parts( + dotcore_artifact::ModelArtifactCache::new("/test-cache".into()).expect("test cache"), + Some(crate::model::controller::test_support::ready_oci_preparer()), + ); + let controller = sweep_controller( + &desired, + ¤t, + builder.build().expect("Runtime map"), + artifacts.clone(), + ); + + let (reconciliation, records, output) = + capture_records_async(|| controller.inner.reconcile()).await; + reconciliation.expect("startup reconciliation"); + + // Failure text: the recorded failure shows the output with the key redacted. + let state = current.snapshot().expect("current state").model_deployments[0] + .state + .clone() + .expect("deployment state"); + assert_eq!(state.status, ModelDeploymentStatus::Failed as i32); + assert_eq!( + state.failure.as_ref().expect("startup failure").diagnostics, + [ + "Stage: engine startup", + "Exit status: 17", + "Output tail: refusing api key [REDACTED]", + ] + ); + assert!( + !format!("{:?}", current.snapshot().expect("current state")).contains(SENTINEL), + "current state" + ); + // Log lines: every event and span of the reconciliation, at every level. + assert!(!records.is_empty()); + assert!(!output.contains(SENTINEL), "{output}"); + assert!(!format!("{records:?}").contains(SENTINEL), "{records:#?}"); + // Telemetry record inputs and the launch command, plan and materialized workload alike. + let resolved = resolved_launch(&deployment).expect("resolved launch"); + assert!(!contains_sentinel( + &resolved.telemetry_inputs().encode_to_vec() + )); + assert!(!contains_sentinel(&resolved.plan.encode_to_vec())); + let launch = workload_launch( + &deployment, + selected_target(&test_site(), &deployment).expect("execution target"), + &artifacts + .expected_model_paths(&resolved.artifacts) + .expect("artifact paths"), + ) + .expect("workload launch"); + for value in launch + .spec + .arguments + .iter() + .chain(launch.spec.environment.values()) + { + assert!(!format!("{value:?}").contains(SENTINEL)); + } + + // Runtime errors quoting the key, transient then definitive, are logged and recorded + // redacted. + let quoting_errors = Arc::new(Mutex::new(vec![ + RuntimeBackendError::ImagePlatformMismatch { + image: format!("strata --api-key {SENTINEL}"), + required: dotcore_domain::ExecutionPlatform::linux_amd64(), + observed: dotcore_domain::ExecutionPlatform::linux_arm64(), + }, + RuntimeBackendError::InvalidLaunch(anyhow!("engine refused api key {SENTINEL}")), + ])); + let current = TestCurrent::new(calls.clone()); + let mut prepared = current_deployment("deployment"); + prepared.state = Some(crate::model::controller::state::realization_prepared_state( + None, + )); + current.upsert(prepared, None).expect("current state"); + let mut builder = RuntimeTargetMapBuilder::new(); + builder + .insert( + "target-one", + QuotingFailureRuntime { + errors: quoting_errors.clone(), + }, + ) + .expect("Runtime adapter"); + let controller = sweep_controller( + &desired, + ¤t, + builder.build().expect("Runtime map"), + artifacts, + ); + + let (transient, transient_records, transient_output) = + capture_records_async(|| controller.inner.reconcile()).await; + transient.expect("transient failure reconciliation"); + let (definitive, definitive_records, definitive_output) = + capture_records_async(|| controller.inner.reconcile()).await; + definitive.expect("definitive failure reconciliation"); + + assert!(quoting_errors.lock().expect("errors").is_empty()); + for (event_kind, records, output) in [ + ( + "\"model_realization_retry\"", + &transient_records, + &transient_output, + ), + ( + "\"model_realization_failed\"", + &definitive_records, + &definitive_output, + ), + ] { + let reason = records + .iter() + .find_map(|record| match record { + Record::Event { fields, .. } + if fields.get("event_kind").map(String::as_str) == Some(event_kind) => + { + fields.get("reason").cloned() + } + _ => None, + }) + .unwrap_or_else(|| panic!("{event_kind} event: {records:#?}")); + assert!(reason.contains("[REDACTED]"), "{reason}"); + assert!(!output.contains(SENTINEL), "{output}"); + assert!(!format!("{records:?}").contains(SENTINEL), "{records:#?}"); + } + let failure = current.snapshot().expect("current state").model_deployments[0] + .state + .clone() + .expect("deployment state") + .failure + .expect("definitive failure"); + assert!( + failure.message.contains("[REDACTED]"), + "{}", + failure.message + ); + assert!(!format!("{failure:?}").contains(SENTINEL), "{failure:?}"); +} + +/// The node realizer of the sentinel sweep over `desired` and `current`, on `runtimes`. +fn sweep_controller( + desired: &TestDesired, + current: &TestCurrent, + runtimes: crate::dispatch::model::RuntimeTargetMap, + artifacts: ArtifactManager, +) -> super::super::test_support::TestRealizer { + let (serving_targets_publisher, serving_targets_source) = + crate::model::controller::serving_targets::ServingTargetsPublisher::new(); + let inner = InferenceControllerInner { + workload_lifecycle: crate::deployment_logs::WorkloadLifecycle::new(), + preparation_logs: crate::deployment_logs::PreparationLogs::on_system_clock( + std::path::Path::new("/test-preparation-logs"), + ), + results: desired.clone(), + target_capabilities: crate::model::controller::test_support::test_target_capability_source( + ), + desired: desired.clone(), + state: current.clone(), + runtimes, + engine_integrations: crate::model::controller::test_support::test_engine_integrations(), + operation: Arc::new(AsyncMutex::new(())), + reconciliation: AsyncMutex::new(()), + preparations: PreparationCoordinator::from_deployments(&[]) + .expect("preparation coordinator"), + observer_started: Mutex::new(false), + wakeup: Arc::new(Notify::new()), + artifacts, + serving_targets_publisher, + serving_targets_source, + accelerator_attribution: None, + shutdown: tokio_util::sync::CancellationToken::new(), + timing: Timing::test().model, + clock: Arc::new(SystemClock), + }; + manager_test_controller( + desired.clone(), + super::super::ModelRealizer { + inner: Arc::new(inner), + }, + ) +} + +/// A runtime with no workloads whose creations fail with the queued errors, last first. +struct QuotingFailureRuntime { + errors: Arc>>, +} + +impl RuntimeBackend for QuotingFailureRuntime { + fn backend_id(&self) -> RuntimeBackendId { + RuntimeBackendId(dotcore_runtime_backends::simulated::BACKEND_ID.to_owned()) + } + + fn materialization_requirements(&self) -> dotcore_runtime::WorkloadMaterializationRequirements { + dotcore_runtime::WorkloadMaterializationRequirements::virtualized() + } + + fn discover_workloads<'a>(&'a self) -> RuntimeFuture<'a, Vec> { + Box::pin(ready(Ok(Vec::new()))) + } + + fn create_workload<'a>( + &'a self, + _workload: &'a WorkloadLaunch, + ) -> RuntimeFuture<'a, WorkloadObservation> { + let error = self + .errors + .lock() + .expect("errors") + .pop() + .unwrap_or_else(test_runtime_error); + Box::pin(ready(Err(error))) + } + + fn start_workload<'a>( + &'a self, + _workload_id: &'a WorkloadId, + ) -> RuntimeFuture<'a, WorkloadObservation> { + Box::pin(ready(Err(test_runtime_error()))) + } + + fn inspect_workload<'a>( + &'a self, + _workload_id: &'a WorkloadId, + ) -> RuntimeFuture<'a, WorkloadObservation> { + Box::pin(ready(Err(test_runtime_error()))) + } + + fn stop_workload<'a>(&'a self, _workload_id: &'a WorkloadId) -> RuntimeFuture<'a, ()> { + Box::pin(ready(Ok(()))) + } + + fn remove_workload<'a>(&'a self, _workload_id: &'a WorkloadId) -> RuntimeFuture<'a, ()> { + Box::pin(ready(Ok(()))) + } + + fn remove_storage<'a>( + &'a self, + _: &'a dotcore_runtime::WorkloadStorageKey, + ) -> RuntimeFuture<'a, ()> { + Box::pin(ready(Ok(()))) + } +} + #[tokio::test(start_paused = true)] async fn simulated_download_precedes_runtime_creation_clears_progress_and_skips_oci() { let calls = Arc::new(Mutex::new(Vec::new())); @@ -3136,7 +3458,7 @@ async fn simulated_start_without_a_download_passes_the_download_and_engine_steps #[tokio::test] async fn teardown_deletes_the_deployment_storage_that_pause_keeps_and_leaves_shared_storage() { - use dotcore_proto::dotcore::catalog::v1::{StorageBinding, StorageKind, StorageScope}; + use dotcore_proto::dotcore::catalog::v1::{StorageBinding, StorageKind, StoragePlacement}; use dotcore_runtime::{WorkloadStorageKey, WorkloadStorageKind}; let calls = Arc::new(Mutex::new(Vec::new())); @@ -3157,14 +3479,14 @@ async fn teardown_deletes_the_deployment_storage_that_pause_keeps_and_leaves_sha .as_mut() .expect("engine artifact") .storage_bindings = [ - ("data", "/data", StorageScope::Deployment), - ("compile-cache", "/cache", StorageScope::EngineArtifact), + ("data", "/data", StoragePlacement::Deployment), + ("compile-cache", "/cache", StoragePlacement::EngineArtifact), ] - .map(|(binding_name, mount_path, scope)| StorageBinding { + .map(|(binding_name, mount_path, placement)| StorageBinding { binding_name: binding_name.to_owned(), mount_path: mount_path.to_owned(), kind: StorageKind::Cache as i32, - scope: scope as i32, + placements: vec![placement as i32], environment: Vec::new(), }) .to_vec(); @@ -3252,7 +3574,7 @@ async fn teardown_deletes_the_deployment_storage_that_pause_keeps_and_leaves_sha #[tokio::test] async fn teardown_after_a_relaunch_that_failed_before_its_workload_deletes_only_its_own_storage() { - use dotcore_proto::dotcore::catalog::v1::{StorageBinding, StorageKind, StorageScope}; + use dotcore_proto::dotcore::catalog::v1::{StorageBinding, StorageKind, StoragePlacement}; use dotcore_runtime::{WorkloadStorageKey, WorkloadStorageKind}; let calls = Arc::new(Mutex::new(Vec::new())); @@ -3276,7 +3598,7 @@ async fn teardown_after_a_relaunch_that_failed_before_its_workload_deletes_only_ binding_name: "data".to_owned(), mount_path: "/data".to_owned(), kind: StorageKind::Cache as i32, - scope: StorageScope::Deployment as i32, + placements: vec![StoragePlacement::Deployment as i32], environment: Vec::new(), }]; desired.persist(deployment); diff --git a/backend/core/src/model/controller/test_support.rs b/backend/core/src/model/controller/test_support.rs index c3a8dab5..12280cb9 100644 --- a/backend/core/src/model/controller/test_support.rs +++ b/backend/core/src/model/controller/test_support.rs @@ -1061,6 +1061,20 @@ pub(super) fn test_strata_desired_deployment(model_id: &str) -> ModelDeploymentD } } +/// Returns a present deployment of the in-code Strata fixture whose engine declares a secret API +/// key, retained as `api_key`. +pub(super) fn test_strata_desired_deployment_with_api_key( + model_id: &str, + api_key: &str, +) -> ModelDeploymentDeclaration { + ModelDeploymentDeclaration { + snapshot: Some( + crate::test_support::strata::local_model_snapshot_with_api_key(model_id, api_key), + ), + ..test_desired_deployment(model_id, ModelDeploymentDesiredState::Present) + } +} + pub(super) fn valid_upstream_desired_deployment() -> ModelDeploymentDeclaration { ModelDeploymentDeclaration { model_deployment_id: "upstream-deployment".to_owned(), diff --git a/backend/core/src/model/engine/declaration.rs b/backend/core/src/model/engine/declaration.rs index d3663081..10d53e60 100644 --- a/backend/core/src/model/engine/declaration.rs +++ b/backend/core/src/model/engine/declaration.rs @@ -6,8 +6,8 @@ use dotcore_catalog::{ }; use dotcore_domain::Accelerator; use dotcore_proto::dotcore::catalog::v1::{ - EngineParameterValues, EngineRelease, HardwareClass, InferenceRequestKind, Model, - ModelEngineConfiguration, model_option, + EngineRelease, HardwareClass, InferenceRequestKind, Model, ModelEngineConfiguration, + ParameterValues, model_option, }; use super::definition::{definition, unsigned_parameter_maximum}; @@ -67,7 +67,7 @@ impl EngineDeclarationValidator for EngineFacade { model: &Model, configuration: &ModelEngineConfiguration, engine: &EngineRelease, - ) -> EngineParameterValues { + ) -> ParameterValues { definition(engine.engine_kind) .map(|definition| definition.parameter_defaults(model, configuration)) .unwrap_or_default() @@ -185,9 +185,9 @@ mod tests { }; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterDefinition, EngineParameterSchema, EngineRelease, InferenceRequestKind, - Model, ModelEngineConfiguration, UnsignedIntegerEngineParameterSchema, - engine_parameter_definition::ParameterType, + EngineParameterSchema, EngineRelease, InferenceRequestKind, Model, + ModelEngineConfiguration, ParameterDefinition, UnsignedIntegerParameterSchema, + parameter_definition::ParameterType, }, engine::v1::{EngineKind, EngineModelConfiguration}, }; @@ -202,10 +202,10 @@ mod tests { }; let release = EngineRelease { parameter_schema: Some(EngineParameterSchema { - parameters: vec![EngineParameterDefinition { + parameters: vec![ParameterDefinition { parameter_id: CONTEXT_PER_REQUEST_PARAMETER_ID.to_owned(), parameter_type: Some(ParameterType::UnsignedInteger( - UnsignedIntegerEngineParameterSchema { + UnsignedIntegerParameterSchema { minimum: 1024, maximum: 131_072, unit: "tokens".to_owned(), diff --git a/backend/core/src/model/engine/definition.rs b/backend/core/src/model/engine/definition.rs index 823f4028..17416e75 100644 --- a/backend/core/src/model/engine/definition.rs +++ b/backend/core/src/model/engine/definition.rs @@ -13,9 +13,9 @@ use dotcore_catalog::shared_parameters; use dotcore_domain::WorkloadArtifactKind; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterSchema, EngineParameterValue, EngineParameterValues, EngineRelease, - HardwareClass, InferenceRequestKind, Model, ModelArtifactBinding, ModelEngineConfiguration, - ModelOptionValues, engine_parameter_definition, engine_parameter_value, + EngineParameterSchema, EngineRelease, HardwareClass, InferenceRequestKind, Model, + ModelArtifactBinding, ModelEngineConfiguration, ModelOptionValues, ParameterValue, + ParameterValues, parameter_definition, parameter_value, }, engine::v1::{EngineKind, ResolvedEngineModelState}, model::v1::{PortableModelLaunchPlan, ResolvedLocalModelDeploymentSnapshot}, @@ -74,8 +74,8 @@ pub(crate) trait EngineDefinition: Send + Sync { &self, _model: &Model, _configuration: &ModelEngineConfiguration, - ) -> EngineParameterValues { - EngineParameterValues::default() + ) -> ParameterValues { + ParameterValues::default() } /// Context per request for values that leave it out, which counts as derived. By default it is @@ -84,9 +84,9 @@ pub(crate) trait EngineDefinition: Send + Sync { fn filled_context_per_request( &self, schema: &EngineParameterSchema, - values: &EngineParameterValues, + values: &ParameterValues, maximum_context_tokens: u32, - ) -> Option { + ) -> Option { shared_parameters::filled_context_per_request(schema, values, maximum_context_tokens) } @@ -97,7 +97,7 @@ pub(crate) trait EngineDefinition: Send + Sync { &self, _model: &Model, _configuration: &ModelEngineConfiguration, - _values: &EngineParameterValues, + _values: &ParameterValues, ) -> anyhow::Result<()> { Ok(()) } @@ -195,7 +195,7 @@ pub(crate) fn definition(kind: i32) -> Option<&'static dyn EngineDefinition> { /// Returns the unsigned value of one engine parameter, or none when it is absent or not unsigned. pub(crate) fn unsigned_parameter_value( - values: &EngineParameterValues, + values: &ParameterValues, parameter_id: &str, ) -> Option { values @@ -203,7 +203,7 @@ pub(crate) fn unsigned_parameter_value( .iter() .find(|value| value.parameter_id == parameter_id) .and_then(|value| match value.value { - Some(engine_parameter_value::Value::UnsignedInteger(value)) => Some(value), + Some(parameter_value::Value::UnsignedInteger(value)) => Some(value), _ => None, }) } @@ -229,7 +229,7 @@ pub(crate) fn unsigned_parameter_range( .iter() .find(|definition| definition.parameter_id == parameter_id) .and_then(|definition| match definition.parameter_type.as_ref() { - Some(engine_parameter_definition::ParameterType::UnsignedInteger(range)) => { + Some(parameter_definition::ParameterType::UnsignedInteger(range)) => { Some(range.minimum..=range.maximum) } _ => None, diff --git a/backend/core/src/model/engine/llama_cpp/declaration.rs b/backend/core/src/model/engine/llama_cpp/declaration.rs index 8cfae5a8..5c3e1a87 100644 --- a/backend/core/src/model/engine/llama_cpp/declaration.rs +++ b/backend/core/src/model/engine/llama_cpp/declaration.rs @@ -12,13 +12,12 @@ use dotcore_catalog::{ use dotcore_domain::Accelerator; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterDefinition, EngineParameterValue, EngineParameterValues, EngineRelease, - HardwareClass, Model, ModelConfigurationOption, ModelEngineConfiguration, - ModelInputModality, SpeculativeDecodingMethod, engine_parameter_definition::ParameterType, - engine_parameter_value::Value as EngineParameterValueKind, - hugging_face_artifact_source::Content as HuggingFaceContent, + EngineRelease, HardwareClass, Model, ModelConfigurationOption, ModelEngineConfiguration, + ModelInputModality, ParameterDefinition, ParameterValue, ParameterValues, + SpeculativeDecodingMethod, hugging_face_artifact_source::Content as HuggingFaceContent, model_option::Kind as ModelOptionKind, model_option_parameter_definition::ParameterType as ModelOptionParameterType, + parameter_definition::ParameterType, parameter_value::Value as EngineParameterValueKind, }, engine::{ llama_cpp::v1::{ @@ -130,12 +129,12 @@ pub(crate) fn validate_hardware_support( pub(crate) fn parameter_defaults( model: &Model, configuration: &ModelEngineConfiguration, -) -> EngineParameterValues { +) -> ParameterValues { if !uses_single_micro_batch(configuration) { - return EngineParameterValues::default(); + return ParameterValues::default(); } - EngineParameterValues { - values: vec![EngineParameterValue { + ParameterValues { + values: vec![ParameterValue { parameter_id: TOTAL_CONTEXT_PARAMETER_ID.to_owned(), value: Some(EngineParameterValueKind::UnsignedInteger(u64::from( maximum_context_tokens(model, configuration), @@ -176,7 +175,7 @@ pub(crate) fn validate_total_context_default( pub(crate) fn validate_parameter_values( model: &Model, configuration: &ModelEngineConfiguration, - values: &EngineParameterValues, + values: &ParameterValues, ) -> anyhow::Result<()> { if !uses_single_micro_batch(configuration) { return Ok(()); @@ -661,7 +660,7 @@ pub(crate) enum LlamaCppSamplingError { /// Accepts a shared parameter bounded by llama.cpp's own limit. The bound on the total context is /// high enough that a large pool exceeds memory before it exceeds the bound. -fn is_bounded(definition: &EngineParameterDefinition, parameter_id: &str, maximum: u64) -> bool { +fn is_bounded(definition: &ParameterDefinition, parameter_id: &str, maximum: u64) -> bool { definition.parameter_id == parameter_id && matches!( definition.parameter_type.as_ref(), @@ -669,7 +668,7 @@ fn is_bounded(definition: &EngineParameterDefinition, parameter_id: &str, maximu ) } -fn is_prompt_cache_memory(definition: &EngineParameterDefinition) -> bool { +fn is_prompt_cache_memory(definition: &ParameterDefinition) -> bool { definition.parameter_id == PROMPT_CACHE_MEMORY_PARAMETER_ID && definition.display_name == "Prompt cache memory" && matches!( @@ -686,12 +685,12 @@ mod tests { use dotcore_catalog::{Catalog, OciArtifactPolicy, test_support::vendor_hardware_class}; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterDefinition, HuggingFaceArtifactSource, HuggingFaceFileArtifact, - HuggingFaceSnapshotArtifact, HuggingFaceSnapshotFile, Model, ModelArtifact, - ModelArtifactBinding, ModelAuxiliaryArtifact, ModelConfigurationOption, - ModelEngineConfiguration, SpeculativeDecodingMethod, SpeculativeDecodingModelOption, - VisionModelOption, engine_parameter_definition::ParameterType, - hugging_face_artifact_source::Content as HuggingFaceContent, model_option, + HuggingFaceArtifactSource, HuggingFaceFileArtifact, HuggingFaceSnapshotArtifact, + HuggingFaceSnapshotFile, Model, ModelArtifact, ModelArtifactBinding, + ModelAuxiliaryArtifact, ModelConfigurationOption, ModelEngineConfiguration, + ParameterDefinition, SpeculativeDecodingMethod, SpeculativeDecodingModelOption, + VisionModelOption, hugging_face_artifact_source::Content as HuggingFaceContent, + model_option, parameter_definition::ParameterType, }, engine::{ llama_cpp::v1::{ @@ -837,7 +836,7 @@ mod tests { .find(|release| release.engine_kind == EngineKind::LlamaCpp as i32) .expect("llama.cpp release"); assert!(validate_engine_release(&release).is_empty()); - let rejection = |edit: &dyn Fn(&mut Vec)| { + let rejection = |edit: &dyn Fn(&mut Vec)| { let mut release = release.clone(); edit( &mut release @@ -851,7 +850,7 @@ mod tests { .map(ToString::to_string) .collect::>() }; - let maximum = |parameters: &mut Vec, index: usize, value| { + let maximum = |parameters: &mut Vec, index: usize, value| { if let Some(ParameterType::UnsignedInteger(range)) = parameters[index].parameter_type.as_mut() { @@ -1252,9 +1251,7 @@ mod tests { #[test] fn a_single_micro_batch_maximum_context_must_be_a_valid_total_context() { use dotcore_proto::dotcore::{ - catalog::v1::{ - EngineParameterSchema, EngineRelease, UnsignedIntegerEngineParameterSchema, - }, + catalog::v1::{EngineParameterSchema, EngineRelease, UnsignedIntegerParameterSchema}, engine::{ llama_cpp::v1::{LlamaCppFixedModelSettings, LlamaCppModelConfiguration}, v1::EngineModelConfiguration, @@ -1267,10 +1264,10 @@ mod tests { }; let engine = EngineRelease { parameter_schema: Some(EngineParameterSchema { - parameters: vec![EngineParameterDefinition { + parameters: vec![ParameterDefinition { parameter_id: "total-context".to_owned(), parameter_type: Some(ParameterType::UnsignedInteger( - UnsignedIntegerEngineParameterSchema { + UnsignedIntegerParameterSchema { minimum: 1024, maximum: 8192, ..Default::default() diff --git a/backend/core/src/model/engine/llama_cpp/mod.rs b/backend/core/src/model/engine/llama_cpp/mod.rs index b8d1272f..ae414be6 100644 --- a/backend/core/src/model/engine/llama_cpp/mod.rs +++ b/backend/core/src/model/engine/llama_cpp/mod.rs @@ -17,8 +17,8 @@ use dotcore_artifact::PreparedModelArtifacts; use dotcore_domain::WorkloadArtifactKind; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterValues, EngineRelease, HardwareClass, InferenceRequestKind, Model, - ModelArtifactBinding, ModelEngineConfiguration, ModelOptionValues, + EngineRelease, HardwareClass, InferenceRequestKind, Model, ModelArtifactBinding, + ModelEngineConfiguration, ModelOptionValues, ParameterValues, }, engine::v1::{EngineKind, ResolvedEngineModelState}, model::v1::{PortableModelLaunchPlan, ResolvedLocalModelDeploymentSnapshot}, @@ -82,7 +82,7 @@ impl EngineDefinition for LlamaCpp { &self, model: &Model, configuration: &ModelEngineConfiguration, - ) -> EngineParameterValues { + ) -> ParameterValues { declaration::parameter_defaults(model, configuration) } @@ -90,7 +90,7 @@ impl EngineDefinition for LlamaCpp { &self, model: &Model, configuration: &ModelEngineConfiguration, - values: &EngineParameterValues, + values: &ParameterValues, ) -> anyhow::Result<()> { declaration::validate_parameter_values(model, configuration, values) } diff --git a/backend/core/src/model/engine/llama_cpp/plan.rs b/backend/core/src/model/engine/llama_cpp/plan.rs index 213137f1..9bcbc7c6 100644 --- a/backend/core/src/model/engine/llama_cpp/plan.rs +++ b/backend/core/src/model/engine/llama_cpp/plan.rs @@ -14,7 +14,7 @@ use dotcore_catalog::{ }; use dotcore_proto::{ dotcore::{ - catalog::v1::{ModelOptionValues, engine_parameter_value::Value as EngineParameterValue}, + catalog::v1::{ModelOptionValues, parameter_value::Value as EngineParameterValueKind}, engine::{ llama_cpp::v1::{ LlamaCppFixedModelSettings, LlamaCppResolvedModelState, LlamaCppSpeculativeMethod, @@ -431,7 +431,7 @@ fn unsigned_parameter( .find(|value| value.parameter_id == parameter_id) }) .and_then(|value| match value.value.as_ref() { - Some(EngineParameterValue::UnsignedInteger(value)) => Some(*value), + Some(EngineParameterValueKind::UnsignedInteger(value)) => Some(*value), _ => None, }) } @@ -628,7 +628,7 @@ pub(crate) enum LlamaCppPlanError { #[cfg(test)] mod tests { use dotcore_proto::dotcore::catalog::v1::{ - EngineParameterDefinition, EngineParameterValue as EngineParameterValueMessage, + ParameterDefinition, ParameterValue as EngineParameterValueMessage, ParameterVisibility, }; use super::*; @@ -910,7 +910,7 @@ mod tests { .retain(|value| value.parameter_id != PROMPT_CACHE_MEMORY_PARAMETER_ID); values.values.push(EngineParameterValueMessage { parameter_id: PROMPT_CACHE_MEMORY_PARAMETER_ID.to_owned(), - value: Some(EngineParameterValue::UnsignedInteger(mib)), + value: Some(EngineParameterValueKind::UnsignedInteger(mib)), }); assert_eq!( @@ -951,11 +951,11 @@ mod tests { .as_mut() .expect("engine parameter schema") .parameters - .push(EngineParameterDefinition { + .push(ParameterDefinition { parameter_id: PROMPT_CACHE_MEMORY_PARAMETER_ID.to_owned(), display_name: "Prompt cache memory".to_owned(), parameter_type: None, - exposed: true, + visibility: ParameterVisibility::Advanced as i32, description: String::new(), }); for token in tokens { @@ -1753,7 +1753,7 @@ mod tests { .retain(|value| value.parameter_id != parameter_id); values.values.push(EngineParameterValueMessage { parameter_id: parameter_id.to_owned(), - value: Some(EngineParameterValue::UnsignedInteger(value)), + value: Some(EngineParameterValueKind::UnsignedInteger(value)), }); } diff --git a/backend/core/src/model/engine/mod.rs b/backend/core/src/model/engine/mod.rs index c7cac54a..a13a6143 100644 --- a/backend/core/src/model/engine/mod.rs +++ b/backend/core/src/model/engine/mod.rs @@ -54,9 +54,9 @@ pub(crate) fn engine_artifact_kinds(kind: i32) -> &'static [dotcore_domain::Work pub(crate) fn filled_context_per_request( kind: i32, schema: &dotcore_proto::dotcore::catalog::v1::EngineParameterSchema, - values: &dotcore_proto::dotcore::catalog::v1::EngineParameterValues, + values: &dotcore_proto::dotcore::catalog::v1::ParameterValues, maximum_context_tokens: u32, -) -> Option { +) -> Option { match definition::definition(kind) { Some(engine) => engine.filled_context_per_request(schema, values, maximum_context_tokens), None => dotcore_catalog::shared_parameters::filled_context_per_request( @@ -74,7 +74,7 @@ pub(crate) fn validate_parameter_values( kind: i32, model: &dotcore_proto::dotcore::catalog::v1::Model, configuration: &dotcore_proto::dotcore::catalog::v1::ModelEngineConfiguration, - values: &dotcore_proto::dotcore::catalog::v1::EngineParameterValues, + values: &dotcore_proto::dotcore::catalog::v1::ParameterValues, ) -> anyhow::Result<()> { validate_context_within_total(values)?; definition::definition(kind).map_or(Ok(()), |engine| { @@ -85,7 +85,7 @@ pub(crate) fn validate_parameter_values( /// Rejects a context per request above the total context every request shares, since no request /// could use it. fn validate_context_within_total( - values: &dotcore_proto::dotcore::catalog::v1::EngineParameterValues, + values: &dotcore_proto::dotcore::catalog::v1::ParameterValues, ) -> anyhow::Result<()> { use dotcore_catalog::shared_parameters::{ CONTEXT_PER_REQUEST_PARAMETER_ID, TOTAL_CONTEXT_PARAMETER_ID, diff --git a/backend/core/src/model/engine/ninfer/declaration.rs b/backend/core/src/model/engine/ninfer/declaration.rs index 797e4923..c2030baa 100644 --- a/backend/core/src/model/engine/ninfer/declaration.rs +++ b/backend/core/src/model/engine/ninfer/declaration.rs @@ -19,12 +19,12 @@ use dotcore_catalog::{ use dotcore_domain::Accelerator; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterValues, EngineRelease, HardwareClass, Model, ModelConfigurationOption, - ModelEngineConfiguration, ModelInputModality, ModelParallelShape, - SpeculativeDecodingMethod, engine_parameter_definition::ParameterType, + EngineRelease, HardwareClass, Model, ModelConfigurationOption, ModelEngineConfiguration, + ModelInputModality, ModelParallelShape, ParameterValues, SpeculativeDecodingMethod, hugging_face_artifact_source::Content as HuggingFaceContent, model_option::Kind as ModelOptionKind, model_option_parameter_definition::ParameterType as ModelOptionParameterType, + parameter_definition::ParameterType, }, engine::{ ninfer::v1::{ @@ -52,7 +52,7 @@ const MODEL_FILE_EXTENSION: &str = ".ninfer"; /// Rejects a total context ninfer refuses at startup: above what the parallel requests use at /// their full context per request, so part of it could never hold a request. -pub(crate) fn validate_parameter_values(values: &EngineParameterValues) -> anyhow::Result<()> { +pub(crate) fn validate_parameter_values(values: &ParameterValues) -> anyhow::Result<()> { let (Some(total_context), Some(context_per_request), Some(parallel_requests)) = ( unsigned_parameter_value(values, TOTAL_CONTEXT_PARAMETER_ID), unsigned_parameter_value(values, CONTEXT_PER_REQUEST_PARAMETER_ID), @@ -460,9 +460,9 @@ mod tests { use dotcore_domain::ExecutionPlatform; use dotcore_proto::dotcore::{ catalog::v1::{ - BooleanEngineParameterSchema, EngineParameterDefinition, HuggingFaceArtifactSource, - HuggingFaceFileArtifact, HuggingFaceSnapshotArtifact, SpeculativeDecodingModelOption, - UnsignedIntegerEngineParameterSchema, + BooleanParameterSchema, HuggingFaceArtifactSource, HuggingFaceFileArtifact, + HuggingFaceSnapshotArtifact, ParameterDefinition, SpeculativeDecodingModelOption, + UnsignedIntegerParameterSchema, }, model::v1::{ LocalModelDeploymentConfiguration, ResolvedLocalModelDeploymentSnapshot, @@ -506,7 +506,7 @@ mod tests { .as_mut() .expect("schema") .parameters[3] - .parameter_type = Some(ParameterType::Boolean(BooleanEngineParameterSchema {})); + .parameter_type = Some(ParameterType::Boolean(BooleanParameterSchema::default())); assert_eq!( messages(validate_engine_release(&boolean)), ["ninfer engine release requires non-empty unsigned-integer parameter definitions"] @@ -578,7 +578,7 @@ mod tests { .as_mut() .expect("schema") .parameters - .push(EngineParameterDefinition { + .push(ParameterDefinition { parameter_id: "prefill-chunk".to_owned(), parameter_type: Some(unsigned(128, 8192, "tokens")), ..Default::default() @@ -996,10 +996,10 @@ version: "1" parameterSchema: schemaId: "fixture-ninfer-v2" parameters: - - { parameterId: "total-context", displayName: "Total context", exposed: true, description: "total-context", unsignedInteger: { minimum: "1024", maximum: "2097152", unit: "tokens", defaultValue: "32768" } } - - { parameterId: "context-per-request", displayName: "Context per request", exposed: true, description: "context-per-request", unsignedInteger: { minimum: "1024", maximum: "262144", unit: "tokens" } } - - { parameterId: "parallel-requests", displayName: "Parallel requests", exposed: true, description: "parallel-requests", unsignedInteger: { minimum: "1", maximum: "8", unit: "requests", defaultValue: "4" } } - - { parameterId: "host-context-mib", displayName: "Host context memory", exposed: true, description: "host-context-mib", unsignedInteger: { minimum: "0", maximum: "1048576", unit: "MiB" } } + - { parameterId: "total-context", displayName: "Total context", visibility: "PARAMETER_VISIBILITY_BASIC", description: "total-context", unsignedInteger: { minimum: "1024", maximum: "2097152", unit: "tokens", defaultValue: "32768" } } + - { parameterId: "context-per-request", displayName: "Context per request", visibility: "PARAMETER_VISIBILITY_BASIC", description: "context-per-request", unsignedInteger: { minimum: "1024", maximum: "262144", unit: "tokens" } } + - { parameterId: "parallel-requests", displayName: "Parallel requests", visibility: "PARAMETER_VISIBILITY_ADVANCED", description: "parallel-requests", unsignedInteger: { minimum: "1", maximum: "8", unit: "requests", defaultValue: "4" } } + - { parameterId: "host-context-mib", displayName: "Host context memory", visibility: "PARAMETER_VISIBILITY_ADVANCED", description: "host-context-mib", unsignedInteger: { minimum: "0", maximum: "1048576", unit: "MiB" } } artifacts: - artifactId: "linux-amd64-cuda" oci: @@ -1153,6 +1153,7 @@ profiles: &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) } @@ -1192,7 +1193,7 @@ profiles: } fn unsigned(minimum: u64, maximum: u64, unit: &str) -> ParameterType { - ParameterType::UnsignedInteger(UnsignedIntegerEngineParameterSchema { + ParameterType::UnsignedInteger(UnsignedIntegerParameterSchema { minimum, maximum, unit: unit.to_owned(), diff --git a/backend/core/src/model/engine/ninfer/mod.rs b/backend/core/src/model/engine/ninfer/mod.rs index 0011bf23..4bdaa16d 100644 --- a/backend/core/src/model/engine/ninfer/mod.rs +++ b/backend/core/src/model/engine/ninfer/mod.rs @@ -16,8 +16,8 @@ use dotcore_artifact::PreparedModelArtifacts; use dotcore_domain::WorkloadArtifactKind; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterValues, EngineRelease, HardwareClass, InferenceRequestKind, Model, - ModelArtifactBinding, ModelEngineConfiguration, ModelOptionValues, + EngineRelease, HardwareClass, InferenceRequestKind, Model, ModelArtifactBinding, + ModelEngineConfiguration, ModelOptionValues, ParameterValues, }, engine::v1::{EngineKind, ResolvedEngineModelState}, model::v1::{PortableModelLaunchPlan, ResolvedLocalModelDeploymentSnapshot}, @@ -78,7 +78,7 @@ impl EngineDefinition for Ninfer { &self, _model: &Model, _configuration: &ModelEngineConfiguration, - values: &EngineParameterValues, + values: &ParameterValues, ) -> anyhow::Result<()> { declaration::validate_parameter_values(values) } diff --git a/backend/core/src/model/engine/ninfer/plan.rs b/backend/core/src/model/engine/ninfer/plan.rs index bd8f2064..9ddb9d36 100644 --- a/backend/core/src/model/engine/ninfer/plan.rs +++ b/backend/core/src/model/engine/ninfer/plan.rs @@ -16,7 +16,7 @@ use dotcore_catalog::{ }, }; use dotcore_proto::dotcore::{ - catalog::v1::{ModelOptionValues, engine_parameter_value::Value as EngineParameterValue}, + catalog::v1::{ModelOptionValues, parameter_value::Value as EngineParameterValueKind}, engine::{ ninfer::v1::{ NinferFixedModelSettings, NinferKvCacheDataType, NinferResolvedModelState, @@ -367,7 +367,7 @@ fn parameter( .find(|value| value.parameter_id == parameter_id) }) .and_then(|value| match value.value.as_ref() { - Some(EngineParameterValue::UnsignedInteger(value)) => Some(*value), + Some(EngineParameterValueKind::UnsignedInteger(value)) => Some(*value), _ => None, }) .ok_or_else(|| NinferPlanError::ParameterNotConfigured(parameter_id.to_owned())) @@ -711,7 +711,7 @@ mod tests { .iter_mut() .find(|value| value.parameter_id == parameter_id) .expect("parameter") - .value = Some(EngineParameterValue::UnsignedInteger(0)); + .value = Some(EngineParameterValueKind::UnsignedInteger(0)); assert_eq!( translate(&local).expect_err("zero parameter").to_string(), diff --git a/backend/core/src/model/engine/registry.rs b/backend/core/src/model/engine/registry.rs index e6e524a4..8c87f5f1 100644 --- a/backend/core/src/model/engine/registry.rs +++ b/backend/core/src/model/engine/registry.rs @@ -9,7 +9,7 @@ use std::{collections::BTreeMap, sync::Arc}; use dotcore_artifact::PreparedModelArtifacts; use dotcore_domain::WorkloadArtifactKind; use dotcore_proto::dotcore::{ - catalog::v1::StorageScope, + catalog::v1::StoragePlacement, engine::v1::EngineKind, model::v1::{PortableModelLaunchPlan, ResolvedLocalModelDeploymentSnapshot}, }; @@ -31,7 +31,7 @@ use crate::{ NativeInferenceProvider, }, model::resolved::ResolvedModelLaunch, - storage::{StorageBindingError, workload_storage}, + storage::{StorageBindingError, engine_placement, workload_storage}, }; /// Engine-specific transport factories keyed without cross-engine fallback. @@ -179,7 +179,7 @@ impl EngineIntegrationRegistry { } /// Builds the workload of one model launch: the engine's own spec plus the storage its - /// engine release declares, keyed by `deployment_key` for deployment-scoped storage. + /// engine release declares, keyed by `deployment_key` for deployment-placed storage. pub(crate) fn materialize_workload( &self, model: &ResolvedModelLaunch, @@ -200,17 +200,22 @@ impl EngineIntegrationRegistry { )); } + let bindings = model + .engine_storage_bindings() + .iter() + .map(|binding| { + engine_placement(binding) + .map(|placement| (binding.binding_name.clone(), binding, placement)) + }) + .collect::, _>>()?; let storage = workload_storage( - model - .engine_storage_bindings() - .iter() - .map(|binding| (binding.binding_name.clone(), binding)), - |scope| match scope { - StorageScope::Deployment => Some(deployment_key.to_owned()), - StorageScope::EngineArtifact => { + bindings, + |placement| match placement { + StoragePlacement::Deployment => Some(deployment_key.to_owned()), + StoragePlacement::EngineArtifact => { Some(engine_artifact_storage_key(&model.engine.launch_identity)) } - StorageScope::Application | StorageScope::Unspecified => None, + StoragePlacement::Application | StoragePlacement::Unspecified => None, }, |name| spec.environment.contains_key(name), )?; @@ -496,18 +501,18 @@ mod tests { } #[test] - fn workload_attaches_the_engine_release_storage_keyed_by_scope() { + fn workload_attaches_the_engine_release_storage_keyed_by_placement() { let resolved = resolved_launch(vec![ cache_binding( "compile-cache", "/var/cache/compile", - StorageScope::Deployment, + StoragePlacement::Deployment, "COMPILE_CACHE", ), cache_binding( "kernel-cache", "/var/cache/kernels", - StorageScope::EngineArtifact, + StoragePlacement::EngineArtifact, "KERNEL_CACHE", ), ]); @@ -582,14 +587,14 @@ mod tests { fn cache_binding( name: &str, mount_path: &str, - scope: StorageScope, + placement: StoragePlacement, variable: &str, ) -> StorageBinding { StorageBinding { binding_name: name.to_owned(), mount_path: mount_path.to_owned(), kind: dotcore_proto::dotcore::catalog::v1::StorageKind::Cache as i32, - scope: scope as i32, + placements: vec![placement as i32], environment: vec![ dotcore_proto::dotcore::catalog::v1::StorageEnvironmentVariable { name: variable.to_owned(), diff --git a/backend/core/src/model/engine/strata/declaration.rs b/backend/core/src/model/engine/strata/declaration.rs index c338e096..487436f2 100644 --- a/backend/core/src/model/engine/strata/declaration.rs +++ b/backend/core/src/model/engine/strata/declaration.rs @@ -1,27 +1,29 @@ //! Strata-specific validation of catalog engine releases and model engine configurations. //! -//! A release must declare well-formed unsigned-integer parameters, at most the parallel requests -//! Strata's start wrapper accepts, Linux artifacts only, and the deployment's data folder the -//! wrapper writes to. A configuration must bind model artifacts shaped as its fixed settings say, -//! with the draft layer they name, run text-only on exactly one NVIDIA accelerator, declare no -//! options, and reference every other release parameter. Launch values must give Strata a total -//! context that is exactly what its parallel requests hold. +//! A release must declare well-formed unsigned-integer or secret parameters, at most the parallel +//! requests Strata's start wrapper accepts, Linux artifacts only, and the deployment's data folder +//! the wrapper writes to. A configuration must bind model artifacts shaped as its fixed settings +//! say, with the draft layer they name, run text-only on exactly one NVIDIA accelerator, declare no +//! options, and reference every other unsigned-integer release parameter. Launch values must give +//! Strata a total context that is exactly what its parallel requests hold. use std::collections::BTreeSet; -use dotcore_catalog::shared_parameters::{ - self, CONTEXT_PER_REQUEST_PARAMETER_ID, PARALLEL_REQUESTS_PARAMETER_ID, - TOTAL_CONTEXT_PARAMETER_ID, is_shared_parameter, +use dotcore_catalog::{ + shared_parameters::{ + self, CONTEXT_PER_REQUEST_PARAMETER_ID, PARALLEL_REQUESTS_PARAMETER_ID, + TOTAL_CONTEXT_PARAMETER_ID, is_shared_parameter, + }, + typed_parameters::is_secret, }; use dotcore_domain::Accelerator; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterSchema, EngineParameterValue, EngineParameterValues, EngineRelease, - HardwareClass, HuggingFaceArtifactSource, Model, ModelEngineConfiguration, - ModelInputModality, ModelParallelShape, StorageScope, - engine_parameter_definition::ParameterType, - engine_parameter_value::Value as EngineParameterValueKind, + EngineParameterSchema, EngineRelease, HardwareClass, HuggingFaceArtifactSource, Model, + ModelEngineConfiguration, ModelInputModality, ModelParallelShape, ParameterValue, + ParameterValues, StoragePlacement, hugging_face_artifact_source::Content as HuggingFaceContent, + parameter_definition::ParameterType, parameter_value::Value as EngineParameterValueKind, }, engine::{ strata::v1::{ @@ -54,9 +56,9 @@ const READY_PACK_FILES: [&str; 2] = ["native_experts.txt", "tokenizer/vocab.json /// for each of them. Returns none when the shared rule would fill nothing. pub(crate) fn filled_context_per_request( schema: &EngineParameterSchema, - values: &EngineParameterValues, + values: &ParameterValues, maximum_context_tokens: u32, -) -> Option { +) -> Option { // The shared rule with no maximum yields the total context, or none when it fills nothing. let total_context = shared_parameters::filled_context_per_request(schema, values, u32::MAX) .and_then(|filled| match filled.value { @@ -66,7 +68,7 @@ pub(crate) fn filled_context_per_request( let parallel_requests = unsigned_parameter_value(values, PARALLEL_REQUESTS_PARAMETER_ID) .unwrap_or(1) .max(1); - Some(EngineParameterValue { + Some(ParameterValue { parameter_id: CONTEXT_PER_REQUEST_PARAMETER_ID.to_owned(), value: Some(EngineParameterValueKind::UnsignedInteger( (total_context / parallel_requests).min(u64::from(maximum_context_tokens)), @@ -80,7 +82,7 @@ pub(crate) fn filled_context_per_request( pub(crate) fn validate_parameter_values( model: &Model, configuration: &ModelEngineConfiguration, - values: &EngineParameterValues, + values: &ParameterValues, ) -> anyhow::Result<()> { let (Some(total_context), Some(context_per_request), Some(parallel_requests)) = ( unsigned_parameter_value(values, TOTAL_CONTEXT_PARAMETER_ID), @@ -108,8 +110,8 @@ pub(crate) fn validate_parameter_values( Ok(()) } -/// Accepts a release of non-empty unsigned-integer parameters that declares at most the parallel -/// requests the start wrapper accepts, ships only Linux artifacts, and declares a +/// Accepts a release of non-empty unsigned-integer or secret parameters that declares at most the +/// parallel requests the start wrapper accepts, ships only Linux artifacts, and declares a /// deployment-scoped storage at the wrapper's data folder. The catalog checks the shared /// parameters' fixed shape. pub(crate) fn validate_engine_release(engine: &EngineRelease) -> Vec { @@ -123,16 +125,18 @@ pub(crate) fn validate_engine_release(engine: &EngineRelease) -> Vec= range.minimum && !range.unit.is_empty() - ) + && match definition.parameter_type.as_ref() { + Some(ParameterType::UnsignedInteger(range)) => { + range.maximum >= range.minimum && !range.unit.is_empty() + } + Some(ParameterType::Secret(_)) => true, + _ => false, + } }) }); if !valid { errors.push(anyhow::anyhow!( - "Strata engine release requires non-empty unsigned-integer parameter definitions" + "Strata engine release requires non-empty unsigned-integer or secret parameter definitions" )); } if unsigned_parameter_maximum(engine, PARALLEL_REQUESTS_PARAMETER_ID) @@ -158,11 +162,12 @@ pub(crate) fn validate_engine_release(engine: &EngineRelease) -> Vec, ) { + // Secret parameters never enter the launch plan, so the reference rule below leaves them + // out. let declared = engine .parameter_schema .as_ref() @@ -260,6 +267,7 @@ fn validate_parameters( schema .parameters .iter() + .filter(|definition| !is_secret(definition)) .map(|definition| definition.parameter_id.as_str()) .collect::>() }) @@ -272,8 +280,9 @@ fn validate_parameters( )); } - // Every release parameter must reach the launch plan, so none can be set without effect. The - // total context reaches it through the parallel requests and the context per request. + // Every other release parameter must reach the launch plan, so none can be set without + // effect. The total context reaches it through the parallel requests and the context per + // request. let launched = BTreeSet::from([ expert_vram, TOTAL_CONTEXT_PARAMETER_ID, @@ -454,9 +463,9 @@ mod tests { use dotcore_domain::ExecutionPlatform; use dotcore_proto::dotcore::{ catalog::v1::{ - BooleanEngineParameterSchema, EngineParameterDefinition, HuggingFaceArtifactSource, - HuggingFaceFileArtifact, ModelConfigurationOption, ModelImageInputCapacity, - UnsignedIntegerEngineParameterSchema, + BooleanParameterSchema, HuggingFaceArtifactSource, HuggingFaceFileArtifact, + ModelConfigurationOption, ModelImageInputCapacity, ParameterDefinition, + UnsignedIntegerParameterSchema, }, model::v1::{ LocalModelDeploymentConfiguration, ResolvedLocalModelDeploymentSnapshot, @@ -471,8 +480,10 @@ mod tests { }, test_support::{ strata::{ - DRAFT_LAYER_ARTIFACT_ID, EXPERT_VRAM_MIB_PARAMETER_ID, configuration, model, - ready_pack_configuration, release, split_gguf_configuration, + API_KEY_PARAMETER_ID, CATALOG_ENGINE, CATALOG_MODEL, DRAFT_LAYER_ARTIFACT_ID, + EXPERT_VRAM_MIB_PARAMETER_ID, catalog_engine_with_api_key, configuration, + load_catalog, model, ready_pack_configuration, release, release_with_api_key, + split_gguf_configuration, }, unsigned_parameter_values, }, @@ -514,10 +525,12 @@ mod tests { let mut boolean = release(); parameter(&mut boolean, EXPERT_VRAM_MIB_PARAMETER_ID).parameter_type = - Some(ParameterType::Boolean(BooleanEngineParameterSchema {})); + Some(ParameterType::Boolean(BooleanParameterSchema::default())); assert_eq!( messages(validate_engine_release(&boolean)), - ["Strata engine release requires non-empty unsigned-integer parameter definitions"] + [ + "Strata engine release requires non-empty unsigned-integer or secret parameter definitions" + ] ); } @@ -568,14 +581,14 @@ mod tests { let mut elsewhere = release(); elsewhere.storage_bindings[0].mount_path = "/cache".to_owned(); let mut shared = release(); - shared.storage_bindings[0].scope = StorageScope::EngineArtifact as i32; + shared.storage_bindings[0].placements = vec![StoragePlacement::EngineArtifact as i32]; let mut none = release(); none.storage_bindings.clear(); for release in [elsewhere, shared, none] { assert_eq!( messages(validate_engine_release(&release)), - ["Strata engine release must declare deployment-scoped storage at /data"] + ["Strata engine release must declare deployment-placed storage at /data"] ); } } @@ -601,6 +614,41 @@ mod tests { } } + #[test] + fn accepts_a_secret_parameter_beyond_the_launched_ones() { + let release = release_with_api_key(); + + assert_eq!( + messages(validate_engine_release(&release)), + Vec::::new() + ); + assert_eq!( + messages(validate_model_configuration( + &model(), + &configuration(), + &release + )), + Vec::::new() + ); + load_catalog(&catalog_engine_with_api_key(), CATALOG_MODEL) + .expect("a profile leaving the secret to creation"); + + let mut configuration = configuration(); + fixed_settings(&mut configuration).expert_vram_mib_parameter_id = + API_KEY_PARAMETER_ID.to_owned(); + assert_eq!( + messages(validate_model_configuration( + &model(), + &configuration, + &release + )), + [ + "configuration \"strata-nvidia\" has an invalid Strata engine parameter reference", + "configuration \"strata-nvidia\" must reference every Strata engine release parameter", + ] + ); + } + #[test] fn rejects_release_parameters_the_configuration_does_not_reference() { let mut release = release(); @@ -609,7 +657,7 @@ mod tests { .as_mut() .expect("schema") .parameters - .push(EngineParameterDefinition { + .push(ParameterDefinition { parameter_id: "kv-bits".to_owned(), parameter_type: Some(unsigned(4, 16, "bits")), ..Default::default() @@ -838,7 +886,7 @@ mod tests { ); assert_ne!(model, CATALOG_MODEL); - let error = load_catalog(&model).expect_err("no expert cap"); + let error = load_catalog(CATALOG_ENGINE, &model).expect_err("no expert cap"); assert!( format!("{error:#}").contains( @@ -1008,111 +1056,9 @@ mod tests { ); } - const CATALOG_ENGINE: &str = r#" -engineReleaseId: "fixture-strata" -engineKind: "ENGINE_KIND_STRATA" -version: "1" -parameterSchema: - schemaId: "fixture-strata-v1" - parameters: - - { parameterId: "total-context", displayName: "Total context", exposed: true, description: "total-context", unsignedInteger: { minimum: "1024", maximum: "2097152", unit: "tokens", defaultValue: "32768" } } - - { parameterId: "context-per-request", displayName: "Context per request", exposed: true, description: "context-per-request", unsignedInteger: { minimum: "1024", maximum: "262144", unit: "tokens" } } - - { parameterId: "parallel-requests", displayName: "Parallel requests", exposed: true, description: "parallel-requests", unsignedInteger: { minimum: "1", maximum: "8", unit: "requests", defaultValue: "1" } } - - { parameterId: "expert-vram-mib", displayName: "Expert accelerator memory", exposed: true, description: "expert-vram-mib", unsignedInteger: { minimum: "1", maximum: "1048576", unit: "MiB" } } -storageBindings: - - { bindingName: "data", mountPath: "/data", kind: "STORAGE_KIND_CACHE", scope: "STORAGE_SCOPE_DEPLOYMENT" } -artifacts: - - artifactId: "linux-amd64-cuda" - oci: - repository: { registry: { host: "registry.dotcore.test" }, name: "strata" } - digest: { algorithm: "sha256", encoded: "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" } - selectors: - - platform: { operatingSystem: "linux", architecture: "amd64" } - accelerators: ["nvidia"] - minimumNvidiaDriverMajorVersion: 580 -"#; - - const CATALOG_MODEL: &str = r#" -modelId: "fixture-strata-model" -displayName: "Fixture Strata model" -version: "1" -maximumContextTokens: 262144 -artifacts: - - artifactId: "fixture-strata-artifact" - weightQuantization: "Q4_K_XL" - source: - repository: "dotcore/fixture-strata" - revision: "0123456789abcdef0123456789abcdef01234567" - file: - path: "fixture-Q4_K_XL.gguf" - provenanceDigest: "74d2c57145e6ff11d1d2faa79594477f9bc903a611af1fb20218189fbbb77d82" -configurations: - - configurationId: "strata-nvidia" - engineArtifacts: [{ engineReleaseId: "fixture-strata", engineArtifactId: "linux-amd64-cuda" }] - artifactBindings: [{ bindingId: "fixture-strata-artifact", modelArtifactId: "fixture-strata-artifact" }] - sizing: - parameters: { baseBytes: 1000, bytesPerKvToken: 10, expertMib: 20000, expertVramMib: 8192 } - derivedParameters: - expert-vram-mib: "${expertVramMib}" - estimates: - acceleratorMemory: - unit: bytes - equation: "${baseBytes} + ${bytesPerKvToken} * ${total-context} + ${expert-vram-mib} * MiB" - hostMemory: - unit: bytes - equation: "${expertMib} * MiB" - hardwareSupport: - - hardwareClassId: "nvidia" - hardwareRequirements: - minimumMemoryBytesPerDevice: "32000000000" - minimumSystemMemoryBytes: "64000000000" - bindingSizing: - - bindingId: "fixture-strata-artifact" - engineConfiguration: - engineKind: "ENGINE_KIND_STRATA" - strata: - fixedSettings: - expertVramMibParameterId: "expert-vram-mib" - architecture: "qwen35moe" - mtpDraftTokens: 2 - singleGguf: {} - executionCapabilities: { supportedParallelShapes: ["MODEL_PARALLEL_SHAPE_ONE"] } - requestKinds: ['INFERENCE_REQUEST_KIND_CHAT'] - inputCapabilities: - supportedModalities: ["MODEL_INPUT_MODALITY_TEXT"] -profiles: - - profileId: "strata-q4" - displayName: "Q4 Strata" - description: "Runs the fixture through Strata." - configurationId: "strata-nvidia" - defaultModelArtifactBindingId: "fixture-strata-artifact" - engineParameterDefaults: {} - "#; - /// Loads the profile-shaped engine and model with the shared hardware classes. fn profile_shaped_catalog() -> dotcore_catalog::Catalog { - load_catalog(CATALOG_MODEL).expect("Strata catalog") - } - - /// Loads the engine and the given model with the shared hardware classes. - fn load_catalog(model: &str) -> anyhow::Result { - let root = tempfile::tempdir().expect("Catalog root"); - for (path, content) in [ - ("engines/strata.yaml", CATALOG_ENGINE), - ("models/strata-model.yaml", model), - ] { - let path = root.path().join(path); - std::fs::create_dir_all(path.parent().expect("directory")).expect("directory"); - std::fs::write(path, content).expect("declaration"); - } - dotcore_catalog::test_support::write_hardware_classes(root.path()); - - dotcore_catalog::Catalog::load_with_policy( - root.path(), - dotcore_catalog::OciArtifactPolicy::Development, - crate::model::engine::declaration_validator(), - ) - .map_err(anyhow::Error::new) + load_catalog(CATALOG_ENGINE, CATALOG_MODEL).expect("Strata catalog") } /// Resolves the profile on an NVIDIA target with the supplied engine parameter values after @@ -1143,6 +1089,7 @@ profiles: &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) } @@ -1184,7 +1131,7 @@ profiles: fn parameter<'a>( release: &'a mut EngineRelease, parameter_id: &str, - ) -> &'a mut EngineParameterDefinition { + ) -> &'a mut ParameterDefinition { release .parameter_schema .as_mut() @@ -1196,7 +1143,7 @@ profiles: } fn unsigned(minimum: u64, maximum: u64, unit: &str) -> ParameterType { - ParameterType::UnsignedInteger(UnsignedIntegerEngineParameterSchema { + ParameterType::UnsignedInteger(UnsignedIntegerParameterSchema { minimum, maximum, unit: unit.to_owned(), diff --git a/backend/core/src/model/engine/strata/mod.rs b/backend/core/src/model/engine/strata/mod.rs index 73c05eb5..662812ed 100644 --- a/backend/core/src/model/engine/strata/mod.rs +++ b/backend/core/src/model/engine/strata/mod.rs @@ -20,9 +20,9 @@ use dotcore_artifact::PreparedModelArtifacts; use dotcore_domain::WorkloadArtifactKind; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterSchema, EngineParameterValue, EngineParameterValues, EngineRelease, - HardwareClass, InferenceRequestKind, Model, ModelArtifactBinding, ModelEngineConfiguration, - ModelOptionValues, + EngineParameterSchema, EngineRelease, HardwareClass, InferenceRequestKind, Model, + ModelArtifactBinding, ModelEngineConfiguration, ModelOptionValues, ParameterValue, + ParameterValues, }, engine::{ strata::v1::{StrataFixedModelSettings, strata_fixed_model_settings::ModelFiles}, @@ -91,9 +91,9 @@ impl EngineDefinition for Strata { fn filled_context_per_request( &self, schema: &EngineParameterSchema, - values: &EngineParameterValues, + values: &ParameterValues, maximum_context_tokens: u32, - ) -> Option { + ) -> Option { declaration::filled_context_per_request(schema, values, maximum_context_tokens) } @@ -101,7 +101,7 @@ impl EngineDefinition for Strata { &self, model: &Model, configuration: &ModelEngineConfiguration, - values: &EngineParameterValues, + values: &ParameterValues, ) -> anyhow::Result<()> { declaration::validate_parameter_values(model, configuration, values) } diff --git a/backend/core/src/model/engine/strata/plan.rs b/backend/core/src/model/engine/strata/plan.rs index 5b811693..450b4479 100644 --- a/backend/core/src/model/engine/strata/plan.rs +++ b/backend/core/src/model/engine/strata/plan.rs @@ -12,7 +12,7 @@ use dotcore_catalog::shared_parameters::{ CONTEXT_PER_REQUEST_PARAMETER_ID, PARALLEL_REQUESTS_PARAMETER_ID, }; use dotcore_proto::dotcore::{ - catalog::v1::engine_parameter_value::Value as EngineParameterValue, + catalog::v1::parameter_value::Value as EngineParameterValueKind, engine::{ strata::v1::{StrataResolvedModelState, strata_fixed_model_settings::ModelFiles}, v1::{EngineKind, resolved_engine_model_state::State}, @@ -213,7 +213,7 @@ fn positive_parameter( .find(|value| value.parameter_id == parameter_id) }) .and_then(|value| match value.value.as_ref() { - Some(EngineParameterValue::UnsignedInteger(value)) => Some(*value), + Some(EngineParameterValueKind::UnsignedInteger(value)) => Some(*value), _ => None, }) .filter(|value| *value > 0) @@ -591,7 +591,7 @@ mod tests { .iter_mut() .find(|value| value.parameter_id == parameter_id) .expect("parameter") - .value = Some(EngineParameterValue::UnsignedInteger(0)); + .value = Some(EngineParameterValueKind::UnsignedInteger(0)); let mut missing = local("qwen3-6-35b-a3b"); missing .engine_parameter_values diff --git a/backend/core/src/model/engine/vllm/declaration.rs b/backend/core/src/model/engine/vllm/declaration.rs index 5c881920..fc6838c8 100644 --- a/backend/core/src/model/engine/vllm/declaration.rs +++ b/backend/core/src/model/engine/vllm/declaration.rs @@ -17,10 +17,10 @@ use dotcore_proto::dotcore::{ catalog::v1::{ EngineRelease, HardwareClass, Model, ModelConfigurationOption, ModelEngineConfiguration, ModelInputModality, ModelParallelShape, SpeculativeDecodingMethod, - engine_parameter_definition::ParameterType, hugging_face_artifact_source::Content as HuggingFaceContent, model_option::Kind as ModelOptionKind, model_option_parameter_definition::ParameterType as ModelOptionParameterType, + parameter_definition::ParameterType, }, engine::{ v1::engine_model_configuration::Configuration, @@ -62,7 +62,12 @@ pub(crate) fn validate_engine_release(engine: &EngineRelease) -> Vec= range.minimum && !range.unit.is_empty() } Some(ParameterType::Boolean(_)) => true, - None => false, + Some( + ParameterType::StringEnum(_) + | ParameterType::Text(_) + | ParameterType::Secret(_), + ) + | None => false, } }) }); @@ -431,10 +436,10 @@ mod tests { use dotcore_proto::dotcore::{ catalog::v1::{ - BooleanEngineParameterSchema, EngineParameterDefinition, EngineParameterSchema, - EngineRelease, HuggingFaceArtifactSource, Model, ModelArtifactBinding, - ModelAuxiliaryArtifact, ModelExecutionCapabilities, SpeculativeDecodingModelOption, - UnsignedIntegerEngineParameterSchema, + BooleanParameterSchema, EngineParameterSchema, EngineRelease, + HuggingFaceArtifactSource, Model, ModelArtifactBinding, ModelAuxiliaryArtifact, + ModelExecutionCapabilities, ParameterDefinition, SpeculativeDecodingModelOption, + UnsignedIntegerParameterSchema, }, engine::v1::{ EngineKind, EngineModelConfiguration, engine_model_configuration::Configuration, @@ -461,10 +466,10 @@ mod tests { }; fn release(kind: EngineKind) -> EngineRelease { - let unsigned = |parameter_id: &str, maximum: u64, unit: &str| EngineParameterDefinition { + let unsigned = |parameter_id: &str, maximum: u64, unit: &str| ParameterDefinition { parameter_id: parameter_id.to_owned(), parameter_type: Some(ParameterType::UnsignedInteger( - UnsignedIntegerEngineParameterSchema { + UnsignedIntegerParameterSchema { minimum: 1, maximum, unit: unit.to_owned(), @@ -497,10 +502,10 @@ mod tests { .as_mut() .expect("schema") .parameters - .push(EngineParameterDefinition { + .push(ParameterDefinition { parameter_id: "feature-enabled".to_owned(), display_name: "Feature enabled".to_owned(), - parameter_type: Some(ParameterType::Boolean(BooleanEngineParameterSchema {})), + parameter_type: Some(ParameterType::Boolean(BooleanParameterSchema::default())), ..Default::default() }); assert!(validate_engine_release(&with_boolean).is_empty()); @@ -870,6 +875,7 @@ mod tests { &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("vLLM launch on a GPU of a mixed machine"); let total = launch @@ -1041,13 +1047,13 @@ version: "1" parameterSchema: schemaId: "fixture-vllm-v2" parameters: - - { parameterId: "total-context", displayName: "Total context", exposed: true, unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } } - - { parameterId: "context-per-request", displayName: "Context per request", exposed: true, unsignedInteger: { minimum: "1024", maximum: "1048576", unit: "tokens" } } - - { parameterId: "parallel-requests", displayName: "Parallel requests", exposed: true, unsignedInteger: { minimum: "1", maximum: "1024", unit: "requests", defaultValue: "4" } } - - { parameterId: "maximum-batched-tokens", displayName: "Maximum batched tokens", unsignedInteger: { minimum: "1", maximum: "1048576", unit: "tokens" } } - - { parameterId: "maximum-cudagraph-capture-size", displayName: "Maximum CUDA graph capture size", unsignedInteger: { minimum: "1", maximum: "1024", unit: "sequences" } } - - { parameterId: "kv-cache-memory-bytes", displayName: "KV cache memory", unsignedInteger: { minimum: "1", maximum: "1099511627776", unit: "bytes" } } - - { parameterId: "kv-offloading-size-gib", displayName: "KV offloading size", exposed: true, unsignedInteger: { minimum: "0", maximum: "1024", unit: "GiB" } } + - { parameterId: "total-context", displayName: "Total context", visibility: "PARAMETER_VISIBILITY_BASIC", unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } } + - { parameterId: "context-per-request", displayName: "Context per request", visibility: "PARAMETER_VISIBILITY_BASIC", unsignedInteger: { minimum: "1024", maximum: "1048576", unit: "tokens" } } + - { parameterId: "parallel-requests", displayName: "Parallel requests", visibility: "PARAMETER_VISIBILITY_ADVANCED", unsignedInteger: { minimum: "1", maximum: "1024", unit: "requests", defaultValue: "4" } } + - { parameterId: "maximum-batched-tokens", displayName: "Maximum batched tokens", visibility: "PARAMETER_VISIBILITY_HIDDEN", unsignedInteger: { minimum: "1", maximum: "1048576", unit: "tokens" } } + - { parameterId: "maximum-cudagraph-capture-size", displayName: "Maximum CUDA graph capture size", visibility: "PARAMETER_VISIBILITY_HIDDEN", unsignedInteger: { minimum: "1", maximum: "1024", unit: "sequences" } } + - { parameterId: "kv-cache-memory-bytes", displayName: "KV cache memory", visibility: "PARAMETER_VISIBILITY_HIDDEN", unsignedInteger: { minimum: "1", maximum: "1099511627776", unit: "bytes" } } + - { parameterId: "kv-offloading-size-gib", displayName: "KV offloading size", visibility: "PARAMETER_VISIBILITY_ADVANCED", unsignedInteger: { minimum: "0", maximum: "1024", unit: "GiB" } } artifacts: - artifactId: "linux-amd64-cuda" oci: @@ -1206,6 +1212,7 @@ profiles: &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) } diff --git a/backend/core/src/model/engine/vllm/plan.rs b/backend/core/src/model/engine/vllm/plan.rs index ce737dc3..044c5d63 100644 --- a/backend/core/src/model/engine/vllm/plan.rs +++ b/backend/core/src/model/engine/vllm/plan.rs @@ -12,7 +12,7 @@ use dotcore_catalog::{ shared_parameters::{CONTEXT_PER_REQUEST_PARAMETER_ID, PARALLEL_REQUESTS_PARAMETER_ID}, }; use dotcore_proto::dotcore::{ - catalog::v1::engine_parameter_value::Value as EngineParameterValue, + catalog::v1::parameter_value::Value as EngineParameterValueKind, engine::{ v1::{EngineKind, resolved_engine_model_state::State}, vllm::v1::{ @@ -443,7 +443,7 @@ fn unsigned_integer_parameter( .find(|value| value.parameter_id == parameter_id) }) .and_then(|value| match value.value.as_ref() { - Some(EngineParameterValue::UnsignedInteger(value)) => Some(*value), + Some(EngineParameterValueKind::UnsignedInteger(value)) => Some(*value), _ => None, }) .ok_or_else(|| VllmPlanError::ParameterNotConfigured(parameter_id.to_owned())) @@ -595,8 +595,8 @@ pub(crate) enum VllmPlanError { mod tests { use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterValue, EngineParameterValues, ModelArtifact, ModelAuxiliaryArtifact, - ModelOptionParameterValue, ModelOptionValues, engine_parameter_value::Value, + ModelArtifact, ModelAuxiliaryArtifact, ModelOptionParameterValue, ModelOptionValues, + ParameterValue, ParameterValues, parameter_value::Value, }, engine::v1::{ResolvedEngineModelState, resolved_engine_model_state::State}, engine::vllm::v1::{ @@ -657,7 +657,7 @@ mod tests { semantics_version: super::super::realization::SEMANTICS_VERSION, state: Some(State::Vllm(state)), }), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: [ ("bytes", 7000_u64), (PARALLEL_REQUESTS_PARAMETER_ID, 8), @@ -667,7 +667,7 @@ mod tests { (CONTEXT_PER_REQUEST_PARAMETER_ID, 32768), ] .into_iter() - .map(|(parameter_id, value)| EngineParameterValue { + .map(|(parameter_id, value)| ParameterValue { parameter_id: parameter_id.to_owned(), value: Some(Value::UnsignedInteger(value)), }) diff --git a/backend/core/src/model/recommendation/tests.rs b/backend/core/src/model/recommendation/tests.rs index 7366e95f..64fdd470 100644 --- a/backend/core/src/model/recommendation/tests.rs +++ b/backend/core/src/model/recommendation/tests.rs @@ -910,10 +910,10 @@ version: "1" parameterSchema: schemaId: "llama-v2" parameters: - - { parameterId: "total-context", displayName: "Total context", exposed: true, unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } } - - { parameterId: "context-per-request", displayName: "Context per request", exposed: true, unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens" } } - - { parameterId: "parallel-requests", displayName: "Parallel requests", exposed: true, unsignedInteger: { minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" } } - - { parameterId: "prompt-cache-memory-mib", displayName: "Prompt cache memory", unsignedInteger: { minimum: "0", maximum: "65536", unit: "MiB" } } + - { parameterId: "total-context", displayName: "Total context", visibility: "PARAMETER_VISIBILITY_BASIC", unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } } + - { parameterId: "context-per-request", displayName: "Context per request", visibility: "PARAMETER_VISIBILITY_BASIC", unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens" } } + - { parameterId: "parallel-requests", displayName: "Parallel requests", visibility: "PARAMETER_VISIBILITY_ADVANCED", unsignedInteger: { minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" } } + - { parameterId: "prompt-cache-memory-mib", displayName: "Prompt cache memory", visibility: "PARAMETER_VISIBILITY_HIDDEN", unsignedInteger: { minimum: "0", maximum: "65536", unit: "MiB" } } artifacts: - artifactId: "linux-amd64" oci: diff --git a/backend/core/src/model/repository.rs b/backend/core/src/model/repository.rs index ab7767fa..33743df7 100644 --- a/backend/core/src/model/repository.rs +++ b/backend/core/src/model/repository.rs @@ -41,7 +41,11 @@ use super::{ recommendation::{RecommendationFit, UseCaseRecommendations, recommend_models}, sizing_scenario::SizingScenario, }; -use crate::route_manager::{RouteDeclarationRepository, RouteRepositoryError}; +use crate::{ + RandomSecretGenerator, SecretGenerator, + model::resolution::EngineSecrets, + route_manager::{RouteDeclarationRepository, RouteRepositoryError}, +}; #[derive(Clone)] pub struct ModelDeclarationRepository { @@ -50,6 +54,7 @@ pub struct ModelDeclarationRepository { routes: RouteDeclarationRepository, target_capabilities: Arc>, lifecycle: Option, + secret_generator: Arc, } impl ModelDeclarationRepository { @@ -64,6 +69,7 @@ impl ModelDeclarationRepository { catalog, target_capabilities: Arc::new(RwLock::new(target_capabilities)), lifecycle: None, + secret_generator: Arc::new(RandomSecretGenerator), } } @@ -192,6 +198,7 @@ impl ModelDeclarationRepository { profile_id, placement, configuration, + EngineSecrets::Generate(self.secret_generator.as_ref()), )?; deployment.model_deployment_id = deployment_id; deployment.lifecycle_generation = @@ -223,8 +230,14 @@ impl ModelDeclarationRepository { ModelDeclarationRepositoryError, > { let site = self.site.model_snapshot()?; - let deployment = - self.resolve_model_deployment(&site, model_id, profile_id, placement, configuration)?; + let deployment = self.resolve_model_deployment( + &site, + model_id, + profile_id, + placement, + configuration, + EngineSecrets::Withhold, + )?; match deployment.snapshot.and_then(|snapshot| snapshot.source) { Some(ModelDeploymentSnapshotSource::Local(local)) => Ok(local), Some(ModelDeploymentSnapshotSource::Upstream(_)) | None => { @@ -602,6 +615,7 @@ impl ModelDeclarationRepository { profile_id: &str, placement: &ExactPlacement, configuration: &LocalModelDeploymentConfiguration, + secrets: EngineSecrets<'_>, ) -> Result { let catalog_model_id = CatalogModelId(model_id.to_owned()); validate_model_profile( @@ -629,6 +643,7 @@ impl ModelDeclarationRepository { .unwrap_or_else(|value| value.into_inner()), // The SiteNode named by the exact placement pins a local-development engine itself. None, + secrets, ) .map_err(launch_resolution_error)?; if local.model_id != model_id || local.profile_id != profile_id { @@ -653,6 +668,13 @@ impl ModelDeclarationRepository { node_resolved_launch_identities: Vec::new(), }) } + + /// Replaces the generator of the secret values new deployments retain. + #[cfg(test)] + pub(crate) fn with_secret_generator(mut self, generator: Arc) -> Self { + self.secret_generator = generator; + self + } } /// The target a resource estimate is evaluated on: its placement, the target observations that @@ -813,6 +835,9 @@ pub(crate) fn launch_resolution_error( error @ crate::model::resolution::ModelLaunchResolutionError::SizingUnavailable { .. } => ModelDeclarationRepositoryError::Sizing(Box::new(error)), + error @ crate::model::resolution::ModelLaunchResolutionError::SecretGeneration { + .. + } => ModelDeclarationRepositoryError::SecretGeneration(Box::new(error)), } } @@ -930,6 +955,9 @@ pub enum ModelDeclarationRepositoryError { #[error("could not pin the engine launch identity: {0}")] EngineLaunchIdentity(#[source] Box), + #[error(transparent)] + SecretGeneration(Box), + #[error("Catalog model state is invalid: {0}")] InvalidState(&'static str), @@ -1022,9 +1050,9 @@ mod tests { ResolvedApplicationDeploymentSnapshot, }, catalog::v1::{ - EngineParameterValue, EngineParameterValues, OciPlatform, UpstreamModelBoundary, + OciPlatform, ParameterValue, ParameterValues, UpstreamModelBoundary, UpstreamModelDataEgressClass, UpstreamModelLocality, - engine_parameter_value::Value as EngineParameterValueKind, + parameter_value::Value as EngineParameterValueKind, }, deployment::v1::ExactPlacement, inference::v1::{FallbackGroup, FallbackGroupSelection, InferenceRoute}, @@ -1231,6 +1259,211 @@ mod tests { assert!(!site.is_dirty()); } + #[test] + fn creation_generates_a_missing_engine_secret_once_after_sizing_and_keeps_a_typed_one() { + use crate::test_support::strata::{ + API_KEY_GENERATED_LENGTH, API_KEY_PARAMETER_ID, EXPERT_VRAM_MIB_PARAMETER_ID, + }; + + let directory = test_directory(); + let site_path = strata_site(directory.path()); + let repository = strata_repository(&site_path, crate::test_support::strata::CATALOG_MODEL); + let placement = fixture_placement(); + let configuration = LocalModelDeploymentConfiguration::default(); + let typed_key = "typed-api-key-0123456789"; + let typed = LocalModelDeploymentConfiguration { + engine_parameter_values: Some(ParameterValues { + values: vec![secret_value(API_KEY_PARAMETER_ID, typed_key)], + }), + ..Default::default() + }; + + // Advisory reads size and preview the launch without any secret value. + let estimate = repository + .estimate_local_model_resources("fixture-strata-model", "strata-q4", &typed, None) + .expect("estimate without a generated secret"); + assert_eq!(estimate.hardware_class_estimates.len(), 1); + let preview = repository + .preview_local_model("fixture-strata-model", "strata-q4", &placement, &typed) + .expect("preview without a secret value"); + assert_eq!( + secret_of(&preview, API_KEY_PARAMETER_ID), + None, + "a preview withholds even a typed secret" + ); + + let ((generated, kept), records, output) = + dotcore_domain::test_support::tracing::capture_records(|| { + let load = |configuration| { + repository.load_model( + "fixture-strata-model", + "strata-q4", + &placement, + configuration, + &ModelDeploymentExposureIntent::Hidden, + None, + ) + }; + ( + load(&configuration).expect("deployment with a generated secret"), + load(&typed).expect("deployment with a typed secret"), + ) + }); + + // The previews and estimate drew nothing: the first generated value is the first call. + let generated_local = local_snapshot(&generated); + assert_eq!( + secret_of(&generated_local, API_KEY_PARAMETER_ID), + Some(crate::test_support::sequential_secret( + 1, + API_KEY_GENERATED_LENGTH as usize + )) + ); + assert_eq!( + generated_local + .sizing + .as_ref() + .map(|sizing| sizing.derived_engine_parameter_ids.clone()), + Some(vec![ + "context-per-request".to_owned(), + EXPERT_VRAM_MIB_PARAMETER_ID.to_owned(), + ]) + ); + let kept_local = local_snapshot(&kept); + assert_eq!( + secret_of(&kept_local, API_KEY_PARAMETER_ID), + Some(typed_key.to_owned()) + ); + assert_eq!( + generated_local.desired_launch_plan, kept_local.desired_launch_plan, + "the secret never reaches the launch plan" + ); + assert_eq!( + generated_local.desired_launch_plan, + preview.desired_launch_plan + ); + for secret in [typed_key, &crate::test_support::sequential_secret(1, 32)] { + assert!(!output.contains(secret), "{output}"); + assert!(!format!("{records:?}").contains(secret), "{records:#?}"); + } + + // A relaunch reads the retained snapshot, which a reloaded site keeps as created. + let reloaded = + SiteRepository::load(&site_path, crate::test_support::site_declaration_rules()) + .expect("reloaded site"); + let retained = reloaded + .model_snapshot() + .expect("reloaded deployments") + .deployments; + for created in [&generated, &kept] { + let retained = retained + .iter() + .find(|deployment| deployment.model_deployment_id == created.model_deployment_id) + .expect("retained deployment"); + assert_eq!(retained.snapshot, created.snapshot); + crate::model::resolved::ResolvedModelLaunch::from_declaration(retained) + .expect("relaunch from the retained snapshot"); + } + } + + #[test] + fn a_typed_engine_secret_is_checked_without_being_quoted() { + use crate::test_support::strata::API_KEY_PARAMETER_ID; + + let directory = test_directory(); + let site_path = strata_site(directory.path()); + let repository = strata_repository(&site_path, crate::test_support::strata::CATALOG_MODEL); + + for (value, reason) in [ + ("short-key", "it is shorter than the minimum length of 16"), + ( + "unsafe=api-key-0123456789", + "it contains `=` or a control character", + ), + ] { + let error = repository + .load_model( + "fixture-strata-model", + "strata-q4", + &fixture_placement(), + &LocalModelDeploymentConfiguration { + engine_parameter_values: Some(ParameterValues { + values: vec![secret_value(API_KEY_PARAMETER_ID, value)], + }), + ..Default::default() + }, + &ModelDeploymentExposureIntent::Hidden, + None, + ) + .expect_err("invalid typed secret") + .to_string(); + + assert_eq!( + error, + format!( + "invalid engine parameter values: profile \"strata-q4\" engine parameter \"api-key\" value is invalid: {reason}" + ) + ); + } + } + + #[test] + fn the_snapshot_retains_the_profile_engine_parameter_defaults_and_visibility_overrides() { + use dotcore_proto::dotcore::catalog::v1::{ + ParameterVisibility, ParameterVisibilityOverride, + }; + + let directory = test_directory(); + let site_path = strata_site(directory.path()); + let model = crate::test_support::strata::CATALOG_MODEL.replace( + " engineParameterDefaults: {}\n", + r#" engineParameterDefaults: + values: + - { parameterId: "parallel-requests", unsignedInteger: "2" } + engineParameterVisibilityOverrides: + - { parameterId: "expert-vram-mib", visibility: "PARAMETER_VISIBILITY_HIDDEN" } + - { parameterId: "api-key", visibility: "PARAMETER_VISIBILITY_BASIC" } +"#, + ); + assert_ne!(model, crate::test_support::strata::CATALOG_MODEL); + let repository = strata_repository(&site_path, &model); + + let created = repository + .load_model( + "fixture-strata-model", + "strata-q4", + &fixture_placement(), + &LocalModelDeploymentConfiguration::default(), + &ModelDeploymentExposureIntent::Hidden, + None, + ) + .expect("deployment"); + + let local = local_snapshot(&created); + assert_eq!( + local.profile_engine_parameter_defaults, + Some(ParameterValues { + values: vec![ + unsigned_value("parallel-requests", 2), + unsigned_value("total-context", 32_768), + ], + }) + ); + assert_eq!( + local.profile_engine_parameter_visibility_overrides, + [ + ParameterVisibilityOverride { + parameter_id: "expert-vram-mib".to_owned(), + visibility: ParameterVisibility::Hidden as i32, + }, + ParameterVisibilityOverride { + parameter_id: "api-key".to_owned(), + visibility: ParameterVisibility::Basic as i32, + }, + ] + ); + } + #[test] fn tag_only_engine_is_created_unpinned_for_its_hosting_node_to_pin() { let directory = test_directory(); @@ -1422,6 +1655,7 @@ mod tests { &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("direct launch resolution"); @@ -3120,7 +3354,7 @@ mod tests { /// Engine parameter values for a llama.cpp deployment: the total context, the prompt cache and /// the parallel requests that sizing never derives. - fn engine_values(total_context: u64) -> EngineParameterValues { + fn engine_values(total_context: u64) -> ParameterValues { unsigned_values(&[ ("total-context", total_context), ("prompt-cache-memory-mib", 0), @@ -3130,7 +3364,7 @@ mod tests { /// The fixture profile's engine parameter defaults: its prompt cache, then the engine defaults /// of the total context and the parallel requests. - fn profile_default_values() -> EngineParameterValues { + fn profile_default_values() -> ParameterValues { unsigned_values(&[ ("prompt-cache-memory-mib", 0), ("total-context", 32_768), @@ -3140,28 +3374,25 @@ mod tests { /// The context per request resolution fills from the total context, within the fixture /// model's maximum context. - fn filled_context_per_request(total_context: u64) -> EngineParameterValues { + fn filled_context_per_request(total_context: u64) -> ParameterValues { unsigned_values(&[("context-per-request", total_context.min(262_144))]) } /// Values in the given order, followed by the filled context per request, as a resolved /// snapshot retains them. - fn resolved_values( - mut values: EngineParameterValues, - total_context: u64, - ) -> EngineParameterValues { + fn resolved_values(mut values: ParameterValues, total_context: u64) -> ParameterValues { values .values .extend(filled_context_per_request(total_context).values); values } - fn unsigned_values(values: &[(&str, u64)]) -> EngineParameterValues { - EngineParameterValues { + fn unsigned_values(values: &[(&str, u64)]) -> ParameterValues { + ParameterValues { values: values .iter() .copied() - .map(|(parameter_id, value)| EngineParameterValue { + .map(|(parameter_id, value)| ParameterValue { parameter_id: parameter_id.to_owned(), value: Some(EngineParameterValueKind::UnsignedInteger(value)), }) @@ -3215,6 +3446,84 @@ mod tests { catalog_root } + /// Copies the simulated-site fixture, whose NVIDIA target runs Strata, into `directory`. + fn strata_site(directory: &Path) -> std::path::PathBuf { + let site_path = directory.join("site.yaml"); + fs::copy( + Path::new(env!("CARGO_MANIFEST_DIR")).join("../fixtures/simulated-site/site.yaml"), + &site_path, + ) + .expect("simulated site fixture"); + site_path + } + + /// A repository over the site at `site_path` and the Strata engine with an API key secret + /// beside `model`, generating predictable secrets. + fn strata_repository(site_path: &Path, model: &str) -> ModelDeclarationRepository { + let catalog = crate::test_support::strata::load_catalog( + &crate::test_support::strata::catalog_engine_with_api_key(), + model, + ) + .expect("Strata catalog with an API key"); + ModelDeclarationRepository::new( + SiteRepository::load(site_path, crate::test_support::site_declaration_rules()) + .expect("validated simulated site"), + Arc::new(catalog), + fixture_target_capabilities(), + ) + .with_secret_generator(crate::test_support::test_secret_generator()) + } + + fn fixture_placement() -> ExactPlacement { + ExactPlacement { + site_node_id: "fixture-node".to_owned(), + execution_target_id: "fixture-target".to_owned(), + } + } + + fn local_snapshot( + deployment: &ModelDeploymentDeclaration, + ) -> dotcore_proto::dotcore::model::v1::ResolvedLocalModelDeploymentSnapshot { + match deployment + .snapshot + .as_ref() + .and_then(|snapshot| snapshot.source.as_ref()) + { + Some(ModelDeploymentSnapshotSource::Local(local)) => local.clone(), + _ => panic!("local snapshot"), + } + } + + /// The retained string value of `parameter_id`, if any. + fn secret_of( + local: &dotcore_proto::dotcore::model::v1::ResolvedLocalModelDeploymentSnapshot, + parameter_id: &str, + ) -> Option { + local + .engine_parameter_values + .iter() + .flat_map(|values| &values.values) + .find(|value| value.parameter_id == parameter_id) + .and_then(|value| match value.value.as_ref() { + Some(EngineParameterValueKind::String(secret)) => Some(secret.clone()), + _ => None, + }) + } + + fn secret_value(parameter_id: &str, value: &str) -> ParameterValue { + ParameterValue { + parameter_id: parameter_id.to_owned(), + value: Some(EngineParameterValueKind::String(value.to_owned())), + } + } + + fn unsigned_value(parameter_id: &str, value: u64) -> ParameterValue { + ParameterValue { + parameter_id: parameter_id.to_owned(), + value: Some(EngineParameterValueKind::UnsignedInteger(value)), + } + } + fn fixture_target_capabilities() -> TargetCapabilityInventory { crate::test_support::execution_target_capabilities( dotcore_domain::ExecutionPlatform::linux_amd64(), diff --git a/backend/core/src/model/resolution.rs b/backend/core/src/model/resolution.rs index b553c2c2..db8649c7 100644 --- a/backend/core/src/model/resolution.rs +++ b/backend/core/src/model/resolution.rs @@ -7,18 +7,18 @@ use dotcore_artifact::{ArtifactManager, LocalImageIdentityResolutionError}; use dotcore_catalog::{ ACCELERATOR_MEMORY_RESOURCE, Catalog, CatalogError, CatalogModelId, HOST_MEMORY_RESOURCE, ScalarResourceEstimate, SupportedHardwareClass, options::resolve_model_option_values, - overlay_parameter_defaults, shared_parameters, validate_parameter_values, + overlay_parameter_defaults, shared_parameters, validate_parameter_values_before_secrets, validate_supplied_parameter_values, }; use dotcore_domain::deployment::artifact::immutable_remote_launch_identity; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineArtifact, EngineParameterSchema, EngineParameterValue, EngineParameterValues, - EngineRelease, Model, ModelArtifact, ModelArtifactBinding, ModelEngineConfiguration, - ModelOptionValues, ModelProfile, NixArtifact, engine_artifact::Kind as EngineArtifactKind, - engine_parameter_definition::ParameterType as EngineParameterType, - engine_parameter_value::Value as EngineParameterValueKind, + EngineArtifact, EngineParameterSchema, EngineRelease, Model, ModelArtifact, + ModelArtifactBinding, ModelEngineConfiguration, ModelOptionValues, ModelProfile, + NixArtifact, ParameterValue, ParameterValues, engine_artifact::Kind as EngineArtifactKind, model_option_parameter_value::Value as OptionParameterValueKind, + parameter_definition::ParameterType as EngineParameterType, + parameter_value::Value as EngineParameterValueKind, }, deployment::v1::ArtifactPreparationPolicy, model::v1::{ @@ -32,6 +32,7 @@ use dotcore_proto::dotcore::{ use dotcore_site::node::target_inventory::TargetCapabilityInventory; use crate::{ + SecretGenerator, hardware::selection::{ HardwareClassSelectionError, SelectedTargetCapabilities, TargetCapabilitySelectionError, select_hardware_class, select_target_capabilities, @@ -53,8 +54,8 @@ use crate::{ /// Only resource estimates keep values a launch would reject, with the reason beside them. pub(crate) struct ResolvedLocalModelValues { pub(crate) selection: LocalModelSelection, - pub(crate) engine_parameter_values: EngineParameterValues, - pub(crate) derived_engine_parameter_values: EngineParameterValues, + pub(crate) engine_parameter_values: ParameterValues, + pub(crate) derived_engine_parameter_values: ParameterValues, pub(crate) engine_parameter_schema: EngineParameterSchema, pub(crate) engine_parameter_schema_id: String, pub(crate) option_values: Vec, @@ -153,7 +154,12 @@ pub(crate) struct LocalModelSelection { /// /// `engine_identity` is the local image ID a caller resolved for a selected tag-only image, which /// may instead be left to the hosting SiteNode. It must name the selected image and is rejected -/// for a digest-pinned image or a Nix artifact. +/// for a digest-pinned image or a Nix artifact. `secrets` says whether the snapshot completes its +/// secret engine parameter values to launch, or withholds them as an advisory snapshot. +#[expect( + clippy::too_many_arguments, + reason = "each input names one independent part of the selection" +)] pub fn resolve_local_launch( catalog: &Catalog, model_id: &CatalogModelId, @@ -162,6 +168,7 @@ pub fn resolve_local_launch( target: &dotcore_proto::dotcore::site::v1::ExecutionTarget, target_inventory: &TargetCapabilityInventory, engine_identity: Option<&EngineLaunchIdentity>, + secrets: EngineSecrets<'_>, ) -> Result { let selected_target = select_target_capabilities(target, target_inventory) .map_err(ModelLaunchResolutionError::TargetCapability)?; @@ -173,11 +180,77 @@ pub fn resolve_local_launch( target, &selected_target, engine_identity, + secrets, ) } +/// What launch resolution does with secret engine parameter values, which sizing never reads. +#[derive(Clone, Copy)] +pub enum EngineSecrets<'a> { + /// Keeps each supplied secret value and generates each missing one, for a snapshot that + /// launches. + Generate(&'a dyn SecretGenerator), + + /// Leaves every secret value out, for an advisory snapshot that never launches. + Withhold, +} + +impl EngineSecrets<'_> { + /// Completes or strips the secret values among resolved `values`. + fn apply( + self, + values: &mut ParameterValues, + schema: &EngineParameterSchema, + ) -> Result<(), ModelLaunchResolutionError> { + let secrets = schema.parameters.iter().filter_map(|definition| { + match definition.parameter_type.as_ref() { + Some(EngineParameterType::Secret(secret)) => Some((definition, secret)), + _ => None, + } + }); + let generator = match self { + Self::Generate(generator) => generator, + Self::Withhold => { + let secret_ids = secrets + .map(|(definition, _)| definition.parameter_id.as_str()) + .collect::>(); + values + .values + .retain(|value| !secret_ids.contains(&value.parameter_id.as_str())); + return Ok(()); + } + }; + for (definition, secret) in secrets { + let parameter_id = &definition.parameter_id; + if values + .values + .iter() + .any(|value| &value.parameter_id == parameter_id) + { + continue; + } + let secret_value = generator + .generate(secret.generated_length as usize) + .map_err(|source| ModelLaunchResolutionError::SecretGeneration { + parameter_id: parameter_id.clone(), + source, + })?; + values.values.push(ParameterValue { + parameter_id: parameter_id.clone(), + value: Some(EngineParameterValueKind::String(secret_value)), + }); + } + Ok(()) + } +} + /// Resolves one local catalog selection into its launch snapshot like [`resolve_local_launch`], -/// on target capabilities the caller already selected. +/// on target capabilities the caller already selected. Secret values are completed after sizing, +/// which never reads them. +#[expect( + clippy::too_many_arguments, + reason = "each input names one independent part of the selection" +)] pub(crate) fn resolve_local_launch_on_selected_target( catalog: &Catalog, model_id: &CatalogModelId, @@ -186,6 +259,7 @@ pub(crate) fn resolve_local_launch_on_selected_target( target: &dotcore_proto::dotcore::site::v1::ExecutionTarget, selected_target: &SelectedTargetCapabilities, engine_identity: Option<&EngineLaunchIdentity>, + secrets: EngineSecrets<'_>, ) -> Result { let resolved = resolve_local_values_for_target( catalog, @@ -196,10 +270,18 @@ pub(crate) fn resolve_local_launch_on_selected_target( )?; let sizing = resolved.resolved_sizing(); let selection = resolved.selection; - let parameter_values = resolved.engine_parameter_values; + let mut parameter_values = resolved.engine_parameter_values; let engine_parameter_schema_id = resolved.engine_parameter_schema_id; let engine_parameter_schema = resolved.engine_parameter_schema; let option_values = resolved.option_values; + secrets.apply(&mut parameter_values, &engine_parameter_schema)?; + let (_, profile_engine_parameter_defaults) = + catalog + .local_defaults_for(model_id, profile_id) + .ok_or_else(|| ModelLaunchResolutionError::ProfileDefaultsUnavailable { + model_id: model_id.0.clone(), + profile_id: profile_id.to_owned(), + })?; let engine_target = selection .engine_target @@ -269,6 +351,10 @@ pub(crate) fn resolve_local_launch_on_selected_target( engine_parameter_values: Some(parameter_values), engine_parameter_schema_id, engine_parameter_schema: Some(engine_parameter_schema), + profile_engine_parameter_defaults: Some(profile_engine_parameter_defaults), + profile_engine_parameter_visibility_overrides: selection + .profile + .engine_parameter_visibility_overrides, desired_launch_plan: None, custom_argument_tokens: configuration.custom_argument_tokens.clone(), resolved_options: option_values @@ -289,10 +375,15 @@ pub(crate) fn resolve_local_launch_on_selected_target( .translate(&local) .map_err(|error| ModelLaunchResolutionError::LaunchPlan(anyhow::Error::new(error)))?, ); - crate::model::resolved::validate_local_snapshot_with( - &local, - dotcore_domain::deployment::artifact::LaunchIdentityResolution::NodeResolved, - ) + let resolution = dotcore_domain::deployment::artifact::LaunchIdentityResolution::NodeResolved; + match secrets { + EngineSecrets::Generate(_) => { + crate::model::resolved::validate_local_snapshot_with(&local, resolution) + } + EngineSecrets::Withhold => { + crate::model::resolved::validate_advisory_local_snapshot(&local, resolution) + } + } .map_err(ModelLaunchResolutionError::CatalogResolution)?; Ok(local) } @@ -753,14 +844,14 @@ fn evaluate_selected_values( ) .map_err(|error| sizing_unavailable(anyhow::Error::new(error)))? .into_iter() - .map(|(parameter_id, value)| EngineParameterValue { + .map(|(parameter_id, value)| ParameterValue { parameter_id, value: Some(EngineParameterValueKind::UnsignedInteger(value)), }) .chain(filled_context_per_request) .collect::>(); derived_values.sort_by(|left, right| left.parameter_id.cmp(&right.parameter_id)); - let derived_engine_parameter_values = EngineParameterValues { + let derived_engine_parameter_values = ParameterValues { values: derived_values, }; validate_derived_ranges(&derived_engine_parameter_values, &schema)?; @@ -769,7 +860,8 @@ fn evaluate_selected_values( .values .extend(derived_engine_parameter_values.values.iter().cloned()); let mut parameter_errors = Vec::new(); - validate_parameter_values( + // Only creation completes secrets, after sizing, so previews and placement never need one. + validate_parameter_values_before_secrets( &complete_values, &selection.engine_release, profile_id, @@ -991,6 +1083,13 @@ pub enum ModelLaunchResolutionError { #[error("could not translate local model launch plan: {0}")] LaunchPlan(#[source] anyhow::Error), + #[error("could not generate secret engine parameter {parameter_id:?}: {source}")] + SecretGeneration { + parameter_id: String, + #[source] + source: anyhow::Error, + }, + #[error( "Engine artifact {reference:?} is digest-pinned or a Nix artifact and takes no resolved launch identity" )] @@ -1055,7 +1154,7 @@ impl ModelLaunchResolutionError { /// Rejects complete engine parameter values, each in range, that a launch can't start together. fn validate_launch_values( selection: &LocalModelSelection, - values: &EngineParameterValues, + values: &ParameterValues, ) -> Result<(), ModelLaunchResolutionError> { validate_context_per_request(selection, values)?; crate::model::engine::validate_parameter_values( @@ -1071,7 +1170,7 @@ fn validate_launch_values( /// maximum context: the model's maximum context, or the configuration's lower maximum. fn validate_context_per_request( selection: &LocalModelSelection, - values: &EngineParameterValues, + values: &ParameterValues, ) -> Result<(), ModelLaunchResolutionError> { let Some(context_tokens) = crate::model::engine::unsigned_parameter_value( values, @@ -1094,7 +1193,7 @@ fn validate_context_per_request( /// Rejects derived values outside their unsigned schema range with the offending value. fn validate_derived_ranges( - derived: &EngineParameterValues, + derived: &ParameterValues, schema: &EngineParameterSchema, ) -> Result<(), ModelLaunchResolutionError> { for value in &derived.values { @@ -1163,10 +1262,9 @@ mod tests { }; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterValue, EngineParameterValues, HardwareRequirements, - ModelOptionParameterValue, ModelOptionValues, NixSystemStorePath, Selector, - StorageBinding, StorageEnvironmentVariable, StorageKind, StorageScope, - engine_parameter_value, model_option_parameter_value, + HardwareRequirements, ModelOptionParameterValue, ModelOptionValues, NixSystemStorePath, + ParameterValue, ParameterValues, Selector, StorageBinding, StorageEnvironmentVariable, + StorageKind, StoragePlacement, model_option_parameter_value, parameter_value, }, engine::{ llama_cpp::v1::LlamaCppResolvedModelState, v1::resolved_engine_model_state::State, @@ -1199,7 +1297,7 @@ mod tests { "fixture-profile", &LocalModelDeploymentConfiguration { model_artifact_binding_id: "fixture-artifact".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: [prompt_cache_value(30_000)] .into_iter() .chain(context_and_request_values()) @@ -1211,6 +1309,7 @@ mod tests { &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("unified snapshot"); @@ -1261,7 +1360,7 @@ mod tests { "fixture-profile", &LocalModelDeploymentConfiguration { model_artifact_binding_id: "fixture-artifact".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: [prompt_cache_value(30_000)] .into_iter() .chain(context_and_request_values()) @@ -1273,6 +1372,9 @@ mod tests { &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate( + &crate::RandomSecretGenerator, + ), ) .expect("snapshot") }; @@ -1295,7 +1397,7 @@ mod tests { "fixture-profile", &LocalModelDeploymentConfiguration { model_artifact_binding_id: "fixture-artifact".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: [prompt_cache_value(30_000)] .into_iter() .chain(context_and_request_values()) @@ -1310,6 +1412,7 @@ mod tests { &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("manager admission"); @@ -1365,7 +1468,7 @@ mod tests { "fixture-profile", &LocalModelDeploymentConfiguration { model_artifact_binding_id: "fixture-artifact".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: [prompt_cache_value(30_000)] .into_iter() .chain(context_and_request_values()) @@ -1376,6 +1479,7 @@ mod tests { &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("snapshot artifact admission"); @@ -1422,7 +1526,7 @@ mod tests { ); let configuration = LocalModelDeploymentConfiguration { model_artifact_binding_id: "sized".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: context_and_request_values(), }), custom_argument_tokens: vec!["--verbose".to_owned()], @@ -1437,13 +1541,14 @@ mod tests { &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("derived snapshot"); let values = resolve_local_values(&catalog, &derived_model_id(), "balanced", &configuration) .expect("derived values"); - let expected_values = EngineParameterValues { + let expected_values = ParameterValues { values: context_and_request_values() .into_iter() .chain([filled_context_per_request(), prompt_cache_value(4096)]) @@ -1467,7 +1572,7 @@ mod tests { assert_eq!(values.engine_parameter_values, expected_values); assert_eq!( values.derived_engine_parameter_values, - EngineParameterValues { + ParameterValues { values: vec![filled_context_per_request(), prompt_cache_value(4096)], } ); @@ -1501,6 +1606,7 @@ mod tests { &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("snapshot on the nvidia class"); let launch = ResolvedHardwareRequirementsValue::single_device( @@ -1573,6 +1679,7 @@ mod tests { &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) }; @@ -1720,7 +1827,7 @@ mod tests { "balanced", &LocalModelDeploymentConfiguration { model_artifact_binding_id: "sized".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: [prompt_cache_value(30_000)] .into_iter() .chain(context_and_request_values()) @@ -1731,12 +1838,13 @@ mod tests { &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("supplied snapshot"); assert_eq!( snapshot.engine_parameter_values, - Some(EngineParameterValues { + Some(ParameterValues { values: [prompt_cache_value(30_000)] .into_iter() .chain(context_and_request_values()) @@ -1772,7 +1880,7 @@ mod tests { "balanced", &LocalModelDeploymentConfiguration { model_artifact_binding_id: "sized".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: context_and_request_values(), }), ..Default::default() @@ -1803,7 +1911,7 @@ mod tests { completed, LocalModelDeploymentConfiguration { model_artifact_binding_id: "sized".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: [prompt_cache_value(30_000)] .into_iter() .chain(context_and_request_values()) @@ -1815,7 +1923,7 @@ mod tests { ); assert_eq!( values.engine_parameter_values, - EngineParameterValues { + ParameterValues { values: [prompt_cache_value(30_000)] .into_iter() .chain(context_and_request_values()) @@ -1825,7 +1933,7 @@ mod tests { ); assert_eq!( values.derived_engine_parameter_values, - EngineParameterValues { + ParameterValues { values: vec![filled_context_per_request()], } ); @@ -1841,7 +1949,7 @@ mod tests { let complete = LocalModelDeploymentConfiguration { model_artifact_binding_id: "second".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: [prompt_cache_value(8)] .into_iter() .chain(context_and_request_values()) @@ -1852,7 +1960,7 @@ mod tests { assert_eq!(completed(complete.clone()), complete); let partial = completed(LocalModelDeploymentConfiguration { - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: vec![prompt_cache_value(8)], }), ..Default::default() @@ -1861,7 +1969,7 @@ mod tests { partial, LocalModelDeploymentConfiguration { model_artifact_binding_id: "sized".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: [prompt_cache_value(8)] .into_iter() .chain(context_and_request_values()) @@ -1874,7 +1982,7 @@ mod tests { resolve_local_values(&catalog, &derived_model_id(), "balanced", &partial) .expect("overlaid values") .derived_engine_parameter_values, - EngineParameterValues { + ParameterValues { values: vec![filled_context_per_request()], } ); @@ -2081,7 +2189,7 @@ mod tests { assert_eq!(values.selection.binding.binding_id, "second"); assert_eq!( values.engine_parameter_values, - EngineParameterValues { + ParameterValues { values: [prompt_cache_value(30_000)] .into_iter() .chain(context_and_request_values()) @@ -2136,7 +2244,7 @@ mod tests { "fixture-profile", &LocalModelDeploymentConfiguration { model_artifact_binding_id: "fixture-artifact".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: [prompt_cache_value(30_000)] .into_iter() .chain(context_and_request_values()) @@ -2154,6 +2262,7 @@ mod tests { &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("disabled vision snapshot"); @@ -2226,6 +2335,7 @@ mod tests { &target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) .expect("digest-pinned snapshot"); @@ -2256,7 +2366,7 @@ mod tests { binding_name: "compile-cache".to_owned(), mount_path: "/cache".to_owned(), kind: StorageKind::Cache as i32, - scope: StorageScope::EngineArtifact as i32, + placements: vec![StoragePlacement::EngineArtifact as i32], environment: vec![ StorageEnvironmentVariable { name: "CACHE_HOME".to_owned(), @@ -2272,7 +2382,7 @@ mod tests { binding_name: "scratch".to_owned(), mount_path: "/scratch".to_owned(), kind: StorageKind::Cache as i32, - scope: StorageScope::Deployment as i32, + placements: vec![StoragePlacement::Deployment as i32], environment: Vec::new(), }, ] @@ -2564,6 +2674,7 @@ mod tests { target, &inventory, None, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) }; @@ -2664,6 +2775,7 @@ mod tests { &target, &inventory, engine_identity, + crate::model::resolution::EngineSecrets::Generate(&crate::RandomSecretGenerator), ) } @@ -2712,7 +2824,7 @@ mod tests { "balanced", &LocalModelDeploymentConfiguration { model_artifact_binding_id: "sized".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: context_and_request_values(), }), ..Default::default() @@ -2799,7 +2911,7 @@ mod tests { fn context_and_request_configuration() -> LocalModelDeploymentConfiguration { LocalModelDeploymentConfiguration { model_artifact_binding_id: "sized".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: context_and_request_values(), }), ..Default::default() @@ -2926,7 +3038,7 @@ mod tests { .iter() .find(|value| value.parameter_id == TOTAL_CONTEXT_PARAMETER_ID) .and_then(|value| match value.value { - Some(engine_parameter_value::Value::UnsignedInteger(value)) => Some(value), + Some(parameter_value::Value::UnsignedInteger(value)) => Some(value), _ => None, }) .expect("resolved total context")) @@ -2969,13 +3081,13 @@ mod tests { root } - fn unsigned_values(values: &[(&str, u64)]) -> EngineParameterValues { - EngineParameterValues { + fn unsigned_values(values: &[(&str, u64)]) -> ParameterValues { + ParameterValues { values: values .iter() - .map(|(parameter_id, value)| EngineParameterValue { + .map(|(parameter_id, value)| ParameterValue { parameter_id: (*parameter_id).to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(*value)), + value: Some(parameter_value::Value::UnsignedInteger(*value)), }) .collect(), } @@ -2985,29 +3097,29 @@ mod tests { CatalogModelId("fixture-model".to_owned()) } - fn prompt_cache_value(value: u64) -> EngineParameterValue { - EngineParameterValue { + fn prompt_cache_value(value: u64) -> ParameterValue { + ParameterValue { parameter_id: "prompt-cache-memory-mib".to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(value)), + value: Some(parameter_value::Value::UnsignedInteger(value)), } } /// The context per request resolution fills from the fixtures' total context. - fn filled_context_per_request() -> EngineParameterValue { - EngineParameterValue { + fn filled_context_per_request() -> ParameterValue { + ParameterValue { parameter_id: "context-per-request".to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(32_768)), + value: Some(parameter_value::Value::UnsignedInteger(32_768)), } } /// Total context and parallel request values the derivation fixtures supply, since their /// sizing derives the prompt cache instead. - fn context_and_request_values() -> Vec { + fn context_and_request_values() -> Vec { [("total-context", 32_768), ("parallel-requests", 4)] .into_iter() - .map(|(parameter_id, value)| EngineParameterValue { + .map(|(parameter_id, value)| ParameterValue { parameter_id: parameter_id.to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(value)), + value: Some(parameter_value::Value::UnsignedInteger(value)), }) .collect() } @@ -3060,7 +3172,7 @@ mod tests { fs::write( &engine_path, format!( - "{engine}storageBindings:\n - bindingName: \"compile-cache\"\n mountPath: \"/cache\"\n kind: \"STORAGE_KIND_CACHE\"\n scope: \"STORAGE_SCOPE_ENGINE_ARTIFACT\"\n environment:\n - name: \"CACHE_HOME\"\n relativePath: \"home\"\n - name: \"CACHE_ROOT\"\n - bindingName: \"scratch\"\n mountPath: \"/scratch\"\n kind: \"STORAGE_KIND_CACHE\"\n scope: \"STORAGE_SCOPE_DEPLOYMENT\"\n" + "{engine}storageBindings:\n - bindingName: \"compile-cache\"\n mountPath: \"/cache\"\n kind: \"STORAGE_KIND_CACHE\"\n placements: [\"STORAGE_PLACEMENT_ENGINE_ARTIFACT\"]\n environment:\n - name: \"CACHE_HOME\"\n relativePath: \"home\"\n - name: \"CACHE_ROOT\"\n - bindingName: \"scratch\"\n mountPath: \"/scratch\"\n kind: \"STORAGE_KIND_CACHE\"\n placements: [\"STORAGE_PLACEMENT_DEPLOYMENT\"]\n" ), ) .expect("engine storage fixture"); diff --git a/backend/core/src/model/resolved.rs b/backend/core/src/model/resolved.rs index cc8fe061..d6d842e4 100644 --- a/backend/core/src/model/resolved.rs +++ b/backend/core/src/model/resolved.rs @@ -5,7 +5,10 @@ use std::collections::{BTreeMap, BTreeSet}; use dotcore_artifact::{ EngineArtifactPreparation, EngineArtifactSource, ModelArtifactSource, OciPlatform, }; -use dotcore_catalog::{CatalogError, CatalogModelId, validate_parameter_values}; +use dotcore_catalog::{ + CatalogError, CatalogModelId, typed_parameters::is_secret, validate_parameter_values, + validate_parameter_values_before_secrets, +}; use dotcore_domain::{ MemoryArchitecture, deployment::artifact::{ @@ -16,10 +19,11 @@ use dotcore_domain::{ }; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineArtifact, EngineParameterValues, EngineRelease, HardwareRequirements, - HuggingFaceArtifactSource, ModelInputCapabilities, StorageBinding, + EngineArtifact, EngineRelease, HardwareRequirements, HuggingFaceArtifactSource, + ModelInputCapabilities, ParameterValues, StorageBinding, engine_artifact::Kind as EngineArtifactKind, hugging_face_artifact_source::Content as HuggingFaceContent, nix_artifact::Provides, + parameter_value::Value as EngineParameterValueKind, }, deployment::v1::ArtifactPreparationPolicy, model::v1::{ @@ -197,7 +201,17 @@ impl ResolvedModelLaunch { self.engine.launch_identity.clone() } }, - engine_parameter_values: snapshot.engine_parameter_values.clone(), + // Sizing reads integers only, so records leave secret values out. + engine_parameter_values: snapshot.engine_parameter_values.as_ref().map(|values| { + ParameterValues { + values: values + .values + .iter() + .filter(|value| !is_secret_value(snapshot, &value.parameter_id)) + .cloned() + .collect(), + } + }), option_values: snapshot .resolved_options .iter() @@ -458,6 +472,54 @@ pub(crate) fn validate_local_snapshot( pub(crate) fn validate_local_snapshot_with( local: &ResolvedLocalModelDeploymentSnapshot, resolution: LaunchIdentityResolution, +) -> Result<(), CatalogError> { + validate_local_snapshot_values(local, resolution, SecretValues::Retained) +} + +/// Validates an advisory local snapshot like [`validate_local_snapshot_with`], but one that +/// withholds every secret engine parameter value, since it never launches. +pub(crate) fn validate_advisory_local_snapshot( + local: &ResolvedLocalModelDeploymentSnapshot, + resolution: LaunchIdentityResolution, +) -> Result<(), CatalogError> { + validate_local_snapshot_values(local, resolution, SecretValues::Withheld) +} + +/// The secret engine parameter values a local deployment retains, which logs, failure text and +/// telemetry never show; none for an upstream deployment. +pub(crate) fn secret_engine_parameter_values( + declaration: &ModelDeploymentDeclaration, +) -> Vec { + let Some(Source::Local(local)) = declaration + .snapshot + .as_ref() + .and_then(|snapshot| snapshot.source.as_ref()) + else { + return Vec::new(); + }; + local + .engine_parameter_values + .iter() + .flat_map(|values| &values.values) + .filter(|value| is_secret_value(local, &value.parameter_id)) + .filter_map(|value| match value.value.as_ref() { + Some(EngineParameterValueKind::String(secret)) => Some(secret.clone()), + _ => None, + }) + .collect() +} + +/// Whether a snapshot's secret engine parameter values are retained or withheld. +#[derive(Clone, Copy, PartialEq, Eq)] +enum SecretValues { + Retained, + Withheld, +} + +fn validate_local_snapshot_values( + local: &ResolvedLocalModelDeploymentSnapshot, + resolution: LaunchIdentityResolution, + secrets: SecretValues, ) -> Result<(), CatalogError> { if local.model_id.is_empty() || local.configuration_id.is_empty() @@ -489,16 +551,30 @@ pub(crate) fn validate_local_snapshot_with( .as_ref() .ok_or_else(|| invalid("local deployment snapshot has no engine parameter values"))?; let mut parameter_errors = Vec::new(); - validate_parameter_values( - values, - &EngineRelease { - parameter_schema: Some(schema.clone()), - ..Default::default() - }, - &local.profile_id, - &mut parameter_errors, - ); - if parameter_errors.into_iter().next().is_some() { + let engine_release = EngineRelease { + parameter_schema: Some(schema.clone()), + ..Default::default() + }; + match secrets { + SecretValues::Retained => validate_parameter_values( + values, + &engine_release, + &local.profile_id, + &mut parameter_errors, + ), + SecretValues::Withheld => validate_parameter_values_before_secrets( + values, + &engine_release, + &local.profile_id, + &mut parameter_errors, + ), + } + let withheld_secret = secrets == SecretValues::Withheld + && values + .values + .iter() + .any(|value| is_secret_value(local, &value.parameter_id)); + if parameter_errors.into_iter().next().is_some() || withheld_secret { return Err(invalid( "local deployment snapshot has invalid engine parameter values", )); @@ -621,10 +697,20 @@ pub(crate) fn selected_target_memory_architecture( ) } +/// Whether the snapshot's retained engine parameter schema declares `parameter_id` as a secret +/// parameter, whose value logs, failure text and telemetry never show. +fn is_secret_value(local: &ResolvedLocalModelDeploymentSnapshot, parameter_id: &str) -> bool { + local + .engine_parameter_schema + .iter() + .flat_map(|schema| &schema.parameters) + .any(|definition| definition.parameter_id == parameter_id && is_secret(definition)) +} + /// Requires every derived parameter id to name one retained engine parameter value. fn validate_sizing( sizing: Option<&ResolvedSizing>, - values: &EngineParameterValues, + values: &ParameterValues, ) -> Result<(), CatalogError> { let Some(sizing) = sizing else { return Ok(()); diff --git a/backend/core/src/model/sizing_scenario/mod.rs b/backend/core/src/model/sizing_scenario/mod.rs index 8969b702..e6d31163 100644 --- a/backend/core/src/model/sizing_scenario/mod.rs +++ b/backend/core/src/model/sizing_scenario/mod.rs @@ -19,8 +19,8 @@ use dotcore_catalog::{ }; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterSchema, EngineParameterValue, EngineParameterValues, HardwareRequirements, - Model, ModelProfile, engine_parameter_value::Value as EngineParameterValueKind, + EngineParameterSchema, HardwareRequirements, Model, ModelProfile, ParameterValue, + ParameterValues, parameter_value::Value as EngineParameterValueKind, }, deployment::v1::MemoryFit, model::v1::{ @@ -406,7 +406,7 @@ fn largest_fitting_total_context( set_unsigned_value( configuration .engine_parameter_values - .get_or_insert_with(EngineParameterValues::default), + .get_or_insert_with(ParameterValues::default), TOTAL_CONTEXT_PARAMETER_ID, total_context, ); @@ -559,7 +559,7 @@ fn with_scenario_values( let values = configuration .engine_parameter_values - .get_or_insert_with(EngineParameterValues::default); + .get_or_insert_with(ParameterValues::default); set_unsigned_value( values, TOTAL_CONTEXT_PARAMETER_ID, @@ -579,8 +579,8 @@ fn with_scenario_values( /// values are the ones a launch would resolve before sizing derivations. fn with_filled_context_per_request( declarations: &CatalogLocalModelDeclarations, - mut values: EngineParameterValues, -) -> EngineParameterValues { + mut values: ParameterValues, +) -> ParameterValues { if let Some(schema) = engine_parameter_schema(declarations) { let filled = crate::model::engine::filled_context_per_request( declarations.engine_kind, @@ -599,7 +599,7 @@ fn with_filled_context_per_request( fn accepts_context_values( declarations: &CatalogLocalModelDeclarations, schema: &EngineParameterSchema, - complete: &EngineParameterValues, + complete: &ParameterValues, ) -> Option { let mut in_range = true; for parameter_id in [TOTAL_CONTEXT_PARAMETER_ID, CONTEXT_PER_REQUEST_PARAMETER_ID] { @@ -658,8 +658,8 @@ fn engine_parameter_schema( } /// Replaces the value of one parameter, or adds it when the values leave it out. -fn set_unsigned_value(values: &mut EngineParameterValues, parameter_id: &str, value: u64) { - let value = EngineParameterValue { +fn set_unsigned_value(values: &mut ParameterValues, parameter_id: &str, value: u64) { + let value = ParameterValue { parameter_id: parameter_id.to_owned(), value: Some(EngineParameterValueKind::UnsignedInteger(value)), }; diff --git a/backend/core/src/model/sizing_scenario/tests.rs b/backend/core/src/model/sizing_scenario/tests.rs index 5ff3346a..b685d0e3 100644 --- a/backend/core/src/model/sizing_scenario/tests.rs +++ b/backend/core/src/model/sizing_scenario/tests.rs @@ -1491,10 +1491,10 @@ version: "1" parameterSchema: schemaId: "llama-v2" parameters: - - { parameterId: "total-context", displayName: "Total context", exposed: true, unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } } - - { parameterId: "context-per-request", displayName: "Context per request", exposed: true, unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens" } } - - { parameterId: "parallel-requests", displayName: "Parallel requests", exposed: true, unsignedInteger: { minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" } } - - { parameterId: "prompt-cache-memory-mib", displayName: "Prompt cache memory", unsignedInteger: { minimum: "0", maximum: "65536", unit: "MiB" } } + - { parameterId: "total-context", displayName: "Total context", visibility: "PARAMETER_VISIBILITY_BASIC", unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } } + - { parameterId: "context-per-request", displayName: "Context per request", visibility: "PARAMETER_VISIBILITY_BASIC", unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens" } } + - { parameterId: "parallel-requests", displayName: "Parallel requests", visibility: "PARAMETER_VISIBILITY_ADVANCED", unsignedInteger: { minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" } } + - { parameterId: "prompt-cache-memory-mib", displayName: "Prompt cache memory", visibility: "PARAMETER_VISIBILITY_HIDDEN", unsignedInteger: { minimum: "0", maximum: "65536", unit: "MiB" } } artifacts: - artifactId: "linux" oci: @@ -1601,13 +1601,13 @@ version: "1" parameterSchema: schemaId: "vllm-v2" parameters: - - { parameterId: "total-context", displayName: "Total context", exposed: true, unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } } - - { parameterId: "context-per-request", displayName: "Context per request", exposed: true, unsignedInteger: { minimum: "1024", maximum: "1048576", unit: "tokens" } } - - { parameterId: "parallel-requests", displayName: "Parallel requests", exposed: true, unsignedInteger: { minimum: "1", maximum: "1024", unit: "requests", defaultValue: "4" } } - - { parameterId: "maximum-batched-tokens", displayName: "Maximum batched tokens", unsignedInteger: { minimum: "1", maximum: "1048576", unit: "tokens" } } - - { parameterId: "maximum-cudagraph-capture-size", displayName: "Maximum CUDA graph capture size", unsignedInteger: { minimum: "1", maximum: "1024", unit: "sequences" } } - - { parameterId: "kv-cache-memory-bytes", displayName: "KV cache memory", unsignedInteger: { minimum: "1", maximum: "68719476736", unit: "bytes" } } - - { parameterId: "kv-offloading-size-gib", displayName: "KV offloading size", unsignedInteger: { minimum: "0", maximum: "1024", unit: "GiB" } } + - { parameterId: "total-context", displayName: "Total context", visibility: "PARAMETER_VISIBILITY_BASIC", unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } } + - { parameterId: "context-per-request", displayName: "Context per request", visibility: "PARAMETER_VISIBILITY_BASIC", unsignedInteger: { minimum: "1024", maximum: "1048576", unit: "tokens" } } + - { parameterId: "parallel-requests", displayName: "Parallel requests", visibility: "PARAMETER_VISIBILITY_ADVANCED", unsignedInteger: { minimum: "1", maximum: "1024", unit: "requests", defaultValue: "4" } } + - { parameterId: "maximum-batched-tokens", displayName: "Maximum batched tokens", visibility: "PARAMETER_VISIBILITY_HIDDEN", unsignedInteger: { minimum: "1", maximum: "1048576", unit: "tokens" } } + - { parameterId: "maximum-cudagraph-capture-size", displayName: "Maximum CUDA graph capture size", visibility: "PARAMETER_VISIBILITY_HIDDEN", unsignedInteger: { minimum: "1", maximum: "1024", unit: "sequences" } } + - { parameterId: "kv-cache-memory-bytes", displayName: "KV cache memory", visibility: "PARAMETER_VISIBILITY_HIDDEN", unsignedInteger: { minimum: "1", maximum: "68719476736", unit: "bytes" } } + - { parameterId: "kv-offloading-size-gib", displayName: "KV offloading size", visibility: "PARAMETER_VISIBILITY_HIDDEN", unsignedInteger: { minimum: "0", maximum: "1024", unit: "GiB" } } artifacts: - artifactId: "linux" oci: @@ -1695,10 +1695,10 @@ version: "1" parameterSchema: schemaId: "ninfer-v2" parameters: - - { parameterId: "total-context", displayName: "Total context", exposed: true, unsignedInteger: { minimum: "1024", maximum: "2097152", unit: "tokens", defaultValue: "32768" } } - - { parameterId: "context-per-request", displayName: "Context per request", exposed: true, unsignedInteger: { minimum: "1024", maximum: "262144", unit: "tokens" } } - - { parameterId: "parallel-requests", displayName: "Parallel requests", exposed: true, unsignedInteger: { minimum: "1", maximum: "8", unit: "requests", defaultValue: "4" } } - - { parameterId: "host-context-mib", displayName: "Host context memory", unsignedInteger: { minimum: "0", maximum: "1048576", unit: "MiB" } } + - { parameterId: "total-context", displayName: "Total context", visibility: "PARAMETER_VISIBILITY_BASIC", unsignedInteger: { minimum: "1024", maximum: "2097152", unit: "tokens", defaultValue: "32768" } } + - { parameterId: "context-per-request", displayName: "Context per request", visibility: "PARAMETER_VISIBILITY_BASIC", unsignedInteger: { minimum: "1024", maximum: "262144", unit: "tokens" } } + - { parameterId: "parallel-requests", displayName: "Parallel requests", visibility: "PARAMETER_VISIBILITY_ADVANCED", unsignedInteger: { minimum: "1", maximum: "8", unit: "requests", defaultValue: "4" } } + - { parameterId: "host-context-mib", displayName: "Host context memory", visibility: "PARAMETER_VISIBILITY_HIDDEN", unsignedInteger: { minimum: "0", maximum: "1048576", unit: "MiB" } } artifacts: - artifactId: "linux" oci: diff --git a/backend/core/src/route_manager/projection.rs b/backend/core/src/route_manager/projection.rs index 38dd58a6..5e33261b 100644 --- a/backend/core/src/route_manager/projection.rs +++ b/backend/core/src/route_manager/projection.rs @@ -248,7 +248,7 @@ fn modalities(supports_text: bool, supports_image: bool) -> Vec { #[cfg(test)] mod tests { use dotcore_proto::dotcore::{ - catalog::v1::engine_parameter_value, + catalog::v1::parameter_value, inference::v1::{FallbackGroup, FallbackGroupSelection, InferenceRoute}, model::v1::ModelDeploymentSnapshot, }; @@ -394,7 +394,7 @@ mod tests { .expect("fixture parameter values"); for value in &mut values.values { if value.parameter_id == CONTEXT_PER_REQUEST_PARAMETER_ID { - value.value = Some(engine_parameter_value::Value::UnsignedInteger(context)); + value.value = Some(parameter_value::Value::UnsignedInteger(context)); } } ModelDeploymentDeclaration { diff --git a/backend/core/src/secret_generator.rs b/backend/core/src/secret_generator.rs new file mode 100644 index 00000000..3195c0fc --- /dev/null +++ b/backend/core/src/secret_generator.rs @@ -0,0 +1,71 @@ +//! Random values for deployment secret parameters. +//! +//! Application and model deployments generate a value for each secret parameter an operator +//! leaves out when the deployment is created. Generators are injected, so tests can predict the +//! values, while production draws them from the operating system's cryptographic random source. + +/// Source of the random values generated for secret parameters an operator leaves out. +pub trait SecretGenerator: Send + Sync { + /// Returns `length` characters drawn uniformly from ASCII letters and digits. + fn generate(&self, length: usize) -> anyhow::Result; +} + +/// Generates secret values from the operating system's cryptographic random source. +#[derive(Clone, Copy, Debug, Default)] +pub struct RandomSecretGenerator; + +impl SecretGenerator for RandomSecretGenerator { + fn generate(&self, length: usize) -> anyhow::Result { + // 248 is the largest multiple of the alphabet size within a byte; rejecting bytes above + // it keeps every character equally likely. + const ACCEPTED_BYTES: u8 = + (u8::MAX / SECRET_ALPHABET.len() as u8) * SECRET_ALPHABET.len() as u8; + let mut value = String::with_capacity(length); + let mut bytes = [0_u8; 64]; + while value.len() < length { + getrandom::fill(&mut bytes).map_err(|source| { + anyhow::Error::new(EntropyError(source)).context("Secret entropy failed") + })?; + value.extend( + bytes + .iter() + .filter(|byte| **byte < ACCEPTED_BYTES) + .map(|byte| { + char::from(SECRET_ALPHABET[usize::from(*byte) % SECRET_ALPHABET.len()]) + }) + .take(length - value.len()), + ); + } + Ok(value) + } +} + +/// Characters a generated secret is drawn from: safe in environment values, URLs and shells. +const SECRET_ALPHABET: &[u8; 62] = + b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"; + +#[derive(Debug, thiserror::Error)] +#[error("{0}")] +struct EntropyError(getrandom::Error); + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn random_secrets_have_the_requested_length_from_the_alphanumeric_alphabet() { + let generator = RandomSecretGenerator; + for length in [1, 32, 200] { + let value = generator.generate(length).expect("generated secret"); + assert_eq!(value.len(), length); + assert!( + value.bytes().all(|byte| SECRET_ALPHABET.contains(&byte)), + "{value}" + ); + } + assert_ne!( + generator.generate(32).expect("first secret"), + generator.generate(32).expect("second secret") + ); + } +} diff --git a/backend/core/src/storage.rs b/backend/core/src/storage.rs index 1c47ef97..5cb8580a 100644 --- a/backend/core/src/storage.rs +++ b/backend/core/src/storage.rs @@ -1,19 +1,20 @@ //! Turns catalog storage bindings into the storage attachments and environment of one workload. //! -//! Engines and applications declare writable storage the same way, and each owner only says which -//! scopes it can resolve and to which key. This module picks the namespace of each scope, so keys -//! of different scopes never share a folder, and fails instead of overriding an environment -//! variable the workload already sets. +//! Engines and applications declare writable storage the same way. An application deployment +//! chooses each binding's placement at creation; an engine binding has one fixed placement. Each +//! owner only says which placements it can resolve and to which key. This module picks the +//! namespace of each placement, so keys of different placements never share a folder, and fails +//! instead of overriding an environment variable the workload already sets. -use dotcore_proto::dotcore::catalog::v1::{StorageBinding, StorageKind, StorageScope}; +use dotcore_proto::dotcore::catalog::v1::{StorageBinding, StorageKind, StoragePlacement}; use dotcore_runtime::{WorkloadStorageAttachment, WorkloadStorageKey, WorkloadStorageKind}; -/// The runtime namespace of storage scoped to one deployment. +/// The runtime namespace of storage placed with one deployment. pub(crate) const DEPLOYMENT_NAMESPACE: &str = "deployment"; -/// Returns the storage one deployment-scoped binding of the given kind attaches for one owner key. +/// Returns the storage one deployment-placed binding of the given kind attaches for one owner key. /// -/// Launch attaches deployment-scoped bindings under this key and teardown removes it, so both +/// Launch attaches deployment-placed bindings under this key and teardown removes it, so both /// name the same folder. pub(crate) fn deployment_storage(kind: WorkloadStorageKind, key: &str) -> WorkloadStorageKey { WorkloadStorageKey { @@ -23,17 +24,19 @@ pub(crate) fn deployment_storage(kind: WorkloadStorageKind, key: &str) -> Worklo } } -/// Returns the storage keys launch attaches for the deployment-scoped bindings among `bindings` -/// under one owner key, one per storage kind, so teardown removes exactly those folders. +/// Returns the storage keys launch attaches for the deployment-placed engine bindings among +/// `bindings` under one owner key, one per storage kind, so teardown removes exactly those folders. pub(crate) fn deployment_storage_keys<'a>( bindings: impl IntoIterator, key: &str, ) -> Result, StorageBindingError> { - Ok(bindings - .into_iter() - .filter(|binding| binding.scope == StorageScope::Deployment as i32) - .map(storage_kind) - .collect::, _>>()? + let mut kinds = std::collections::BTreeSet::new(); + for binding in bindings { + if engine_placement(binding)? == StoragePlacement::Deployment { + kinds.insert(storage_kind(binding)?); + } + } + Ok(kinds .into_iter() .map(|kind| deployment_storage(kind, key)) .collect()) @@ -52,8 +55,11 @@ pub(crate) enum StorageBindingError { #[error("storage binding {binding:?} has no usable kind")] InvalidKind { binding: String }, - #[error("storage binding {binding:?} uses a scope its owner cannot resolve")] - UnsupportedScope { binding: String }, + #[error("storage binding {binding:?} uses a placement its owner cannot resolve")] + UnsupportedPlacement { binding: String }, + + #[error("engine storage binding {binding:?} does not declare exactly one placement")] + EnginePlacement { binding: String }, #[error( "storage binding {binding:?} sets environment variable {variable:?}, which the workload already sets" @@ -61,24 +67,24 @@ pub(crate) enum StorageBindingError { EnvironmentAlreadySet { binding: String, variable: String }, } -/// Maps each `(attachment name, binding)` pair to a storage attachment and its environment. +/// Maps each `(attachment name, binding, placement)` triple to a storage attachment and its +/// environment. /// -/// `scope_key` returns the owner's key for a scope, or `None` for a scope it cannot resolve; -/// `is_set` tells whether the workload already sets an environment variable. +/// `placement_key` returns the owner's key for a placement, or `None` for a placement it cannot +/// resolve; `is_set` tells whether the workload already sets an environment variable. pub(crate) fn workload_storage<'a>( - bindings: impl IntoIterator, - scope_key: impl Fn(StorageScope) -> Option, + bindings: impl IntoIterator, + placement_key: impl Fn(StoragePlacement) -> Option, is_set: impl Fn(&str) -> bool, ) -> Result { let mut storage = WorkloadStorage::default(); - for (name, binding) in bindings { + for (name, binding, placement) in bindings { let kind = storage_kind(binding)?; - let unsupported = || StorageBindingError::UnsupportedScope { + let unsupported = || StorageBindingError::UnsupportedPlacement { binding: binding.binding_name.clone(), }; - let scope = StorageScope::try_from(binding.scope).map_err(|_| unsupported())?; - let namespace = scope_namespace(scope).ok_or_else(unsupported)?; - let key = scope_key(scope).ok_or_else(unsupported)?; + let namespace = placement_namespace(placement).ok_or_else(unsupported)?; + let key = placement_key(placement).ok_or_else(unsupported)?; for variable in &binding.environment { if is_set(&variable.name) { @@ -102,12 +108,29 @@ pub(crate) fn workload_storage<'a>( namespace: namespace.to_owned(), key, read_only: false, - expires_when_unused: scope != StorageScope::Deployment, + expires_when_unused: placement != StoragePlacement::Deployment, }); } Ok(storage) } +/// Returns the one placement an engine storage binding declares, which every deployment of the +/// engine uses. +pub(crate) fn engine_placement( + binding: &StorageBinding, +) -> Result { + match binding.placements.as_slice() { + [placement] => StoragePlacement::try_from(*placement).map_err(|_| { + StorageBindingError::UnsupportedPlacement { + binding: binding.binding_name.clone(), + } + }), + _ => Err(StorageBindingError::EnginePlacement { + binding: binding.binding_name.clone(), + }), + } +} + /// Returns the runtime storage kind one binding attaches. fn storage_kind(binding: &StorageBinding) -> Result { match StorageKind::try_from(binding.kind) { @@ -119,13 +142,14 @@ fn storage_kind(binding: &StorageBinding) -> Result Option<&'static str> { - match scope { - StorageScope::Deployment => Some(DEPLOYMENT_NAMESPACE), - StorageScope::Application => Some("application"), - StorageScope::EngineArtifact => Some("engine-artifact"), - StorageScope::Unspecified => None, +/// Returns the runtime namespace that keeps one placement's keys apart from every other +/// placement's. +fn placement_namespace(placement: StoragePlacement) -> Option<&'static str> { + match placement { + StoragePlacement::Deployment => Some(DEPLOYMENT_NAMESPACE), + StoragePlacement::Application => Some("application"), + StoragePlacement::EngineArtifact => Some("engine-artifact"), + StoragePlacement::Unspecified => None, } } @@ -137,11 +161,17 @@ mod tests { #[test] fn maps_a_binding_to_its_attachment_and_environment() { - let binding = binding(StorageScope::Deployment); + let binding = binding(StoragePlacement::Deployment); let storage = workload_storage( - [("engine-cache".to_owned(), &binding)], - |scope| (scope == StorageScope::Deployment).then(|| "deployment-one".to_owned()), + [( + "engine-cache".to_owned(), + &binding, + StoragePlacement::Deployment, + )], + |placement| { + (placement == StoragePlacement::Deployment).then(|| "deployment-one".to_owned()) + }, |_| false, ) .expect("storage"); @@ -171,10 +201,14 @@ mod tests { #[test] fn rejects_an_environment_variable_the_workload_already_sets() { - let binding = binding(StorageScope::Deployment); + let binding = binding(StoragePlacement::Deployment); let error = workload_storage( - [("engine-cache".to_owned(), &binding)], + [( + "engine-cache".to_owned(), + &binding, + StoragePlacement::Deployment, + )], |_| Some("deployment-one".to_owned()), |name| name == "TRITON_CACHE", ) @@ -187,30 +221,94 @@ mod tests { } #[test] - fn rejects_a_scope_the_owner_cannot_resolve() { - for scope in [StorageScope::Application, StorageScope::Unspecified] { - let binding = binding(scope); + fn maps_each_application_placement_to_its_namespace_key_and_lifetime() { + let binding = binding(StoragePlacement::Application); + let attach = |placement| { + workload_storage( + [("web-models".to_owned(), &binding, placement)], + |placement| match placement { + StoragePlacement::Deployment => Some("deployment-one".to_owned()), + StoragePlacement::Application => Some("comfyui".to_owned()), + _ => None, + }, + |_| false, + ) + .expect("storage") + .attachments + .into_iter() + .map(|attachment| { + ( + attachment.namespace, + attachment.key, + attachment.expires_when_unused, + ) + }) + .collect::>() + }; + assert_eq!( + attach(StoragePlacement::Deployment), + [("deployment".to_owned(), "deployment-one".to_owned(), false)] + ); + assert_eq!( + attach(StoragePlacement::Application), + [("application".to_owned(), "comfyui".to_owned(), true)] + ); + } + + #[test] + fn rejects_a_placement_the_owner_cannot_resolve() { + let binding = binding(StoragePlacement::Deployment); + for placement in [StoragePlacement::Application, StoragePlacement::Unspecified] { let error = workload_storage( - [("engine-cache".to_owned(), &binding)], - |scope| (scope == StorageScope::Deployment).then(|| "deployment-one".to_owned()), + [("engine-cache".to_owned(), &binding, placement)], + |placement| { + (placement == StoragePlacement::Deployment).then(|| "deployment-one".to_owned()) + }, |_| false, ) - .expect_err("unsupported scope"); + .expect_err("unsupported placement"); assert!( - matches!(error, StorageBindingError::UnsupportedScope { ref binding } if binding == "engine-cache"), + matches!(error, StorageBindingError::UnsupportedPlacement { ref binding } if binding == "engine-cache"), "{error:?}" ); } } - fn binding(scope: StorageScope) -> StorageBinding { + #[test] + fn deployment_storage_keys_follow_the_single_engine_placement() { + let deployment = binding(StoragePlacement::Deployment); + let shared = binding(StoragePlacement::EngineArtifact); + + assert_eq!( + deployment_storage_keys([&deployment, &shared], "deployment-one").expect("keys"), + [deployment_storage( + WorkloadStorageKind::Cache, + "deployment-one" + )] + ); + + let several = StorageBinding { + placements: vec![ + StoragePlacement::Deployment as i32, + StoragePlacement::EngineArtifact as i32, + ], + ..binding(StoragePlacement::Deployment) + }; + let error = deployment_storage_keys([&several], "deployment-one").expect_err("several"); + assert!( + matches!(error, StorageBindingError::EnginePlacement { ref binding } if binding == "engine-cache"), + "{error:?}" + ); + } + + fn binding(placement: StoragePlacement) -> StorageBinding { StorageBinding { binding_name: "engine-cache".to_owned(), mount_path: "/var/cache/engine".to_owned(), kind: StorageKind::Cache as i32, - scope: scope as i32, + placements: vec![placement as i32], environment: vec![ StorageEnvironmentVariable { name: "CACHE_HOME".to_owned(), diff --git a/backend/core/src/telemetry/fit/mod.rs b/backend/core/src/telemetry/fit/mod.rs index 1f92c589..ee677ef2 100644 --- a/backend/core/src/telemetry/fit/mod.rs +++ b/backend/core/src/telemetry/fit/mod.rs @@ -26,9 +26,7 @@ use dotcore_catalog::{ shared_parameters::TOTAL_CONTEXT_PARAMETER_ID, }; use dotcore_proto::dotcore::{ - catalog::v1::{ - EngineParameterValues, ModelEngineConfiguration, ModelOptionValues, engine_parameter_value, - }, + catalog::v1::{ModelEngineConfiguration, ModelOptionValues, ParameterValues, parameter_value}, telemetry::v1::{ VllmStartupFacts, WorkloadResourceObservationKind, WorkloadResourceRecord, workload_resource_observation::EngineFacts, @@ -134,7 +132,7 @@ impl ObservedQuantity { .iter() .find(|value| &value.parameter_id == parameter) .and_then(|value| match value.value { - Some(engine_parameter_value::Value::UnsignedInteger(value)) => Some(value), + Some(parameter_value::Value::UnsignedInteger(value)) => Some(value), _ => None, }), } @@ -934,7 +932,7 @@ fn predict( fn record_inputs( configuration: &ModelEngineConfiguration, record: &WorkloadResourceRecord, -) -> (EngineParameterValues, Vec) { +) -> (ParameterValues, Vec) { let (engine_values, mut option_values) = record .inputs .as_ref() @@ -1113,7 +1111,7 @@ fn total_context(record: &WorkloadResourceRecord) -> Option { .iter() .find(|value| value.parameter_id == TOTAL_CONTEXT_PARAMETER_ID) .and_then(|value| match value.value { - Some(engine_parameter_value::Value::UnsignedInteger(value)) => Some(value), + Some(parameter_value::Value::UnsignedInteger(value)) => Some(value), _ => None, }) } diff --git a/backend/core/src/telemetry/fit/tests.rs b/backend/core/src/telemetry/fit/tests.rs index c1bb56b8..8ae3e18d 100644 --- a/backend/core/src/telemetry/fit/tests.rs +++ b/backend/core/src/telemetry/fit/tests.rs @@ -5,7 +5,7 @@ use dotcore_catalog::test_support::{ use dotcore_proto::{ dotcore::{ catalog::v1::{ - EngineParameterValue, ModelOptionParameterValue, ModelOptionValues, + ModelOptionParameterValue, ModelOptionValues, ParameterValue, model_option_parameter_value, }, telemetry::v1::{ @@ -1466,10 +1466,10 @@ fn record( model_artifact_id: "sized-weights".to_owned(), engine_release_id: "fixture-engine".to_owned(), engine_artifact_id: "linux-amd64".to_owned(), - engine_parameter_values: Some(EngineParameterValues { - values: vec![EngineParameterValue { + engine_parameter_values: Some(ParameterValues { + values: vec![ParameterValue { parameter_id: "total-context".to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(context_size)), + value: Some(parameter_value::Value::UnsignedInteger(context_size)), }], }), ..WorkloadResourceInputs::default() diff --git a/backend/core/src/telemetry/mod.rs b/backend/core/src/telemetry/mod.rs index d8fbb3bb..bf453c7a 100644 --- a/backend/core/src/telemetry/mod.rs +++ b/backend/core/src/telemetry/mod.rs @@ -187,9 +187,9 @@ mod tests { use dotcore_proto::{ dotcore::{ catalog::v1::{ - EngineParameterValue, EngineParameterValues, ModelOptionParameterValue, - ModelOptionValues, engine_parameter_value::Value as EngineValue, + ModelOptionParameterValue, ModelOptionValues, ParameterValue, ParameterValues, model_option_parameter_value::Value as OptionValue, + parameter_value::Value as EngineValue, }, observation::v1::AcceleratorMemoryArchitecture, telemetry::v1::{ @@ -339,13 +339,13 @@ mod tests { engine_release_id: "vllm-v0-32-0".to_owned(), engine_artifact_id: "linux-amd64-cuda".to_owned(), engine_artifact_reference: "docker.io/vllm/vllm-openai@sha256:d392".to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: vec![ - EngineParameterValue { + ParameterValue { parameter_id: "kv-cache-memory-bytes".to_owned(), value: Some(EngineValue::UnsignedInteger(6_000_000_000)), }, - EngineParameterValue { + ParameterValue { parameter_id: "prefix-caching".to_owned(), value: Some(EngineValue::Boolean(false)), }, diff --git a/backend/core/src/test_support/mod.rs b/backend/core/src/test_support/mod.rs index 1eca75c4..54f016c4 100644 --- a/backend/core/src/test_support/mod.rs +++ b/backend/core/src/test_support/mod.rs @@ -37,15 +37,15 @@ pub(crate) fn host_process_load_guard() -> std::sync::MutexGuard<'static, ()> { /// Unsigned engine parameter values for the given parameter and value pairs, in order. pub(crate) fn unsigned_parameter_values<'a>( values: impl IntoIterator, -) -> dotcore_proto::dotcore::catalog::v1::EngineParameterValues { +) -> dotcore_proto::dotcore::catalog::v1::ParameterValues { use dotcore_proto::dotcore::catalog::v1::{ - EngineParameterValue, EngineParameterValues, engine_parameter_value::Value, + ParameterValue, ParameterValues, parameter_value::Value, }; - EngineParameterValues { + ParameterValues { values: values .into_iter() - .map(|(parameter_id, value)| EngineParameterValue { + .map(|(parameter_id, value)| ParameterValue { parameter_id: parameter_id.to_owned(), value: Some(Value::UnsignedInteger(value)), }) @@ -276,3 +276,29 @@ pub(crate) fn routed_site(deployment_ids: &[&str]) -> dotcore_proto::dotcore::si pub(crate) fn site_declaration_rules() -> SiteDeclarationRules { crate::model::site_declaration_rules() } + +/// A secret generator whose values tests can predict: the `n`th call returns `secret` padded +/// with `x` to the requested length. +pub(crate) fn test_secret_generator() -> std::sync::Arc { + std::sync::Arc::new(SequentialSecretGenerator::default()) +} + +/// The `n`th value [`test_secret_generator`] returns at `length`, counting from 1. +pub(crate) fn sequential_secret(n: usize, length: usize) -> String { + let mut value = format!("secret{n}"); + value.extend(std::iter::repeat_n('x', length.saturating_sub(value.len()))); + value.truncate(length); + value +} + +#[derive(Default)] +struct SequentialSecretGenerator { + calls: std::sync::atomic::AtomicUsize, +} + +impl crate::SecretGenerator for SequentialSecretGenerator { + fn generate(&self, length: usize) -> anyhow::Result { + let n = self.calls.fetch_add(1, std::sync::atomic::Ordering::SeqCst) + 1; + Ok(sequential_secret(n, length)) + } +} diff --git a/backend/core/src/test_support/ninfer.rs b/backend/core/src/test_support/ninfer.rs index db974f00..bca2ced1 100644 --- a/backend/core/src/test_support/ninfer.rs +++ b/backend/core/src/test_support/ninfer.rs @@ -14,18 +14,18 @@ use dotcore_catalog::{ }; use dotcore_proto::dotcore::{ catalog::v1::{ - BooleanModelOptionParameterSchema, EngineParameterDefinition, EngineParameterSchema, - EngineParameterValue, EngineParameterValues, EngineRelease, HardwareRequirements, - HardwareSupport, HuggingFaceArtifactSource, HuggingFaceFileArtifact, InferenceRequestKind, - Model, ModelArtifact, ModelArtifactBinding, ModelConfigurationOption, + BooleanModelOptionParameterSchema, EngineParameterSchema, EngineRelease, + HardwareRequirements, HardwareSupport, HuggingFaceArtifactSource, HuggingFaceFileArtifact, + InferenceRequestKind, Model, ModelArtifact, ModelArtifactBinding, ModelConfigurationOption, ModelEngineConfiguration, ModelExecutionCapabilities, ModelImageDetail, ModelImageInputCapacity, ModelImageMediaType, ModelInputCapabilities, ModelInputModality, ModelOption, ModelOptionParameterDefinition, ModelOptionParameterValue, ModelOptionValues, - ModelParallelShape, SpeculativeDecodingMethod, SpeculativeDecodingModelOption, - UnsignedIntegerEngineParameterSchema, UnsignedIntegerModelOptionParameterSchema, - VisionModelOption, engine_parameter_definition, engine_parameter_value, - hugging_face_artifact_source, model_option, model_option_parameter_definition, - model_option_parameter_value, + ModelParallelShape, ParameterDefinition, ParameterValue, ParameterValues, + ParameterVisibility, SpeculativeDecodingMethod, SpeculativeDecodingModelOption, + UnsignedIntegerModelOptionParameterSchema, UnsignedIntegerParameterSchema, + VisionModelOption, hugging_face_artifact_source, model_option, + model_option_parameter_definition, model_option_parameter_value, parameter_definition, + parameter_value, }, engine::{ ninfer::v1::{ @@ -116,13 +116,13 @@ pub(crate) fn refresh_launch_plan(local: &mut ResolvedLocalModelDeploymentSnapsh } /// Returns ninfer engine parameter values for the given pairs. -pub(crate) fn parameter_values(values: [(&str, u64); N]) -> EngineParameterValues { - EngineParameterValues { +pub(crate) fn parameter_values(values: [(&str, u64); N]) -> ParameterValues { + ParameterValues { values: values .into_iter() - .map(|(parameter_id, value)| EngineParameterValue { + .map(|(parameter_id, value)| ParameterValue { parameter_id: parameter_id.to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(value)), + value: Some(parameter_value::Value::UnsignedInteger(value)), }) .collect(), } @@ -173,20 +173,21 @@ pub(crate) fn release() -> EngineRelease { .into_iter() .map( |(parameter_id, display_name, minimum, maximum, unit, default_value)| { - EngineParameterDefinition { + ParameterDefinition { parameter_id: parameter_id.to_owned(), display_name: display_name.to_owned(), - parameter_type: Some( - engine_parameter_definition::ParameterType::UnsignedInteger( - UnsignedIntegerEngineParameterSchema { - minimum, - maximum, - unit: unit.to_owned(), - default_value, - }, - ), - ), - exposed: true, + parameter_type: Some(parameter_definition::ParameterType::UnsignedInteger( + UnsignedIntegerParameterSchema { + minimum, + maximum, + unit: unit.to_owned(), + default_value, + }, + )), + visibility: match parameter_id { + "total-context" | "context-per-request" => ParameterVisibility::Basic, + _ => ParameterVisibility::Advanced, + } as i32, description: parameter_id.to_owned(), } }, diff --git a/backend/core/src/test_support/nix.rs b/backend/core/src/test_support/nix.rs index 1271206f..b94b99b5 100644 --- a/backend/core/src/test_support/nix.rs +++ b/backend/core/src/test_support/nix.rs @@ -6,8 +6,7 @@ use dotcore_domain::{ }; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineArtifact, NixArtifact, engine_artifact::Kind as EngineArtifactKind, - engine_parameter_value, + EngineArtifact, NixArtifact, engine_artifact::Kind as EngineArtifactKind, parameter_value, }, deployment::v1::ArtifactPreparationPolicy, engine::v1::resolved_engine_model_state, @@ -78,7 +77,7 @@ pub(crate) fn apple_metal_snapshot(model_id: &str) -> ModelDeploymentSnapshot { .iter_mut() .find(|value| value.parameter_id == "prompt-cache-memory-mib") .expect("prompt cache value") - .value = Some(engine_parameter_value::Value::UnsignedInteger(2048)); + .value = Some(parameter_value::Value::UnsignedInteger(2048)); local.desired_launch_plan = Some( crate::model::engine::EngineIntegrationRegistry::production() diff --git a/backend/core/src/test_support/strata.rs b/backend/core/src/test_support/strata.rs index 4e0091d9..375e14f8 100644 --- a/backend/core/src/test_support/strata.rs +++ b/backend/core/src/test_support/strata.rs @@ -11,14 +11,14 @@ use dotcore_catalog::shared_parameters::{ }; use dotcore_proto::dotcore::{ catalog::v1::{ - EngineParameterDefinition, EngineParameterSchema, EngineParameterValue, - EngineParameterValues, EngineRelease, HardwareRequirements, HardwareSupport, + EngineParameterSchema, EngineRelease, HardwareRequirements, HardwareSupport, HuggingFaceArtifactSource, HuggingFaceFileArtifact, HuggingFaceSnapshotArtifact, HuggingFaceSnapshotFile, InferenceRequestKind, Model, ModelArtifact, ModelArtifactBinding, ModelAuxiliaryArtifact, ModelEngineConfiguration, ModelExecutionCapabilities, - ModelInputCapabilities, ModelInputModality, ModelParallelShape, StorageBinding, - StorageKind, StorageScope, UnsignedIntegerEngineParameterSchema, - engine_parameter_definition, engine_parameter_value, hugging_face_artifact_source, + ModelInputCapabilities, ModelInputModality, ModelParallelShape, ParameterDefinition, + ParameterValue, ParameterValues, ParameterVisibility, SecretParameterSchema, + StorageBinding, StorageKind, StoragePlacement, UnsignedIntegerParameterSchema, + hugging_face_artifact_source, parameter_definition, parameter_value, }, engine::{ strata::v1::{ @@ -38,6 +38,12 @@ pub(crate) const EXPERT_VRAM_MIB_PARAMETER_ID: &str = "expert-vram-mib"; /// Auxiliary artifact holding the fixture model's separate draft layer. pub(crate) const DRAFT_LAYER_ARTIFACT_ID: &str = "fixture-draft-layer"; +/// Secret engine parameter some fixtures add: the deployment's API key. +pub(crate) const API_KEY_PARAMETER_ID: &str = "api-key"; + +/// Length of a generated [`API_KEY_PARAMETER_ID`] value. +pub(crate) const API_KEY_GENERATED_LENGTH: u32 = 32; + /// Resolved engine parameter values of the profile: the engine's default total context, the /// context per request filled from it, one parallel request and the expert cap from sizing. const PROFILE_PARAMETERS: [(&str, u64); 4] = [ @@ -102,6 +108,30 @@ pub(crate) fn local_model_snapshot_with( snapshot } +/// Returns [`local_model_snapshot`] from [`release_with_api_key`], with `api_key` as the value of +/// its secret. +pub(crate) fn local_model_snapshot_with_api_key( + model_id: &str, + api_key: &str, +) -> ModelDeploymentSnapshot { + let mut snapshot = local_model_snapshot(model_id); + let Some(model_deployment_snapshot::Source::Local(local)) = snapshot.source.as_mut() else { + panic!("local fixture snapshot"); + }; + local.engine_parameter_schema = release_with_api_key().parameter_schema; + local + .engine_parameter_values + .as_mut() + .expect("engine parameter values") + .values + .push(ParameterValue { + parameter_id: API_KEY_PARAMETER_ID.to_owned(), + value: Some(parameter_value::Value::String(api_key.to_owned())), + }); + refresh_launch_plan(local); + snapshot +} + /// Recomputes the canonical launch plan after a test edits a Strata snapshot. pub(crate) fn refresh_launch_plan(local: &mut ResolvedLocalModelDeploymentSnapshot) { local.desired_launch_plan = @@ -109,13 +139,13 @@ pub(crate) fn refresh_launch_plan(local: &mut ResolvedLocalModelDeploymentSnapsh } /// Returns Strata engine parameter values for the given pairs. -pub(crate) fn parameter_values(values: [(&str, u64); N]) -> EngineParameterValues { - EngineParameterValues { +pub(crate) fn parameter_values(values: [(&str, u64); N]) -> ParameterValues { + ParameterValues { values: values .into_iter() - .map(|(parameter_id, value)| EngineParameterValue { + .map(|(parameter_id, value)| ParameterValue { parameter_id: parameter_id.to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(value)), + value: Some(parameter_value::Value::UnsignedInteger(value)), }) .collect(), } @@ -166,20 +196,21 @@ pub(crate) fn release() -> EngineRelease { .into_iter() .map( |(parameter_id, display_name, minimum, maximum, unit, default_value)| { - EngineParameterDefinition { + ParameterDefinition { parameter_id: parameter_id.to_owned(), display_name: display_name.to_owned(), - parameter_type: Some( - engine_parameter_definition::ParameterType::UnsignedInteger( - UnsignedIntegerEngineParameterSchema { - minimum, - maximum, - unit: unit.to_owned(), - default_value, - }, - ), - ), - exposed: true, + parameter_type: Some(parameter_definition::ParameterType::UnsignedInteger( + UnsignedIntegerParameterSchema { + minimum, + maximum, + unit: unit.to_owned(), + default_value, + }, + )), + visibility: match parameter_id { + "total-context" | "context-per-request" => ParameterVisibility::Basic, + _ => ParameterVisibility::Advanced, + } as i32, description: parameter_id.to_owned(), } }, @@ -190,7 +221,7 @@ pub(crate) fn release() -> EngineRelease { binding_name: "data".to_owned(), mount_path: "/data".to_owned(), kind: StorageKind::Cache as i32, - scope: StorageScope::Deployment as i32, + placements: vec![StoragePlacement::Deployment as i32], environment: Vec::new(), }], artifacts: vec![dotcore_domain::test_support::oci::engine_artifact( @@ -206,6 +237,59 @@ pub(crate) fn release() -> EngineRelease { } } +/// Returns [`release`] with the secret [`API_KEY_PARAMETER_ID`] added after its other parameters. +pub(crate) fn release_with_api_key() -> EngineRelease { + let mut release = release(); + release + .parameter_schema + .as_mut() + .expect("schema") + .parameters + .push(ParameterDefinition { + parameter_id: API_KEY_PARAMETER_ID.to_owned(), + display_name: "API key".to_owned(), + parameter_type: Some(parameter_definition::ParameterType::Secret( + SecretParameterSchema { + generated_length: API_KEY_GENERATED_LENGTH, + minimum_length: 16, + }, + )), + visibility: ParameterVisibility::Advanced as i32, + description: API_KEY_PARAMETER_ID.to_owned(), + }); + release +} + +/// Returns [`CATALOG_ENGINE`] with the secret [`API_KEY_PARAMETER_ID`] of +/// [`release_with_api_key`]. +pub(crate) fn catalog_engine_with_api_key() -> String { + CATALOG_ENGINE.replace( + "storageBindings:\n", + " - { parameterId: \"api-key\", displayName: \"API key\", visibility: \"PARAMETER_VISIBILITY_ADVANCED\", description: \"api-key\", secret: { generatedLength: 32, minimumLength: 16 } }\nstorageBindings:\n", + ) +} + +/// Loads the given engine and model declarations with the shared hardware classes. +pub(crate) fn load_catalog(engine: &str, model: &str) -> anyhow::Result { + let root = tempfile::tempdir().expect("Catalog root"); + for (path, content) in [ + ("engines/strata.yaml", engine), + ("models/strata-model.yaml", model), + ] { + let path = root.path().join(path); + std::fs::create_dir_all(path.parent().expect("directory")).expect("directory"); + std::fs::write(path, content).expect("declaration"); + } + dotcore_catalog::test_support::write_hardware_classes(root.path()); + + dotcore_catalog::Catalog::load_with_policy( + root.path(), + dotcore_catalog::OciArtifactPolicy::Development, + crate::model::engine::declaration_validator(), + ) + .map_err(anyhow::Error::new) +} + /// Returns a model with a single GGUF, a split GGUF set, a ready-made pack and a draft layer, and /// no options. pub(crate) fn model() -> Model { @@ -371,3 +455,87 @@ fn snapshot_source(paths: [&str; N]) -> HuggingFaceArtifactSourc )), } } + +/// The catalog declaration of [`release`], for tests that load a catalog. +pub(crate) const CATALOG_ENGINE: &str = r#" +engineReleaseId: "fixture-strata" +engineKind: "ENGINE_KIND_STRATA" +version: "1" +parameterSchema: + schemaId: "fixture-strata-v1" + parameters: + - { parameterId: "total-context", displayName: "Total context", visibility: "PARAMETER_VISIBILITY_BASIC", description: "total-context", unsignedInteger: { minimum: "1024", maximum: "2097152", unit: "tokens", defaultValue: "32768" } } + - { parameterId: "context-per-request", displayName: "Context per request", visibility: "PARAMETER_VISIBILITY_BASIC", description: "context-per-request", unsignedInteger: { minimum: "1024", maximum: "262144", unit: "tokens" } } + - { parameterId: "parallel-requests", displayName: "Parallel requests", visibility: "PARAMETER_VISIBILITY_ADVANCED", description: "parallel-requests", unsignedInteger: { minimum: "1", maximum: "8", unit: "requests", defaultValue: "1" } } + - { parameterId: "expert-vram-mib", displayName: "Expert accelerator memory", visibility: "PARAMETER_VISIBILITY_ADVANCED", description: "expert-vram-mib", unsignedInteger: { minimum: "1", maximum: "1048576", unit: "MiB" } } +storageBindings: + - { bindingName: "data", mountPath: "/data", kind: "STORAGE_KIND_CACHE", placements: ["STORAGE_PLACEMENT_DEPLOYMENT"] } +artifacts: + - artifactId: "linux-amd64-cuda" + oci: + repository: { registry: { host: "registry.dotcore.test" }, name: "strata" } + digest: { algorithm: "sha256", encoded: "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" } + selectors: + - platform: { operatingSystem: "linux", architecture: "amd64" } + accelerators: ["nvidia"] + minimumNvidiaDriverMajorVersion: 580 +"#; + +/// The catalog declaration of a profile-shaped model for [`CATALOG_ENGINE`]: profile `strata-q4`, +/// whose sizing derives the expert cap. +pub(crate) const CATALOG_MODEL: &str = r#" +modelId: "fixture-strata-model" +displayName: "Fixture Strata model" +version: "1" +maximumContextTokens: 262144 +artifacts: + - artifactId: "fixture-strata-artifact" + weightQuantization: "Q4_K_XL" + source: + repository: "dotcore/fixture-strata" + revision: "0123456789abcdef0123456789abcdef01234567" + file: + path: "fixture-Q4_K_XL.gguf" + provenanceDigest: "74d2c57145e6ff11d1d2faa79594477f9bc903a611af1fb20218189fbbb77d82" +configurations: + - configurationId: "strata-nvidia" + engineArtifacts: [{ engineReleaseId: "fixture-strata", engineArtifactId: "linux-amd64-cuda" }] + artifactBindings: [{ bindingId: "fixture-strata-artifact", modelArtifactId: "fixture-strata-artifact" }] + sizing: + parameters: { baseBytes: 1000, bytesPerKvToken: 10, expertMib: 20000, expertVramMib: 8192 } + derivedParameters: + expert-vram-mib: "${expertVramMib}" + estimates: + acceleratorMemory: + unit: bytes + equation: "${baseBytes} + ${bytesPerKvToken} * ${total-context} + ${expert-vram-mib} * MiB" + hostMemory: + unit: bytes + equation: "${expertMib} * MiB" + hardwareSupport: + - hardwareClassId: "nvidia" + hardwareRequirements: + minimumMemoryBytesPerDevice: "32000000000" + minimumSystemMemoryBytes: "64000000000" + bindingSizing: + - bindingId: "fixture-strata-artifact" + engineConfiguration: + engineKind: "ENGINE_KIND_STRATA" + strata: + fixedSettings: + expertVramMibParameterId: "expert-vram-mib" + architecture: "qwen35moe" + mtpDraftTokens: 2 + singleGguf: {} + executionCapabilities: { supportedParallelShapes: ["MODEL_PARALLEL_SHAPE_ONE"] } + requestKinds: ['INFERENCE_REQUEST_KIND_CHAT'] + inputCapabilities: + supportedModalities: ["MODEL_INPUT_MODALITY_TEXT"] +profiles: + - profileId: "strata-q4" + displayName: "Q4 Strata" + description: "Runs the fixture through Strata." + configurationId: "strata-nvidia" + defaultModelArtifactBindingId: "fixture-strata-artifact" + engineParameterDefaults: {} + "#; diff --git a/backend/core/tests/backend/backend_integration.rs b/backend/core/tests/backend/backend_integration.rs index 7c9bf21a..1b6677ec 100644 --- a/backend/core/tests/backend/backend_integration.rs +++ b/backend/core/tests/backend/backend_integration.rs @@ -30,9 +30,9 @@ use dotcore_proto::dotcore::{ application_deployment_service_server::ApplicationDeploymentService as ApplicationServiceContract, }, catalog::v1::{ - EngineParameterValue, EngineParameterValues, ListApplicationsRequest, - ListEngineReleasesRequest, ListModelsRequest, UseCase, - catalog_service_server::CatalogService as CatalogServiceContract, engine_parameter_value, + ListApplicationsRequest, ListEngineReleasesRequest, ListModelsRequest, ParameterValue, + ParameterValues, UseCase, catalog_service_server::CatalogService as CatalogServiceContract, + parameter_value, }, deployment::v1::MemoryFit, inference::v1::{DeleteRouteRequest, InferenceRoute, ListRoutesRequest, ReplaceRouteRequest}, @@ -1952,6 +1952,7 @@ async fn route_input_application_without_a_link_reaches_ready_with_its_literal_f expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("create application without a route link") @@ -2008,6 +2009,7 @@ async fn referenced_application_route_cannot_be_deleted_until_application_remove expected_inference_route_generation: route.generation, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("create route-bound application") @@ -2076,8 +2078,8 @@ fn configure_route_name_application(root: &Path) { .replace("applicationId: \"open-webui\"", "applicationId: \"route-test\"") .replace("displayName: \"Open WebUI\"", "displayName: \"Route Test\"") .replace( - " storageBindings:", - " - bindingName: \"route-name\"\n environmentVariable: \"ROUTE_NAME\"\n literalValue: \"\"\n input:\n inferenceRoute:\n projection: \"APPLICATION_INFERENCE_ROUTE_PROJECTION_ROUTE_NAME\"\n storageBindings:", + " secretBindings:", + " - bindingName: \"route-name\"\n environmentVariable: \"ROUTE_NAME\"\n literalValue: \"\"\n input:\n inferenceRoute:\n projection: \"APPLICATION_INFERENCE_ROUTE_PROJECTION_ROUTE_NAME\"\n secretBindings:", ); fs::create_dir_all(root.join("catalog/applications")) .expect("route-name application directory"); @@ -2174,6 +2176,7 @@ async fn create_route_test_application( expected_inference_route_generation: route.generation, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("create route-linked application") @@ -2248,6 +2251,7 @@ async fn create_application(client: &mut ApplicationClient) -> ApplicationDeploy expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("create application") @@ -2913,6 +2917,7 @@ async fn application_restart_uses_site_and_state_ignores_obsolete_files_and_embe expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }), ) .await @@ -3019,6 +3024,7 @@ async fn application_service_persists_its_snapshot_name_and_logical_storage_unti expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }), ) .await @@ -3104,6 +3110,7 @@ async fn application_service_assigns_timestamped_ids_while_permitting_duplicate_ expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }), ) .await @@ -3123,6 +3130,7 @@ async fn application_service_assigns_timestamped_ids_while_permitting_duplicate_ expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }), ) .await @@ -3263,6 +3271,7 @@ async fn application_metadata_update_persists_and_projects_through_service_reope expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }), ) .await @@ -3342,6 +3351,7 @@ async fn application_metadata_update_persists_same_name_and_uncertain_acceptance expected_inference_route_generation: 0, application_deployment_name: Some("Stable Application".to_owned()), parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("create application") @@ -3433,6 +3443,7 @@ async fn application_metadata_update_acceptance_order_is_deterministic_against_t expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }), ) .await @@ -3515,6 +3526,7 @@ async fn application_metadata_update_acceptance_order_is_deterministic_against_t expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }), ) .await @@ -3820,6 +3832,7 @@ async fn interleaved_model_and_application_mutations_survive_canonical_site_rewr expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("Application mutation") @@ -4897,7 +4910,7 @@ async fn profile_compatibility_sizes_a_scenario_against_the_simulated_target_mem .collect::>(), [( "fixture-profile", - Some(EngineParameterValues { + Some(ParameterValues { values: [ ("prompt-cache-memory-mib", 0), ("total-context", total_context), @@ -4905,9 +4918,9 @@ async fn profile_compatibility_sizes_a_scenario_against_the_simulated_target_mem ("context-per-request", context_per_request), ] .into_iter() - .map(|(parameter_id, value)| EngineParameterValue { + .map(|(parameter_id, value)| ParameterValue { parameter_id: parameter_id.to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(value)), + value: Some(parameter_value::Value::UnsignedInteger(value)), }) .collect(), }), @@ -5198,9 +5211,9 @@ async fn rejected_local_configurations_return_their_reason_through_the_model_ser .values; for (parameter_id, value) in values { parameters.retain(|parameter| parameter.parameter_id != *parameter_id); - parameters.push(EngineParameterValue { + parameters.push(ParameterValue { parameter_id: (*parameter_id).to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(*value)), + value: Some(parameter_value::Value::UnsignedInteger(*value)), }); } CreateLocalModelDeploymentRequest { diff --git a/backend/core/tests/backend/catalog_contracts.rs b/backend/core/tests/backend/catalog_contracts.rs index 1d8e1e38..c1021eb0 100644 --- a/backend/core/tests/backend/catalog_contracts.rs +++ b/backend/core/tests/backend/catalog_contracts.rs @@ -30,9 +30,9 @@ use dotcore_proto::dotcore::{ EngineArtifactReference, EngineRelease, HardwareSupport, HuggingFaceSnapshotArtifact, HuggingFaceSnapshotFile, IntelligenceScore, License, ListEngineReleasesRequest, ListModelsRequest, ListUpstreamModelTemplatesRequest, Model, NixSystemStorePath, - OciPlatform, Selector, UpstreamModelTemplate, UpstreamWireProtocol, - engine_artifact::Kind as EngineArtifactKind, engine_parameter_value, - hugging_face_artifact_source::Content as HuggingFaceContent, + OciPlatform, ParameterVisibility, Selector, UpstreamModelTemplate, UpstreamWireProtocol, + engine_artifact::Kind as EngineArtifactKind, + hugging_face_artifact_source::Content as HuggingFaceContent, parameter_value, }, deployment::v1::ArtifactPreparationPolicy, engine::v1::EngineKind, @@ -678,7 +678,7 @@ fn catalog_sizing_resolves_estimates_into_the_deployment_snapshot() { .iter_mut() .find(|default| default.parameter_id == *parameter_id) .unwrap_or_else(|| panic!("{parameter_id} default")) - .value = Some(engine_parameter_value::Value::UnsignedInteger(*value)); + .value = Some(parameter_value::Value::UnsignedInteger(*value)); } resolve_local_launch_with_placeholder_identity( &catalog, @@ -773,6 +773,7 @@ fn llama_cpp_request_state_scales_the_estimate_with_the_parallel_requests() { &target, &inventory, None, + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator), ) .expect("resolved parallel requests") .sizing @@ -1678,6 +1679,7 @@ fn locked_nix_deployment_retains_its_catalog_store_paths_for_the_hosting_node_ac &darwin, &inventory, identity, + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator), ) }; @@ -1791,6 +1793,7 @@ fn development_source_nix_deployment_leaves_its_store_path_for_the_hosting_node( &darwin, &inventory, identity, + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator), ) }; @@ -1974,6 +1977,7 @@ fn catalog_resolves_file_artifacts_in_repository_subdirectories() { &target, &inventory, None, + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator), ) .expect("resolved nested file artifacts"); @@ -2127,11 +2131,11 @@ fn catalog_rejects_invalid_parameters_and_unsupported_bindings() { let catalog = synthetic_local_catalog(); let model_id = CatalogModelId("fixture-model".to_owned()); let (target, inventory) = target_capabilities(ExecutionPlatform::linux_amd64()); - let invalid = dotcore_proto::dotcore::catalog::v1::EngineParameterValues { - values: vec![dotcore_proto::dotcore::catalog::v1::EngineParameterValue { + let invalid = dotcore_proto::dotcore::catalog::v1::ParameterValues { + values: vec![dotcore_proto::dotcore::catalog::v1::ParameterValue { parameter_id: "total-context".to_owned(), value: Some( - dotcore_proto::dotcore::catalog::v1::engine_parameter_value::Value::UnsignedInteger( + dotcore_proto::dotcore::catalog::v1::parameter_value::Value::UnsignedInteger( 16_777_217, ), ), @@ -2151,7 +2155,8 @@ fn catalog_rejects_invalid_parameters_and_unsupported_bindings() { &invalid_configuration, &target, &inventory, - None + None, + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator) ) .expect_err("out-of-range parameter") .to_string(), @@ -2169,7 +2174,8 @@ fn catalog_rejects_invalid_parameters_and_unsupported_bindings() { }, &target, &inventory, - None + None, + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator) ) .expect_err("unsupported binding") .to_string(), @@ -2199,6 +2205,7 @@ fn llama_cpp_launch_rejects_a_context_per_request_above_the_model_maximum() { &target, &inventory, None, + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator), ) }; @@ -2218,8 +2225,8 @@ fn llama_cpp_launch_rejects_a_context_per_request_above_the_model_maximum() { .iter() .find(|value| value.parameter_id == "context-per-request") }) - .and_then(|value| value.value), - Some(engine_parameter_value::Value::UnsignedInteger(262_144)) + .and_then(|value| value.value.clone()), + Some(parameter_value::Value::UnsignedInteger(262_144)) ); let arguments = filled .desired_launch_plan @@ -2307,6 +2314,7 @@ fn llama_cpp_launch_rejects_a_context_per_request_above_the_configuration_maximu &target, &inventory, None, + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator), ) }; @@ -2379,6 +2387,7 @@ fn vllm_launch_rejects_a_context_per_request_above_the_model_maximum() { &target, &inventory, None, + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator), ) }; @@ -2412,6 +2421,7 @@ fn catalog_retains_boolean_engine_parameters_and_their_schema() { &target, &inventory, None, + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator), ) .expect("boolean parameter resolves"); @@ -2424,7 +2434,7 @@ fn catalog_retains_boolean_engine_parameters_and_their_schema() { .iter() .find(|value| value.parameter_id == "feature-enabled") .and_then(|value| value.value.as_ref()), - Some(&dotcore_proto::dotcore::catalog::v1::engine_parameter_value::Value::Boolean(false)) + Some(&dotcore_proto::dotcore::catalog::v1::parameter_value::Value::Boolean(false)) ); assert_eq!( snapshot @@ -2439,7 +2449,7 @@ fn catalog_retains_boolean_engine_parameters_and_their_schema() { } #[test] -fn catalog_rejects_mismatched_boolean_values_and_invalid_profile_exposure() { +fn catalog_rejects_mismatched_boolean_values_and_invalid_visibility_overrides() { let mut mismatched = synthetic_boolean_model(); mismatched.profiles[0] .engine_parameter_defaults @@ -2449,9 +2459,8 @@ fn catalog_rejects_mismatched_boolean_values_and_invalid_profile_exposure() { .iter_mut() .find(|value| value.parameter_id == "feature-enabled") .expect("feature-enabled default") - .value = Some( - dotcore_proto::dotcore::catalog::v1::engine_parameter_value::Value::UnsignedInteger(1), - ); + .value = + Some(dotcore_proto::dotcore::catalog::v1::parameter_value::Value::UnsignedInteger(1)); let error = catalog_builder() .engine_validator(&ACCEPT_ENGINE_DECLARATIONS) .engine_release("engine.yaml", synthetic_boolean_engine_release()) @@ -2466,24 +2475,57 @@ fn catalog_rejects_mismatched_boolean_values_and_invalid_profile_exposure() { "unexpected validation error: {error}" ); - for exposed_parameter_ids in [ - vec!["total-context".to_owned(), "missing".to_owned()], - vec!["total-context".to_owned(), "total-context".to_owned()], + use ParameterVisibility::{Advanced, Basic, Hidden}; + + for (overrides, expected) in [ + ( + vec![("missing", Hidden)], + "profile \"balanced\" overrides the visibility of engine parameter \"missing\", which is not a parameter of the engine release", + ), + ( + vec![("feature-enabled", Hidden), ("feature-enabled", Advanced)], + "profile \"balanced\" overrides the visibility of engine parameter \"feature-enabled\" more than once", + ), + ( + vec![("total-context", Advanced)], + "profile \"balanced\" engine parameter \"total-context\" may be hidden but not advanced", + ), + ( + vec![("context-per-request", Advanced)], + "profile \"balanced\" engine parameter \"context-per-request\" may be hidden but not advanced", + ), ] { let mut invalid = synthetic_boolean_model(); - invalid.profiles[0].exposed_parameter_ids = exposed_parameter_ids; + invalid.profiles[0].engine_parameter_visibility_overrides = + visibility_overrides(&overrides); let error = catalog_builder() .engine_validator(&ACCEPT_ENGINE_DECLARATIONS) .engine_release("engine.yaml", synthetic_boolean_engine_release()) .model("model.yaml", invalid) .build() - .expect_err("invalid profile exposure") + .expect_err("invalid visibility override") .to_string(); assert!( - error.contains("profile \"balanced\" has invalid exposed engine parameters"), + error.contains(expected), "unexpected validation error: {error}" ); } + + // A profile may hide the contexts, which the engine and resolution fill, and a parameter it + // defaults. + let mut valid = synthetic_boolean_model(); + valid.profiles[0].engine_parameter_visibility_overrides = visibility_overrides(&[ + ("total-context", Hidden), + ("context-per-request", Hidden), + ("feature-enabled", Hidden), + ]); + synthetic_boolean_catalog(valid); + + // A profile may show any other parameter in the main view. + let mut promoted = synthetic_boolean_model(); + promoted.profiles[0].engine_parameter_visibility_overrides = + visibility_overrides(&[("feature-enabled", Basic)]); + synthetic_boolean_catalog(promoted); } /// Returns a fresh target of the production runtime kind for `platform`: Docker with an NVIDIA @@ -2539,7 +2581,7 @@ fn lowest_total_context(catalog: &Catalog, model: &Model, profile_id: &str) -> ( use dotcore_catalog::shared_parameters::{ CONTEXT_PER_REQUEST_PARAMETER_ID, TOTAL_CONTEXT_PARAMETER_ID, }; - use dotcore_proto::dotcore::catalog::v1::engine_parameter_definition::ParameterType; + use dotcore_proto::dotcore::catalog::v1::parameter_definition::ParameterType; let profile = model .profiles @@ -2575,7 +2617,7 @@ fn lowest_total_context(catalog: &Catalog, model: &Model, profile_id: &str) -> ( .flat_map(|defaults| &defaults.values) .find(|value| value.parameter_id == CONTEXT_PER_REQUEST_PARAMETER_ID) .and_then(|value| match value.value { - Some(engine_parameter_value::Value::UnsignedInteger(tokens)) => Some(tokens), + Some(parameter_value::Value::UnsignedInteger(tokens)) => Some(tokens), _ => None, }); let lowest = context_per_request.map_or(smallest_total, |tokens| tokens.max(smallest_total)); @@ -3022,18 +3064,16 @@ fn repository_catalog_root() -> std::path::PathBuf { } /// Builds unsigned engine parameter values from parameter id and value pairs. -fn unsigned_values( - values: &[(&str, u64)], -) -> dotcore_proto::dotcore::catalog::v1::EngineParameterValues { - dotcore_proto::dotcore::catalog::v1::EngineParameterValues { +fn unsigned_values(values: &[(&str, u64)]) -> dotcore_proto::dotcore::catalog::v1::ParameterValues { + dotcore_proto::dotcore::catalog::v1::ParameterValues { values: values .iter() - .map(|(parameter_id, value)| { - dotcore_proto::dotcore::catalog::v1::EngineParameterValue { + .map( + |(parameter_id, value)| dotcore_proto::dotcore::catalog::v1::ParameterValue { parameter_id: (*parameter_id).to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(*value)), - } - }) + value: Some(parameter_value::Value::UnsignedInteger(*value)), + }, + ) .collect(), } } @@ -3108,6 +3148,7 @@ fn resolve_local_launch_with_placeholder_identity( target, inventory, identity.as_ref(), + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator), ) } @@ -3153,7 +3194,8 @@ fn synthetic_sizing_catalog() -> Catalog { "#, &format!( r#" values: -{SIZING_ENGINE_CACHE_DEFAULT} exposedParameterIds: ["total-context", "prompt-cache-memory-mib", "parallel-requests"] +{SIZING_ENGINE_CACHE_DEFAULT} engineParameterVisibilityOverrides: + - {{ parameterId: "prompt-cache-memory-mib", visibility: "PARAMETER_VISIBILITY_ADVANCED" }} "# ), ); @@ -3183,15 +3225,15 @@ fn synthetic_boolean_engine_release() -> EngineRelease { .as_mut() .expect("fixture schema") .parameters - .push(dotcore_proto::dotcore::catalog::v1::EngineParameterDefinition { + .push(dotcore_proto::dotcore::catalog::v1::ParameterDefinition { parameter_id: "feature-enabled".to_owned(), display_name: "Feature enabled".to_owned(), parameter_type: Some( - dotcore_proto::dotcore::catalog::v1::engine_parameter_definition::ParameterType::Boolean( - dotcore_proto::dotcore::catalog::v1::BooleanEngineParameterSchema {}, + dotcore_proto::dotcore::catalog::v1::parameter_definition::ParameterType::Boolean( + dotcore_proto::dotcore::catalog::v1::BooleanParameterSchema::default(), ), ), - exposed: true, + visibility: ParameterVisibility::Advanced as i32, description: "Enables the synthetic feature.".to_owned(), }); engine @@ -3205,16 +3247,30 @@ fn synthetic_boolean_model() -> Model { .as_mut() .expect("fixture defaults") .values - .push(dotcore_proto::dotcore::catalog::v1::EngineParameterValue { + .push(dotcore_proto::dotcore::catalog::v1::ParameterValue { parameter_id: "feature-enabled".to_owned(), value: Some( - dotcore_proto::dotcore::catalog::v1::engine_parameter_value::Value::Boolean(false), + dotcore_proto::dotcore::catalog::v1::parameter_value::Value::Boolean(false), ), }); - profile.exposed_parameter_ids = vec!["total-context".to_owned(), "feature-enabled".to_owned()]; model } +/// Builds profile visibility overrides from parameter id and visibility pairs. +fn visibility_overrides( + overrides: &[(&str, ParameterVisibility)], +) -> Vec { + overrides + .iter() + .map(|(parameter_id, visibility)| { + dotcore_proto::dotcore::catalog::v1::ParameterVisibilityOverride { + parameter_id: (*parameter_id).to_owned(), + visibility: *visibility as i32, + } + }) + .collect() +} + fn synthetic_engine_release() -> EngineRelease { dotcore_proto::yaml::from_str( r#"engineReleaseId: "fixture-engine" @@ -3225,7 +3281,7 @@ parameterSchema: parameters: - parameterId: "total-context" displayName: "Total context" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: minimum: "1024" maximum: "16777216" @@ -3233,14 +3289,14 @@ parameterSchema: defaultValue: "32768" - parameterId: "context-per-request" displayName: "Context per request" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: minimum: "1024" maximum: "16777216" unit: "tokens" - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true + visibility: "PARAMETER_VISIBILITY_ADVANCED" unsignedInteger: minimum: "1" maximum: "64" @@ -3248,6 +3304,7 @@ parameterSchema: defaultValue: "4" - parameterId: "prompt-cache-memory-mib" displayName: "Prompt cache memory" + visibility: "PARAMETER_VISIBILITY_HIDDEN" unsignedInteger: minimum: "0" maximum: "65536" diff --git a/backend/core/tests/backend/declared_machine.rs b/backend/core/tests/backend/declared_machine.rs index 982f4af5..01d60324 100644 --- a/backend/core/tests/backend/declared_machine.rs +++ b/backend/core/tests/backend/declared_machine.rs @@ -597,6 +597,7 @@ async fn a_simulated_app_records_its_services_cpu_and_memory_as_metric_history() expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("create the application") diff --git a/backend/core/tests/gated/apple_hardware_integration.rs b/backend/core/tests/gated/apple_hardware_integration.rs index 06e82b13..4a21cb6c 100644 --- a/backend/core/tests/gated/apple_hardware_integration.rs +++ b/backend/core/tests/gated/apple_hardware_integration.rs @@ -25,7 +25,7 @@ use dotcore_node_hardware::{ hardware::{SystemHardwareSnapshotCollector, SystemNvidiaAcceleratorAdapter}, }; use dotcore_proto::dotcore::{ - catalog::v1::engine_parameter_value, + catalog::v1::parameter_value, deployment::v1::accelerator_memory_observation, model::v1::{ CreateLocalModelDeploymentRequest, CreateModelDeploymentRequest, DirectRouteExposure, @@ -646,7 +646,7 @@ fn model_configuration() -> LocalModelDeploymentConfiguration { .iter_mut() .find(|value| value.parameter_id == "prompt-cache-memory-mib") .expect("prompt cache parameter value"); - prompt_cache.value = Some(engine_parameter_value::Value::UnsignedInteger( + prompt_cache.value = Some(parameter_value::Value::UnsignedInteger( PROMPT_CACHE_MEMORY_MIB, )); configuration diff --git a/backend/core/tests/gated/decision_requests.rs b/backend/core/tests/gated/decision_requests.rs index 5a857b2c..294ecea0 100644 --- a/backend/core/tests/gated/decision_requests.rs +++ b/backend/core/tests/gated/decision_requests.rs @@ -5,7 +5,7 @@ //! and check the response structure and error shapes. Model probabilities are never asserted. use dotcore_proto::dotcore::{ - catalog::v1::engine_parameter_value, model::v1::LocalModelDeploymentConfiguration, + catalog::v1::parameter_value, model::v1::LocalModelDeploymentConfiguration, }; use serde_json::{Value, json}; @@ -39,7 +39,7 @@ pub fn decision_model_configuration() -> LocalModelDeploymentConfiguration { .iter_mut() .find(|value| value.parameter_id == "parallel-requests") .expect("parallel requests parameter value") - .value = Some(engine_parameter_value::Value::UnsignedInteger(1)); + .value = Some(parameter_value::Value::UnsignedInteger(1)); configuration } diff --git a/backend/core/tests/gated/docker_application_integration.rs b/backend/core/tests/gated/docker_application_integration.rs index 13e734c3..a83271e9 100644 --- a/backend/core/tests/gated/docker_application_integration.rs +++ b/backend/core/tests/gated/docker_application_integration.rs @@ -21,14 +21,15 @@ use bollard::{Docker, errors::Error as BollardError, query_parameters::CreateIma use dotcore_domain::SitePaths; use dotcore_proto::dotcore::application::v1::{ ApplicationDeployment, ApplicationDeploymentDesiredState, ApplicationLifecycleState, - ApplicationReadinessState, CreateApplicationDeploymentRequest, FollowDeploymentLogsRequest, - GetApplicationDeploymentRequest, PauseDeploymentRequest, ReadDeploymentLogsRequest, - ReadDeploymentLogsResponse, RemoveApplicationDeploymentRequest, ResumeDeploymentRequest, + ApplicationReadinessState, ApplicationStorageChoice, CreateApplicationDeploymentRequest, + FollowDeploymentLogsRequest, GetApplicationDeploymentRequest, PauseDeploymentRequest, + ReadDeploymentLogsRequest, ReadDeploymentLogsResponse, RemoveApplicationDeploymentRequest, + ResumeDeploymentRequest, application_deployment_service_client::ApplicationDeploymentServiceClient, application_deployment_service_server::ApplicationDeploymentService as ApplicationServiceContract, }; use dotcore_proto::dotcore::catalog::v1::{ - ListApplicationsRequest, OciPlatform, + ListApplicationsRequest, OciPlatform, StoragePlacement, catalog_service_server::CatalogService as CatalogServiceContract, }; use dotcore_proto::dotcore::deployment::v1::ArtifactPreparationPolicy; @@ -82,7 +83,10 @@ const PYTHON_IMAGE: &str = const PYTHON_DIGEST: &str = "b3575dcf816cdacc5292df77a6e3e59b8941631078373a045efa2ac1e9c5c3b2"; const ARM64_DEFAULT_VARIANT: &str = "v8"; const LIFECYCLE_APPLICATION_ID: &str = "synthetic-lifecycle"; -const STORAGE_SCOPES_APPLICATION_ID: &str = "synthetic-storage-scopes"; +const STORAGE_PLACEMENTS_APPLICATION_ID: &str = "synthetic-storage-placements"; +const SECRET_APPLICATION_ID: &str = "secret-fixture"; +/// A typed secret value no other site content contains. +const SECRET_SENTINEL: &str = "SentinelQv8Rk2Ld7Wn4"; const LOCAL_IMAGE_REPOSITORY: &str = "dotcore-test/synthetic-lifecycle"; const UNROUTED_VALUE: &str = "unrouted"; const SUPPORTED_PLATFORMS: &str = " supportedPlatforms:\n - operatingSystem: \"linux\"\n architecture: \"amd64\"\n - operatingSystem: \"linux\"\n architecture: \"arm64\"\n"; @@ -763,13 +767,13 @@ async fn hw_docker_two_deployments_of_one_application_isolate_data_and_networks( #[tokio::test] #[ignore = "hardware: just backend::test-docker"] -async fn hw_docker_application_scoped_storage_outlives_the_deployment_that_wrote_it() { +async fn hw_docker_shared_app_storage_outlives_its_writer_and_deployment_storage_does_not() { let resources = TestResources::new() .await .expect("Docker preflight and temporary site"); resources - .write_storage_scopes_application(&remote_image_artifact()) - .expect("storage scopes application catalog declaration"); + .write_storage_placements_application(&remote_image_artifact()) + .expect("storage placements application catalog declaration"); let backend = support::RunningBackend::start_at_with_simulated_runtime_test_support( resources.configuration.clone(), resources.publication_host.into(), @@ -780,14 +784,14 @@ async fn hw_docker_application_scoped_storage_outlives_the_deployment_that_wrote let mut applications = backend.application_client(); let request = || { let mut request = lifecycle_deployment_request(None); - request.application_id = STORAGE_SCOPES_APPLICATION_ID.to_owned(); + request.application_id = STORAGE_PLACEMENTS_APPLICATION_ID.to_owned(); request }; let application_root = resources .root .join("applications-data") .join("application") - .join(STORAGE_SCOPES_APPLICATION_ID); + .join(STORAGE_PLACEMENTS_APPLICATION_ID); // The first deployment writes one model file and one output file. let first = create_when_target_observed(&mut applications, request()).await?; @@ -852,7 +856,48 @@ async fn hw_docker_application_scoped_storage_outlives_the_deployment_that_wrote wait_for_no_docker_resources_client(&resources, &mut applications, &second_id).await?; resources .wait_for_application_data_removed(&second_data) + .await?; + + // A deployment that keeps its models to itself gets an empty models folder inside its own + // storage, and its teardown deletes it while the shared folder stays. + let own = create_when_target_observed( + &mut applications, + CreateApplicationDeploymentRequest { + storage_choices: vec![ApplicationStorageChoice { + service_name: "web".to_owned(), + binding_name: "models".to_owned(), + placement: StoragePlacement::Deployment as i32, + }], + ..request() + }, + ) + .await?; + let own_id = own.application_deployment_id.clone(); + let own_data = resources.application_data_path(&own_id, &attached_storage(&own, "data")?); + let own_models = + resources.application_data_path(&own_id, &attached_storage(&own, "models")?); + let container = + wait_for_container_exec(&resources, &own_id, "mkdir -p /data/output").await?; + assert_nested_models_mount(&container, &own_data, &own_models)?; + if container_file(&container, "/data/models/model.bin").is_ok() { + return Err("deployment-placed models see the shared model file".to_owned()); + } + applications + .remove_application_deployment(RemoveApplicationDeploymentRequest { + application_deployment_id: own_id.clone(), + }) .await + .map_err(|error| format!("remove own-models application {own_id}: {error}"))?; + wait_for_no_docker_resources_client(&resources, &mut applications, &own_id).await?; + resources + .wait_for_application_data_removed(&own_models) + .await?; + if !first_models.join("model.bin").exists() { + return Err( + "removing a deployment-placed models folder touched the shared one".to_owned(), + ); + } + Ok(()) } .await; backend.stop().await; @@ -1101,6 +1146,7 @@ async fn hw_docker_synthetic_two_service_group_recovers_with_private_dns_and_one expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }), ) .await @@ -1190,6 +1236,7 @@ async fn hw_docker_nvidia_group_units_receive_the_target_device_request() { expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }), ) .await @@ -1263,6 +1310,7 @@ async fn hw_docker_nvidia_group_units_report_attributed_accelerator_memory() { expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }), ) .await @@ -1444,6 +1492,7 @@ async fn hw_docker_lightweight_application_logs_are_owned_filtered_and_cancellab expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }), ) .await @@ -1723,6 +1772,210 @@ async fn hw_docker_lightweight_application_logs_are_owned_filtered_and_cancellab } } +#[tokio::test] +#[ignore = "hardware: just backend::test-docker"] +async fn hw_docker_secret_parameter_reaches_only_the_container_and_site_yaml() { + // Debug-level records reach the site's retained logs, which the sentinel search covers. + dotcore::logging::initialize_with_default_filter( + "info,dotcore=debug,dotcore_runtime=debug,dotcore_runtime_backends=debug,dotcore_site=debug", + ); + let resources = TestResources::new_log_fixture() + .await + .expect("Docker preflight and temporary site"); + resources + .write_secret_application() + .expect("secret application catalog declaration"); + let backend = support::RunningBackend::start_at_with_simulated_runtime_test_support( + resources.configuration.clone(), + resources.publication_host.into(), + dotcore_runtime_backends::simulated::SimulatedRuntimeTestSupport::with_stop_failures(0), + ) + .await; + let result = async { + let mut applications = backend.application_client(); + let serving = + create_when_target_observed(&mut applications, secret_deployment_request("serve")) + .await?; + let serving_id = serving.application_deployment_id.clone(); + let failing = + create_when_target_observed(&mut applications, secret_deployment_request("exit")) + .await?; + let failing_id = failing.application_deployment_id.clone(); + + wait_for_endpoint_client(&mut applications, &serving_id).await?; + let container = resources.running_container_id_for_group(&serving_id, "web")?; + let environment = container_environment(&container)?; + if !environment.contains(&format!("PASSWORD={SECRET_SENTINEL}")) + || !environment.contains(&"USERNAME=admin".to_owned()) + { + return Err(format!( + "the container environment lacks the secret or the username: {} entries", + environment.len() + )); + } + // The serving unit prints the username and the secret; deployment logs redact every + // launch value. + let serving_logs = read_logs_containing( + &mut applications, + &serving_id, + "login [REDACTED] [REDACTED]", + ) + .await?; + assert_no_sentinel("serving deployment logs", &serving_logs)?; + + // The failing unit prints the secret, then exits before it ever serves. + let failing_logs = + read_logs_containing(&mut applications, &failing_id, "stopping with [REDACTED]") + .await?; + assert_no_sentinel("failing deployment logs", &failing_logs)?; + // An exiting unit leaves the deployment unavailable without a recorded failure. + let exited = applications + .get_application_deployment(GetApplicationDeploymentRequest { + application_deployment_id: failing_id.clone(), + }) + .await + .map_err(|error| format!("read failing deployment: {error}"))? + .into_inner() + .deployment + .ok_or_else(|| "failing deployment read returned no deployment".to_owned())?; + let state = exited + .state + .as_ref() + .ok_or_else(|| "failing deployment has no state".to_owned())?; + if state.lifecycle_state == ApplicationLifecycleState::Ready as i32 { + return Err("the failing deployment became ready".to_owned()); + } + assert_no_sentinel("failing deployment state", &format!("{state:?}"))?; + Ok::<_, String>((serving_id, failing_id, container)) + } + .await; + backend.stop().await; + if result.is_ok() { + wait_for_backend_service_release(&resources.configuration) + .await + .expect("release backend repositories before the restart"); + } + let result = match result { + Ok((serving_id, failing_id, container)) => { + let restarted = support::RunningBackend::start_at_with_simulated_runtime_test_support( + resources.configuration.clone(), + resources.publication_host.into(), + dotcore_runtime_backends::simulated::SimulatedRuntimeTestSupport::with_stop_failures(0), + ) + .await; + let restart_result = async { + let mut applications = restarted.application_client(); + let retained = wait_for_endpoint_client(&mut applications, &serving_id).await?; + let password = retained + .snapshot + .as_ref() + .and_then(|snapshot| { + snapshot + .parameter_values + .iter() + .find(|value| value.parameter_id == "password") + }) + .and_then(|value| value.value.clone()); + if password + != Some( + dotcore_proto::dotcore::catalog::v1::parameter_value::Value::String( + SECRET_SENTINEL.to_owned(), + ), + ) + { + return Err("the restart changed the retained secret".to_owned()); + } + let restarted_container = + resources.running_container_id_for_group(&serving_id, "web")?; + if restarted_container != container { + return Err(format!( + "the restart replaced the container {container} with {restarted_container}" + )); + } + // A launch refused on a taken published port records failure text and its cause. + let port = endpoint_port(&endpoint_url(&retained)?)?; + applications + .pause_deployment(PauseDeploymentRequest { + application_deployment_id: serving_id.clone(), + }) + .await + .map_err(|error| format!("pause secret deployment: {error}"))?; + wait_for_application_lifecycle_client( + &mut applications, + &serving_id, + ApplicationLifecycleState::Paused, + ) + .await?; + let holder = std::net::TcpListener::bind((resources.publication_host, port)) + .map_err(|error| format!("hold published port {port}: {error}"))?; + applications + .resume_deployment(ResumeDeploymentRequest { + application_deployment_id: serving_id.clone(), + }) + .await + .map_err(|error| format!("resume secret deployment: {error}"))?; + let refused = + wait_for_failure_code(&mut applications, &serving_id, "PublishedPortInUse") + .await?; + drop(holder); + let failure = refused + .state + .and_then(|state| state.failure) + .ok_or_else(|| "the refused launch recorded no failure".to_owned())?; + if failure.message.is_empty() { + return Err("the refused launch recorded no failure text".to_owned()); + } + assert_no_sentinel("refused launch failure", &format!("{failure:?}"))?; + + // Every retained site file but the declarations, durable debug logs included, + // lacks the secret; the declarations keep it, which proves the search reads them. + let debug_records = support::retained_json_records(&resources.root) + .iter() + .filter(|record| record["level"] == "DEBUG") + .count(); + if debug_records == 0 { + return Err("the site logs retained no debug records".to_owned()); + } + let hits = files_containing(&resources.root, SECRET_SENTINEL.as_bytes())?; + if hits != [resources.root.join("site.yaml")] { + return Err(format!( + "the secret sentinel is retained in {hits:?}, not only site.yaml" + )); + } + for deployment_id in [&serving_id, &failing_id] { + applications + .remove_application_deployment(RemoveApplicationDeploymentRequest { + application_deployment_id: deployment_id.clone(), + }) + .await + .map_err(|error| format!("remove secret deployment: {error}"))?; + } + // The site-wide Docker check passes only once both groups are gone. + for deployment_id in [&serving_id, &failing_id] { + wait_for_no_docker_resources_client( + &resources, + &mut applications, + deployment_id, + ) + .await?; + } + Ok(()) + } + .await; + restarted.stop().await; + restart_result + } + Err(error) => Err(error), + }; + let cleanup = resources.cleanup(); + match (result, cleanup) { + (Ok(()), Ok(())) => {} + (Err(primary), Ok(())) => panic!("{primary}"), + (Ok(()), Err(cleanup)) => panic!("cleanup: {cleanup}"), + (Err(primary), Err(cleanup)) => panic!("{primary}; cleanup: {cleanup}"), + } +} + /// Configuration values the lifecycle application's container must render. struct ConfigurationExpectation { inference_base_url: String, @@ -1742,6 +1995,7 @@ fn lifecycle_deployment_request( expected_inference_route_generation: route.map_or(0, |route| route.generation), application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), } } @@ -1786,6 +2040,142 @@ async fn create_when_target_observed( .map_err(|_| "timed out waiting for the application target platform observation".to_owned())? } +/// Requests a secret fixture deployment with the typed sentinel password and `outcome`. +fn secret_deployment_request(outcome: &str) -> CreateApplicationDeploymentRequest { + let text = |id: &str, value: &str| dotcore_proto::dotcore::catalog::v1::ParameterValue { + parameter_id: id.to_owned(), + value: Some( + dotcore_proto::dotcore::catalog::v1::parameter_value::Value::String(value.to_owned()), + ), + }; + CreateApplicationDeploymentRequest { + application_id: SECRET_APPLICATION_ID.to_owned(), + placement: Some(support::exact_placement( + "docker-apps", + "docker-apps-target", + )), + inference_route_id: None, + expected_inference_route_generation: 0, + application_deployment_name: None, + parameter_values: vec![text("password", SECRET_SENTINEL), text("outcome", outcome)], + storage_choices: Vec::new(), + } +} + +async fn wait_for_endpoint_client( + client: &mut ApplicationClient, + deployment_id: &str, +) -> Result { + let deadline = Instant::now() + READY_TIMEOUT; + loop { + let deployment = client + .get_application_deployment(GetApplicationDeploymentRequest { + application_deployment_id: deployment_id.to_owned(), + }) + .await + .map_err(|error| format!("read application deployment: {error}"))? + .into_inner() + .deployment + .ok_or_else(|| "Application read returned no deployment".to_owned())?; + if deployment + .state + .as_ref() + .and_then(|state| state.endpoint.as_ref()) + .is_some() + { + return Ok(deployment); + } + if Instant::now() >= deadline { + return Err(format!( + "timed out waiting for an application endpoint; last state {:?}", + deployment.state + )); + } + sleep(Duration::from_millis(250)).await; + } +} + +/// Reads a deployment's log history until one entry contains `expected`, and returns every entry +/// of that read joined by line breaks. +async fn read_logs_containing( + client: &mut ApplicationClient, + deployment_id: &str, + expected: &str, +) -> Result { + let deadline = Instant::now() + LOG_HISTORY_TIMEOUT; + loop { + let history = client + .read_deployment_logs(ReadDeploymentLogsRequest { + application_deployment_id: deployment_id.to_owned(), + ..Default::default() + }) + .await + .map_err(|error| format!("read deployment logs: {error}"))? + .into_inner(); + let text = history + .entries + .iter() + .map(|entry| String::from_utf8_lossy(&entry.content).into_owned()) + .collect::>() + .join("\n"); + if text.lines().any(|line| line.contains(expected)) { + return Ok(text); + } + if Instant::now() >= deadline { + return Err(format!( + "timed out waiting for log line {expected:?}; last read {:?}", + text.replace(SECRET_SENTINEL, "") + )); + } + sleep(Duration::from_millis(250)).await; + } +} + +fn assert_no_sentinel(surface: &str, text: &str) -> Result<(), String> { + if text.contains(SECRET_SENTINEL) { + Err(format!("the secret sentinel appears in the {surface}")) + } else { + Ok(()) + } +} + +/// Returns the environment of one container as `NAME=value` entries. +fn container_environment(container: &str) -> Result, String> { + serde_json::from_str(&command( + "docker", + ["inspect", "--format", "{{json .Config.Env}}", container], + )?) + .map_err(|error| format!("decode container environment: {error}")) +} + +/// Returns every file under `root` whose bytes contain `needle`. +fn files_containing(root: &Path, needle: &[u8]) -> Result, String> { + let mut hits = Vec::new(); + let mut pending = vec![root.to_path_buf()]; + while let Some(directory) = pending.pop() { + let entries = fs::read_dir(&directory) + .map_err(|error| format!("list {}: {error}", directory.display()))?; + for entry in entries { + let path = entry + .map_err(|error| format!("list {}: {error}", directory.display()))? + .path(); + let file_type = fs::symlink_metadata(&path) + .map_err(|error| format!("inspect {}: {error}", path.display()))? + .file_type(); + if file_type.is_dir() { + pending.push(path); + } else if file_type.is_file() { + let bytes = + fs::read(&path).map_err(|error| format!("read {}: {error}", path.display()))?; + if bytes.windows(needle.len()).any(|window| window == needle) { + hits.push(path); + } + } + } + } + Ok(hits) +} + async fn wait_for_lifecycle_runtime( resources: &TestResources, client: &mut ApplicationClient, @@ -3055,7 +3445,7 @@ graph: - bindingName: "data" mountPath: "/data" kind: "STORAGE_KIND_DATA" - scope: "STORAGE_SCOPE_DEPLOYMENT" + placements: ["STORAGE_PLACEMENT_DEPLOYMENT"] readiness: httpGet: portName: "http" @@ -3069,12 +3459,12 @@ graph: ) } - fn write_storage_scopes_application(&self, artifact: &str) -> Result<(), String> { + fn write_storage_placements_application(&self, artifact: &str) -> Result<(), String> { self.write_site_catalog_application( - "synthetic-storage-scopes.yaml", + "synthetic-storage-placements.yaml", &format!( - r#"applicationId: "{STORAGE_SCOPES_APPLICATION_ID}" -displayName: "Synthetic storage scopes proof" + r#"applicationId: "{STORAGE_PLACEMENTS_APPLICATION_ID}" +displayName: "Synthetic storage placements proof" version: "1.0.0" graph: services: @@ -3090,11 +3480,11 @@ graph: - bindingName: "data" mountPath: "/data" kind: "STORAGE_KIND_DATA" - scope: "STORAGE_SCOPE_DEPLOYMENT" + placements: ["STORAGE_PLACEMENT_DEPLOYMENT"] - bindingName: "models" mountPath: "/data/models" kind: "STORAGE_KIND_DATA" - scope: "STORAGE_SCOPE_APPLICATION" + placements: ["STORAGE_PLACEMENT_APPLICATION", "STORAGE_PLACEMENT_DEPLOYMENT"] readiness: httpGet: portName: "http" @@ -3240,6 +3630,70 @@ graph: ) } + /// Declares a one-service application with a text username, a secret password and an + /// outcome choice. Its unit prints the password, then serves, or exits when told to. + fn write_secret_application(&self) -> Result<(), String> { + self.write_site_catalog_application( + "secret-fixture.yaml", + &format!( + r#"applicationId: "{SECRET_APPLICATION_ID}" +displayName: "Secret fixture" +version: "1.0.0" +graph: + parameters: + - parameterId: "username" + displayName: "Username" + visibility: "PARAMETER_VISIBILITY_BASIC" + text: + minimumLength: 1 + maximumLength: 64 + defaultValue: "admin" + - parameterId: "password" + displayName: "Password" + visibility: "PARAMETER_VISIBILITY_BASIC" + secret: + generatedLength: 32 + minimumLength: 12 + - parameterId: "outcome" + displayName: "Outcome" + visibility: "PARAMETER_VISIBILITY_ADVANCED" + stringEnum: + values: ["serve", "exit"] + defaultValue: "serve" + services: + - serviceName: "web" + artifact: +{artifact} command: ["sh", "-ec"] + arguments: ["trap 'exit 0' TERM; printf 'login %s %s\\n' \"$USERNAME\" \"$PASSWORD\"; if [ \"$OUTCOME\" = exit ]; then printf 'stopping with %s\\n' \"$PASSWORD\" >&2; exit 3; fi; mkdir -p /www; printf 'ready\\n' > /www/index.html; httpd -f -p 8080 -h /www & wait $!"] + configurationBindings: + - bindingName: "username" + environmentVariable: "USERNAME" + parameterId: "username" + - bindingName: "outcome" + environmentVariable: "OUTCOME" + parameterId: "outcome" + secretBindings: + - bindingName: "password" + environmentVariable: "PASSWORD" + parameterId: "password" + form: "APPLICATION_SECRET_FORM_PLAIN" + ports: + - portName: "http" + portNumber: 8080 + readiness: + httpGet: + portName: "http" + path: "/" + primaryEndpoint: + serviceName: "web" + portName: "http" + protocol: "APPLICATION_ENDPOINT_PROTOCOL_HTTP" +"#, + artifact = remote_image_artifact(), + ), + ) + } + fn write_site_catalog_application( &self, file_name: &str, diff --git a/backend/core/tests/gated/nvidia_docker_integration.rs b/backend/core/tests/gated/nvidia_docker_integration.rs index 175c8b32..31f3ba9e 100644 --- a/backend/core/tests/gated/nvidia_docker_integration.rs +++ b/backend/core/tests/gated/nvidia_docker_integration.rs @@ -491,6 +491,7 @@ fn qualification_case( &target, &inventory, None, + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator), )?; let snapshot = ModelDeploymentSnapshot { source: Some( diff --git a/backend/core/tests/gated/nvidia_qualification.rs b/backend/core/tests/gated/nvidia_qualification.rs index b048dd42..a49ddc0e 100644 --- a/backend/core/tests/gated/nvidia_qualification.rs +++ b/backend/core/tests/gated/nvidia_qualification.rs @@ -46,8 +46,8 @@ use dotcore_node_hardware::hardware::{ use dotcore_proto::dotcore::{ capacity::v1::{MemoryPoolKind, MemorySegmentSource}, catalog::v1::{ - EngineArtifact, EngineParameterValue, engine_artifact::Kind as EngineArtifactKind, - engine_parameter_value, nix_artifact::Provides, + EngineArtifact, ParameterValue, engine_artifact::Kind as EngineArtifactKind, + nix_artifact::Provides, parameter_value, }, deployment::v1::{ ArtifactPreparationPolicy, ExactPlacement, NodeResolvedLaunchIdentity, @@ -959,9 +959,9 @@ fn configuration_at( values .values .retain(|value| value.parameter_id != "total-context"); - values.values.push(EngineParameterValue { + values.values.push(ParameterValue { parameter_id: "total-context".to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(u64::from( + value: Some(parameter_value::Value::UnsignedInteger(u64::from( total_context, ))), }); diff --git a/backend/core/tests/gated/vllm_nvidia_docker_integration.rs b/backend/core/tests/gated/vllm_nvidia_docker_integration.rs index 6ec16978..1a57769f 100644 --- a/backend/core/tests/gated/vllm_nvidia_docker_integration.rs +++ b/backend/core/tests/gated/vllm_nvidia_docker_integration.rs @@ -734,6 +734,7 @@ fn qualification_case( &execution_target, &target_inventory, None, + dotcore::model::resolution::EngineSecrets::Generate(&dotcore::RandomSecretGenerator), ) .with_context(|| format!("resolve vLLM qualification profile {profile_id:?}"))?; let plan = snapshot diff --git a/backend/core/tests/multi_node/node_rejoin.rs b/backend/core/tests/multi_node/node_rejoin.rs index 8b2888e3..c69f717d 100644 --- a/backend/core/tests/multi_node/node_rejoin.rs +++ b/backend/core/tests/multi_node/node_rejoin.rs @@ -63,7 +63,7 @@ graph: arguments: - "trap 'exit 0' TERM; mkdir -p /www; printf 'rejoin-ready' > /www/index.html; httpd -f -p 8080 -h /www & wait $!" ports: [{ portName: "http", portNumber: 8080 }] - storageBindings: [{ bindingName: "data", mountPath: "/data", kind: "STORAGE_KIND_DATA", scope: "STORAGE_SCOPE_DEPLOYMENT" }] + storageBindings: [{ bindingName: "data", mountPath: "/data", kind: "STORAGE_KIND_DATA", placements: ["STORAGE_PLACEMENT_DEPLOYMENT"] }] readiness: { httpGet: { portName: "http", path: "/" } } primaryEndpoint: serviceName: "web" @@ -139,6 +139,7 @@ async fn hw_docker_worker_application_is_relaunched_at_the_new_address_with_its_ expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("manager accepts the Docker application") diff --git a/backend/core/tests/multi_node/remote_application_lifecycle.rs b/backend/core/tests/multi_node/remote_application_lifecycle.rs index 23ceccf9..a67a7cb4 100644 --- a/backend/core/tests/multi_node/remote_application_lifecycle.rs +++ b/backend/core/tests/multi_node/remote_application_lifecycle.rs @@ -97,6 +97,7 @@ async fn manager_controls_a_worker_hosted_application_across_restarts_and_discon expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("manager accepts a worker application target") @@ -425,6 +426,7 @@ async fn a_worker_hosted_application_relaunches_when_its_linked_route_is_renamed expected_inference_route_generation: route.generation, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("manager accepts a route-linked worker application") @@ -540,6 +542,7 @@ async fn a_split_network_worker_application_is_reached_at_the_declared_worker_ad expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("manager accepts a worker application target") @@ -671,6 +674,7 @@ async fn a_worker_follows_pushed_views_and_its_closed_stream_disconnects_it_at_o expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("manager accepts a worker application target") @@ -828,6 +832,7 @@ async fn the_resource_watch_shows_a_worker_deployment_status_and_its_closed_stre expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("manager accepts a worker application target") @@ -919,6 +924,7 @@ async fn a_worker_pins_a_nix_application_on_its_simulated_target_and_reports_the expected_inference_route_generation: 0, application_deployment_name: None, parameter_values: Vec::new(), + storage_choices: Vec::new(), }) .await .expect("manager accepts a Nix application on the worker") @@ -1021,7 +1027,7 @@ graph: input: inferenceRoute: projection: "APPLICATION_INFERENCE_ROUTE_PROJECTION_ROUTE_NAME" - storageBindings: [{ bindingName: "data", mountPath: "/data", kind: "STORAGE_KIND_DATA", scope: "STORAGE_SCOPE_DEPLOYMENT" }] + storageBindings: [{ bindingName: "data", mountPath: "/data", kind: "STORAGE_KIND_DATA", placements: ["STORAGE_PLACEMENT_DEPLOYMENT"] }] readiness: { httpGet: { portName: "http", path: "/health" } } primaryEndpoint: serviceName: "web" diff --git a/backend/core/tests/multi_node/remote_model_lifecycle.rs b/backend/core/tests/multi_node/remote_model_lifecycle.rs index 07bf820a..f0aafb98 100644 --- a/backend/core/tests/multi_node/remote_model_lifecycle.rs +++ b/backend/core/tests/multi_node/remote_model_lifecycle.rs @@ -1645,18 +1645,19 @@ parameterSchema: parameters: - parameterId: "total-context" displayName: "Total context" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } - parameterId: "context-per-request" displayName: "Context per request" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens" } - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true + visibility: "PARAMETER_VISIBILITY_ADVANCED" unsignedInteger: { minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" } - parameterId: "prompt-cache-memory-mib" displayName: "Prompt cache memory" + visibility: "PARAMETER_VISIBILITY_HIDDEN" unsignedInteger: { minimum: "0", maximum: "65536", unit: "MiB" } artifacts: - artifactId: "fixture-dev-linux" diff --git a/backend/core/tests/support/mod.rs b/backend/core/tests/support/mod.rs index a38c2c73..3bdc577f 100644 --- a/backend/core/tests/support/mod.rs +++ b/backend/core/tests/support/mod.rs @@ -35,9 +35,7 @@ use dotcore_domain::{ use dotcore_node_hardware::MachineDescription; use dotcore_proto::{ dotcore::{ - catalog::v1::{ - EngineParameterValue, EngineParameterValues, OciImageReference, engine_parameter_value, - }, + catalog::v1::{OciImageReference, ParameterValue, ParameterValues, parameter_value}, inference::v1::{ListRoutesRequest, route_service_client::RouteServiceClient}, model::v1::{ CreateLocalModelDeploymentRequest, CreateModelDeploymentRequest, DirectRouteExposure, @@ -142,16 +140,16 @@ pub fn local_model_configuration( ) -> dotcore_proto::dotcore::model::v1::LocalModelDeploymentConfiguration { dotcore_proto::dotcore::model::v1::LocalModelDeploymentConfiguration { model_artifact_binding_id: binding_id.to_owned(), - engine_parameter_values: Some(EngineParameterValues { + engine_parameter_values: Some(ParameterValues { values: [ ("total-context", total_context), ("prompt-cache-memory-mib", 0), ("parallel-requests", 4), ] .into_iter() - .map(|(parameter_id, value)| EngineParameterValue { + .map(|(parameter_id, value)| ParameterValue { parameter_id: parameter_id.to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(value)), + value: Some(parameter_value::Value::UnsignedInteger(value)), }) .collect(), }), @@ -2918,18 +2916,19 @@ parameterSchema: parameters: - parameterId: "total-context" displayName: "Total context" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } - parameterId: "context-per-request" displayName: "Context per request" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens" } - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true + visibility: "PARAMETER_VISIBILITY_ADVANCED" unsignedInteger: { minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" } - parameterId: "prompt-cache-memory-mib" displayName: "Prompt cache memory" + visibility: "PARAMETER_VISIBILITY_HIDDEN" unsignedInteger: { minimum: "0", maximum: "65536", unit: "MiB" } artifacts: - artifactId: "fixture-metal" diff --git a/backend/domain/src/test_support/model.rs b/backend/domain/src/test_support/model.rs index acbbb95f..dd3d84bf 100644 --- a/backend/domain/src/test_support/model.rs +++ b/backend/domain/src/test_support/model.rs @@ -3,12 +3,12 @@ use dotcore_proto::dotcore::{ catalog::v1::{ - EngineArtifact, EngineParameterDefinition, EngineParameterSchema, EngineParameterValue, - EngineParameterValues, HardwareRequirements, HuggingFaceArtifactSource, + EngineArtifact, EngineParameterSchema, HardwareRequirements, HuggingFaceArtifactSource, HuggingFaceFileArtifact, InferenceRequestKind, ModelArtifact, ModelExecutionCapabilities, - ModelInputCapabilities, ModelInputModality, ModelParallelShape, OciPlatform, Selector, - UnsignedIntegerEngineParameterSchema, engine_parameter_definition, engine_parameter_value, - hugging_face_artifact_source, + ModelInputCapabilities, ModelInputModality, ModelParallelShape, OciPlatform, + ParameterDefinition, ParameterValue, ParameterValues, ParameterVisibility, Selector, + UnsignedIntegerParameterSchema, hugging_face_artifact_source, parameter_definition, + parameter_value, }, deployment::v1::ArtifactPreparationPolicy, engine::{ @@ -159,6 +159,8 @@ pub fn local_model_snapshot(model_id: &str) -> ModelDeploymentSnapshot { ), ], }), + profile_engine_parameter_defaults: None, + profile_engine_parameter_visibility_overrides: Vec::new(), custom_argument_tokens: Vec::new(), resolved_options: Vec::new(), selected_target: Some(dotcore_proto::dotcore::model::v1::ResolvedSelectedTarget { @@ -217,13 +219,13 @@ pub fn local_model_configuration() -> LocalModelDeploymentConfiguration { } } -fn unsigned_values(values: &[(&str, u64)]) -> EngineParameterValues { - EngineParameterValues { +fn unsigned_values(values: &[(&str, u64)]) -> ParameterValues { + ParameterValues { values: values .iter() - .map(|(parameter_id, value)| EngineParameterValue { + .map(|(parameter_id, value)| ParameterValue { parameter_id: (*parameter_id).to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(*value)), + value: Some(parameter_value::Value::UnsignedInteger(*value)), }) .collect(), } @@ -255,19 +257,22 @@ fn unsigned_parameter( minimum: u64, maximum: u64, unit: &str, -) -> EngineParameterDefinition { - EngineParameterDefinition { +) -> ParameterDefinition { + ParameterDefinition { parameter_id: parameter_id.to_owned(), display_name: display_name.to_owned(), - parameter_type: Some(engine_parameter_definition::ParameterType::UnsignedInteger( - UnsignedIntegerEngineParameterSchema { + parameter_type: Some(parameter_definition::ParameterType::UnsignedInteger( + UnsignedIntegerParameterSchema { minimum, maximum, unit: unit.to_owned(), default_value: None, }, )), - exposed: true, + visibility: match parameter_id { + "total-context" | "context-per-request" => ParameterVisibility::Basic, + _ => ParameterVisibility::Advanced, + } as i32, description: String::new(), } } diff --git a/backend/fixtures/development-catalog/engines/fixture.yaml b/backend/fixtures/development-catalog/engines/fixture.yaml index e883bed8..57b26b5a 100644 --- a/backend/fixtures/development-catalog/engines/fixture.yaml +++ b/backend/fixtures/development-catalog/engines/fixture.yaml @@ -6,30 +6,31 @@ parameterSchema: parameters: - parameterId: "total-context" displayName: "Total context" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } - parameterId: "context-per-request" displayName: "Context per request" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens" } - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true + visibility: "PARAMETER_VISIBILITY_ADVANCED" unsignedInteger: { minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" } - parameterId: "prompt-cache-memory-mib" displayName: "Prompt cache memory" + visibility: "PARAMETER_VISIBILITY_HIDDEN" unsignedInteger: { minimum: "0", maximum: "65536", unit: "MiB" } storageBindings: - bindingName: "compile-cache" mountPath: "/var/cache/compile" kind: "STORAGE_KIND_CACHE" - scope: "STORAGE_SCOPE_DEPLOYMENT" + placements: ["STORAGE_PLACEMENT_DEPLOYMENT"] environment: - { name: "COMPILE_CACHE" } - bindingName: "kernel-cache" mountPath: "/var/cache/kernels" kind: "STORAGE_KIND_CACHE" - scope: "STORAGE_SCOPE_ENGINE_ARTIFACT" + placements: ["STORAGE_PLACEMENT_ENGINE_ARTIFACT"] environment: - { name: "KERNEL_CACHE" } artifacts: diff --git a/backend/fixtures/recommendation-catalog/engines/llama.yaml b/backend/fixtures/recommendation-catalog/engines/llama.yaml index d8fedf34..8fba6adb 100644 --- a/backend/fixtures/recommendation-catalog/engines/llama.yaml +++ b/backend/fixtures/recommendation-catalog/engines/llama.yaml @@ -4,10 +4,10 @@ version: "1" parameterSchema: schemaId: "llama-v2" parameters: - - { parameterId: "total-context", displayName: "Total context", exposed: true, unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } } - - { parameterId: "context-per-request", displayName: "Context per request", exposed: true, unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens" } } - - { parameterId: "parallel-requests", displayName: "Parallel requests", exposed: true, unsignedInteger: { minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" } } - - { parameterId: "prompt-cache-memory-mib", displayName: "Prompt cache memory", unsignedInteger: { minimum: "0", maximum: "65536", unit: "MiB" } } + - { parameterId: "total-context", displayName: "Total context", visibility: "PARAMETER_VISIBILITY_BASIC", unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } } + - { parameterId: "context-per-request", displayName: "Context per request", visibility: "PARAMETER_VISIBILITY_BASIC", unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens" } } + - { parameterId: "parallel-requests", displayName: "Parallel requests", visibility: "PARAMETER_VISIBILITY_ADVANCED", unsignedInteger: { minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" } } + - { parameterId: "prompt-cache-memory-mib", displayName: "Prompt cache memory", visibility: "PARAMETER_VISIBILITY_HIDDEN", unsignedInteger: { minimum: "0", maximum: "65536", unit: "MiB" } } artifacts: - artifactId: "linux-amd64" oci: diff --git a/backend/fixtures/simulated-docker-site/catalog/engines/fixture.yaml b/backend/fixtures/simulated-docker-site/catalog/engines/fixture.yaml index c71c03dc..7d8a5cb4 100644 --- a/backend/fixtures/simulated-docker-site/catalog/engines/fixture.yaml +++ b/backend/fixtures/simulated-docker-site/catalog/engines/fixture.yaml @@ -6,18 +6,19 @@ parameterSchema: parameters: - parameterId: "total-context" displayName: "Total context" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } - parameterId: "context-per-request" displayName: "Context per request" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens" } - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true + visibility: "PARAMETER_VISIBILITY_ADVANCED" unsignedInteger: { minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" } - parameterId: "prompt-cache-memory-mib" displayName: "Prompt cache memory" + visibility: "PARAMETER_VISIBILITY_HIDDEN" unsignedInteger: { minimum: "0", maximum: "65536", unit: "MiB" } artifacts: - artifactId: "fixture-linux" diff --git a/backend/fixtures/simulated-site/catalog/engines/fixture.yaml b/backend/fixtures/simulated-site/catalog/engines/fixture.yaml index c71c03dc..7d8a5cb4 100644 --- a/backend/fixtures/simulated-site/catalog/engines/fixture.yaml +++ b/backend/fixtures/simulated-site/catalog/engines/fixture.yaml @@ -6,18 +6,19 @@ parameterSchema: parameters: - parameterId: "total-context" displayName: "Total context" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } - parameterId: "context-per-request" displayName: "Context per request" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens" } - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true + visibility: "PARAMETER_VISIBILITY_ADVANCED" unsignedInteger: { minimum: "1", maximum: "64", unit: "requests", defaultValue: "4" } - parameterId: "prompt-cache-memory-mib" displayName: "Prompt cache memory" + visibility: "PARAMETER_VISIBILITY_HIDDEN" unsignedInteger: { minimum: "0", maximum: "65536", unit: "MiB" } artifacts: - artifactId: "fixture-linux" diff --git a/backend/fixtures/sizing-fit/catalog/engines/llama-cpp-linux-2026-10-19.yaml b/backend/fixtures/sizing-fit/catalog/engines/llama-cpp-linux-2026-10-19.yaml index 38aeae74..147f0e2d 100644 --- a/backend/fixtures/sizing-fit/catalog/engines/llama-cpp-linux-2026-10-19.yaml +++ b/backend/fixtures/sizing-fit/catalog/engines/llama-cpp-linux-2026-10-19.yaml @@ -7,18 +7,19 @@ parameterSchema: parameters: - parameterId: "total-context" displayName: "Total context" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens", defaultValue: "32768" } - parameterId: "context-per-request" displayName: "Context per request" - exposed: true + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1024", maximum: "16777216", unit: "tokens" } - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true + visibility: "PARAMETER_VISIBILITY_ADVANCED" unsignedInteger: { minimum: "1", maximum: "64", unit: "requests", defaultValue: "1" } - parameterId: "prompt-cache-memory-mib" displayName: "Prompt cache memory" + visibility: "PARAMETER_VISIBILITY_HIDDEN" unsignedInteger: { minimum: "0", maximum: "65536", unit: "MiB" } artifacts: - artifactId: "cuda13" diff --git a/backend/proto/tests/application_contract.rs b/backend/proto/tests/application_contract.rs index 8a60aeeb..c657dac0 100644 --- a/backend/proto/tests/application_contract.rs +++ b/backend/proto/tests/application_contract.rs @@ -210,6 +210,7 @@ fn generated_bindings_preserve_application_graph_and_deployment_state() { ), ), }), + enabled_by_parameter_id: None, } .source, Some(application_configuration_binding::Source::LiteralValue( @@ -445,6 +446,17 @@ fn application_descriptors_define_portable_graph_and_owner_separated_records() { "resolved_value_name", "parameter_id", "input", + "enabled_by_parameter_id", + ] + ); + assert_eq!( + field_names(catalog, "ApplicationSecretBinding"), + [ + "binding_name", + "environment_variable", + "parameter_id", + "form", + "enabled_by_parameter_id", ] ); assert_eq!( @@ -471,8 +483,21 @@ fn application_descriptors_define_portable_graph_and_owner_separated_records() { "expected_inference_route_generation", "application_deployment_name", "parameter_values", + "storage_choices", ] ); + assert_eq!( + field_names(service, "ApplicationStorageChoice"), + ["service_name", "binding_name", "placement"] + ); + assert_eq!( + field_type_name(service, "ApplicationStorageChoice", "placement"), + ".dotcore.catalog.v1.StoragePlacement" + ); + assert_eq!( + field_names(deployment, "ApplicationStorageAttachment"), + ["service_name", "binding_name", "storage_name", "placement"] + ); assert_eq!( field_names(deployment, "ApplicationDeploymentDeclaration"), [ @@ -483,6 +508,7 @@ fn application_descriptors_define_portable_graph_and_owner_separated_records() { "desired_state", "storage_attachments", "configuration_attachments", + "secret_attachments", "inference_route_id", "lifecycle_generation", "node_resolved_launch_identities", @@ -580,6 +606,10 @@ fn application_descriptors_define_portable_graph_and_owner_separated_records() { !field_names(deployment, "ApplicationDeployment").contains(&"configuration_attachments"), "resolved configuration must remain private desired state" ); + assert!( + !field_names(deployment, "ApplicationDeployment").contains(&"secret_attachments"), + "secret attachments must remain private desired state" + ); assert_eq!( field_names(current, "SiteCurrentState"), [ diff --git a/backend/proto/tests/engine_contract.rs b/backend/proto/tests/engine_contract.rs index fe97e8b0..3e52474d 100644 --- a/backend/proto/tests/engine_contract.rs +++ b/backend/proto/tests/engine_contract.rs @@ -282,6 +282,8 @@ fn descriptors_keep_engine_payloads_out_of_catalog_and_deployment_contracts() { ("engine_parameter_values", 13), ("engine_parameter_schema_id", 14), ("engine_parameter_schema", 15), + ("profile_engine_parameter_defaults", 23), + ("profile_engine_parameter_visibility_overrides", 24), ("desired_launch_plan", 16), ("custom_argument_tokens", 17), ("resolved_options", 18), diff --git a/backend/proto/tests/model_contract.rs b/backend/proto/tests/model_contract.rs index e7d72a5c..b9926109 100644 --- a/backend/proto/tests/model_contract.rs +++ b/backend/proto/tests/model_contract.rs @@ -15,20 +15,20 @@ use dotcore_proto::{ }, catalog::v1::{ ApplicationServiceArtifact, BooleanModelOptionParameterSchema, EngineArtifact, - EngineArtifactReference, EngineParameterDefinition, EngineParameterSchema, - EngineParameterValue, EngineParameterValues, EngineRelease, Model, ModelArtifact, + EngineArtifactReference, EngineParameterSchema, EngineRelease, Model, ModelArtifact, ModelArtifactBinding, ModelAuxiliaryArtifact, ModelConfigurationOption, ModelEngineConfiguration, ModelExecutionCapabilities, ModelInputCapabilities, ModelInputModality, ModelOption, ModelOptionParameterDefinition, ModelOptionParameterValue, ModelOptionValues, ModelParallelShape, ModelProfile, - OciImageReference, OciPlatform, Selector, StringEnumModelOptionParameterSchema, - UnsignedIntegerEngineParameterSchema, UnsignedIntegerModelOptionParameterSchema, - VisionModelOption, engine_artifact, - engine_parameter_definition::ParameterType as EngineParameterType, - engine_parameter_value::Value as EngineParameterValueKind, + OciImageReference, OciPlatform, ParameterDefinition, ParameterValue, ParameterValues, + ParameterVisibility, ParameterVisibilityOverride, Selector, + StringEnumModelOptionParameterSchema, UnsignedIntegerModelOptionParameterSchema, + UnsignedIntegerParameterSchema, VisionModelOption, engine_artifact, model_option::Kind as ModelOptionKind, model_option_parameter_definition::ParameterType as ModelOptionParameterType, model_option_parameter_value::Value as ModelOptionValueKind, + parameter_definition::ParameterType as EngineParameterType, + parameter_value::Value as EngineParameterValueKind, }, deployment::v1::{ArtifactPreparationPolicy, MemoryFit}, engine::{ @@ -238,8 +238,8 @@ fn generated_bindings_preserve_typed_status_and_byte_progress() { #[test] fn generated_bindings_preserve_exact_ordered_custom_argument_tokens_in_preview_and_snapshot() { - let context_values = EngineParameterValues { - values: vec![EngineParameterValue { + let context_values = ParameterValues { + values: vec![ParameterValue { parameter_id: "total-context".to_owned(), value: Some(EngineParameterValueKind::UnsignedInteger(30_000)), }], @@ -337,21 +337,23 @@ fn generated_bindings_preserve_exact_ordered_custom_argument_tokens_in_preview_a engine_parameter_schema_id: "llama-cpp-b10156-v1".to_owned(), engine_parameter_schema: Some(EngineParameterSchema { schema_id: "llama-cpp-b10156-v1".to_owned(), - parameters: vec![EngineParameterDefinition { + parameters: vec![ParameterDefinition { parameter_id: "total-context".to_owned(), display_name: "Total context".to_owned(), parameter_type: Some(EngineParameterType::UnsignedInteger( - UnsignedIntegerEngineParameterSchema { + UnsignedIntegerParameterSchema { minimum: 1, maximum: 131_072, unit: "tokens".to_owned(), default_value: None, }, )), - exposed: true, + visibility: ParameterVisibility::Basic as i32, description: "Maximum prompt and generation context.".to_owned(), }], }), + profile_engine_parameter_defaults: None, + profile_engine_parameter_visibility_overrides: Vec::new(), custom_argument_tokens: vec![ "--reasoning-format".to_owned(), "deepseek".to_owned(), @@ -427,8 +429,8 @@ fn generated_bindings_preserve_exact_ordered_custom_argument_tokens_in_preview_a fn generated_bindings_preserve_minimal_generic_resource_estimates() { let configuration = dotcore_proto::dotcore::model::v1::LocalModelDeploymentConfiguration { model_artifact_binding_id: "q4-k-m".to_owned(), - engine_parameter_values: Some(EngineParameterValues { - values: vec![EngineParameterValue { + engine_parameter_values: Some(ParameterValues { + values: vec![ParameterValue { parameter_id: "total-context".to_owned(), value: Some(EngineParameterValueKind::UnsignedInteger(30_000)), }], @@ -466,8 +468,8 @@ fn generated_bindings_preserve_minimal_generic_resource_estimates() { unit: "bytes".to_owned(), }, ], - derived_engine_parameter_values: Some(EngineParameterValues { - values: vec![EngineParameterValue { + derived_engine_parameter_values: Some(ParameterValues { + values: vec![ParameterValue { parameter_id: "kv-cache-memory-bytes".to_owned(), value: Some(EngineParameterValueKind::UnsignedInteger(5_443_884_300)), }], @@ -532,18 +534,18 @@ fn generated_bindings_preserve_minimal_generic_resource_estimates() { fn generated_bindings_round_trip_model_configurations_and_typed_context_schema() { let schema = EngineParameterSchema { schema_id: "llama-cpp-b10156-v1".to_owned(), - parameters: vec![EngineParameterDefinition { + parameters: vec![ParameterDefinition { parameter_id: "total-context".to_owned(), display_name: "Total context".to_owned(), parameter_type: Some(EngineParameterType::UnsignedInteger( - UnsignedIntegerEngineParameterSchema { + UnsignedIntegerParameterSchema { minimum: 512, maximum: 131_072, unit: "tokens".to_owned(), default_value: None, }, )), - exposed: true, + visibility: ParameterVisibility::Basic as i32, description: "Maximum prompt and generation context.".to_owned(), }], }; @@ -581,8 +583,8 @@ fn generated_bindings_round_trip_model_configurations_and_typed_context_schema() profile_id: "fast-start".to_owned(), configuration_id: "cuda".to_owned(), default_model_artifact_binding_id: "q4-k-m".to_owned(), - engine_parameter_defaults: Some(EngineParameterValues { - values: vec![EngineParameterValue { + engine_parameter_defaults: Some(ParameterValues { + values: vec![ParameterValue { parameter_id: "total-context".to_owned(), value: Some(EngineParameterValueKind::UnsignedInteger(30_000)), }], @@ -590,7 +592,10 @@ fn generated_bindings_round_trip_model_configurations_and_typed_context_schema() display_name: "Fast start".to_owned(), description: "Balances model quality and startup speed with Q4 weights.".to_owned(), option_defaults: Vec::new(), - exposed_parameter_ids: vec!["total-context".to_owned()], + engine_parameter_visibility_overrides: vec![ParameterVisibilityOverride { + parameter_id: "total-context".to_owned(), + visibility: ParameterVisibility::Hidden as i32, + }], }; let model = Model { model_id: "gemma-4-e2b".to_owned(), @@ -867,6 +872,7 @@ parameterSchema: parameters: - parameterId: total-context displayName: Total context + visibility: "PARAMETER_VISIBILITY_BASIC" unsignedInteger: { minimum: "1", maximum: "2", unit: tokens } "#, ) @@ -928,6 +934,7 @@ fn model_descriptors_define_typed_configuration_and_resolved_plan_contracts() { .expect("the generated descriptor set must decode"); let application = descriptor_file(&descriptor, "dotcore/catalog/v1/application.proto"); let engine = descriptor_file(&descriptor, "dotcore/catalog/v1/engine_release.proto"); + let parameter = descriptor_file(&descriptor, "dotcore/catalog/v1/parameter.proto"); let nix = descriptor_file(&descriptor, "dotcore/catalog/v1/nix.proto"); let selector = descriptor_file(&descriptor, "dotcore/catalog/v1/selector.proto"); let catalog = descriptor_file(&descriptor, "dotcore/catalog/v1/model.proto"); @@ -996,34 +1003,42 @@ fn model_descriptors_define_typed_configuration_and_resolved_plan_contracts() { field_layout(engine, "EngineParameterSchema"), [ ("schema_id", 1, "TYPE_STRING"), - ( - "parameters", - 2, - ".dotcore.catalog.v1.EngineParameterDefinition" - ), + ("parameters", 2, ".dotcore.catalog.v1.ParameterDefinition"), ] ); assert_eq!( - field_layout(engine, "EngineParameterDefinition"), + field_layout(parameter, "ParameterDefinition"), [ ("parameter_id", 1, "TYPE_STRING"), ("display_name", 2, "TYPE_STRING"), + ("description", 3, "TYPE_STRING"), + ("visibility", 7, ".dotcore.catalog.v1.ParameterVisibility"), + ("boolean", 4, ".dotcore.catalog.v1.BooleanParameterSchema"), ( "unsigned_integer", - 3, - ".dotcore.catalog.v1.UnsignedIntegerEngineParameterSchema" + 5, + ".dotcore.catalog.v1.UnsignedIntegerParameterSchema" ), ( - "boolean", - 4, - ".dotcore.catalog.v1.BooleanEngineParameterSchema", + "string_enum", + 6, + ".dotcore.catalog.v1.StringEnumParameterSchema" ), - ("exposed", 5, "TYPE_BOOL"), - ("description", 6, "TYPE_STRING"), + ("text", 8, ".dotcore.catalog.v1.TextParameterSchema"), + ("secret", 9, ".dotcore.catalog.v1.SecretParameterSchema"), ] ); assert_eq!( - field_numbers(engine, "UnsignedIntegerEngineParameterSchema"), + enum_values(parameter, "ParameterVisibility"), + [ + ("PARAMETER_VISIBILITY_UNSPECIFIED", 0), + ("PARAMETER_VISIBILITY_BASIC", 1), + ("PARAMETER_VISIBILITY_ADVANCED", 2), + ("PARAMETER_VISIBILITY_HIDDEN", 3), + ] + ); + assert_eq!( + field_numbers(parameter, "UnsignedIntegerParameterSchema"), [ ("minimum", 1), ("maximum", 2), @@ -1032,11 +1047,16 @@ fn model_descriptors_define_typed_configuration_and_resolved_plan_contracts() { ] ); assert_eq!( - field_layout(engine, "EngineParameterValue"), + field_layout(parameter, "ParameterValues"), + [("values", 1, ".dotcore.catalog.v1.ParameterValue")] + ); + assert_eq!( + field_layout(parameter, "ParameterValue"), [ ("parameter_id", 1, "TYPE_STRING"), - ("unsigned_integer", 2, "TYPE_UINT64"), - ("boolean", 3, "TYPE_BOOL"), + ("boolean", 2, "TYPE_BOOL"), + ("unsigned_integer", 3, "TYPE_UINT64"), + ("string", 4, "TYPE_STRING"), ] ); @@ -1141,10 +1161,17 @@ fn model_descriptors_define_typed_configuration_and_resolved_plan_contracts() { ("configuration_id", 2), ("default_model_artifact_binding_id", 3), ("engine_parameter_defaults", 4), + ("engine_parameter_visibility_overrides", 9), ("display_name", 5), ("description", 6), ("option_defaults", 7), - ("exposed_parameter_ids", 8), + ] + ); + assert_eq!( + field_layout(catalog, "ParameterVisibilityOverride"), + [ + ("parameter_id", 1, "TYPE_STRING"), + ("visibility", 2, ".dotcore.catalog.v1.ParameterVisibility"), ] ); assert_eq!( @@ -1439,7 +1466,7 @@ fn model_descriptors_define_typed_configuration_and_resolved_plan_contracts() { ( "engine_parameter_values", 3, - ".dotcore.catalog.v1.EngineParameterValues" + ".dotcore.catalog.v1.ParameterValues" ), ( "hardware_class_estimates", @@ -1531,6 +1558,8 @@ fn model_descriptors_define_typed_configuration_and_resolved_plan_contracts() { ("engine_parameter_values", 13), ("engine_parameter_schema_id", 14), ("engine_parameter_schema", 15), + ("profile_engine_parameter_defaults", 23), + ("profile_engine_parameter_visibility_overrides", 24), ("desired_launch_plan", 16), ("custom_argument_tokens", 17), ("resolved_options", 18), diff --git a/backend/proto/tests/site_contract.rs b/backend/proto/tests/site_contract.rs index 5671f938..e2347ab7 100644 --- a/backend/proto/tests/site_contract.rs +++ b/backend/proto/tests/site_contract.rs @@ -19,14 +19,15 @@ use dotcore_proto::{ ApplicationEndpointProtocol, ApplicationGraph, ApplicationHttpReadiness, ApplicationPrimaryEndpoint, ApplicationService, ApplicationServiceArtifact, ApplicationServiceDependency, ApplicationServicePort, ApplicationServiceReadiness, - EngineArtifact, EngineParameterDefinition, EngineParameterSchema, EngineParameterValue, - EngineParameterValues, HuggingFaceArtifactSource, HuggingFaceFileArtifact, - ModelArtifact, ModelExecutionCapabilities, ModelInputCapabilities, ModelInputModality, - ModelParallelShape, OciDigest, OciImageReference, OciPlatform, OciRegistryLocation, - OciRepository, Selector, StorageBinding, StorageEnvironmentVariable, StorageKind, - StorageScope, UnsignedIntegerEngineParameterSchema, application_configuration_binding, + EngineArtifact, EngineParameterSchema, HuggingFaceArtifactSource, + HuggingFaceFileArtifact, ModelArtifact, ModelExecutionCapabilities, + ModelInputCapabilities, ModelInputModality, ModelParallelShape, OciDigest, + OciImageReference, OciPlatform, OciRegistryLocation, OciRepository, + ParameterDefinition, ParameterValue, ParameterValues, ParameterVisibility, Selector, + StorageBinding, StorageEnvironmentVariable, StorageKind, StoragePlacement, + UnsignedIntegerParameterSchema, application_configuration_binding, application_service_artifact, application_service_readiness, engine_artifact, - engine_parameter_definition, engine_parameter_value, hugging_face_artifact_source, + hugging_face_artifact_source, parameter_definition, parameter_value, }, deployment::v1::{ArtifactPreparationPolicy, NodeResolvedLaunchIdentity}, engine::{ @@ -285,6 +286,7 @@ fn complete_site_contract_round_trips_restricted_yaml_and_rejects_unknown_fields service_name: "web".to_owned(), binding_name: "data".to_owned(), storage_name: "web-data".to_owned(), + placement: StoragePlacement::Deployment as i32, }], configuration_attachments: vec![ApplicationConfigurationAttachment { service_name: "web".to_owned(), @@ -292,6 +294,7 @@ fn complete_site_contract_round_trips_restricted_yaml_and_rejects_unknown_fields environment_variable: "ENDPOINT".to_owned(), value: "http://192.0.2.1/v1".to_owned(), }], + secret_attachments: Vec::new(), inference_route_id: None, lifecycle_generation: 4, node_resolved_launch_identities: vec![NodeResolvedLaunchIdentity { @@ -400,7 +403,7 @@ fn complete_model_snapshot() -> ModelDeploymentSnapshot { binding_name: "compile-cache".to_owned(), mount_path: "/cache".to_owned(), kind: StorageKind::Cache as i32, - scope: StorageScope::EngineArtifact as i32, + placements: vec![StoragePlacement::EngineArtifact as i32], environment: vec![StorageEnvironmentVariable { name: "CACHE_HOME".to_owned(), relative_path: "home".to_owned(), @@ -450,21 +453,21 @@ fn complete_model_snapshot() -> ModelDeploymentSnapshot { supported_modalities: vec![ModelInputModality::Text as i32], image_input_capacity: None, }), - engine_parameter_values: Some(EngineParameterValues { - values: vec![EngineParameterValue { + engine_parameter_values: Some(ParameterValues { + values: vec![ParameterValue { parameter_id: "total-context".to_owned(), - value: Some(engine_parameter_value::Value::UnsignedInteger(8_192)), + value: Some(parameter_value::Value::UnsignedInteger(8_192)), }], }), engine_parameter_schema_id: "schema-one".to_owned(), engine_parameter_schema: Some(EngineParameterSchema { schema_id: "schema-one".to_owned(), - parameters: vec![EngineParameterDefinition { + parameters: vec![ParameterDefinition { parameter_id: "total-context".to_owned(), display_name: "Total context".to_owned(), parameter_type: Some( - engine_parameter_definition::ParameterType::UnsignedInteger( - UnsignedIntegerEngineParameterSchema { + parameter_definition::ParameterType::UnsignedInteger( + UnsignedIntegerParameterSchema { minimum: 1, maximum: 131_072, unit: "tokens".to_owned(), @@ -472,10 +475,12 @@ fn complete_model_snapshot() -> ModelDeploymentSnapshot { }, ), ), - exposed: true, + visibility: ParameterVisibility::Basic as i32, description: "Maximum prompt and generation context.".to_owned(), }], }), + profile_engine_parameter_defaults: None, + profile_engine_parameter_visibility_overrides: Vec::new(), custom_argument_tokens: Vec::new(), resolved_options: Vec::new(), selected_target: Some(dotcore_proto::dotcore::model::v1::ResolvedSelectedTarget { @@ -598,7 +603,7 @@ fn complete_application() -> Application { binding_name: "database-data".to_owned(), mount_path: "/var/lib/database".to_owned(), kind: StorageKind::Data as i32, - scope: StorageScope::Deployment as i32, + placements: vec![StoragePlacement::Deployment as i32], environment: Vec::new(), }], secret_bindings: Vec::new(), @@ -642,12 +647,13 @@ fn complete_application() -> Application { "inference-openai-url".to_owned(), )), input: None, + enabled_by_parameter_id: None, }], storage_bindings: vec![StorageBinding { binding_name: "data".to_owned(), mount_path: "/app/data".to_owned(), kind: StorageKind::Data as i32, - scope: StorageScope::Deployment as i32, + placements: vec![StoragePlacement::Deployment as i32], environment: Vec::new(), }], secret_bindings: Vec::new(), diff --git a/backend/runtime/src/workload_group.rs b/backend/runtime/src/workload_group.rs index 1f3b8b63..ec6f9c06 100644 --- a/backend/runtime/src/workload_group.rs +++ b/backend/runtime/src/workload_group.rs @@ -2,6 +2,7 @@ use std::{ collections::{BTreeMap, BTreeSet}, + fmt, net::Ipv4Addr, num::NonZeroU16, time::{Duration, SystemTime}, @@ -245,8 +246,9 @@ pub struct WorkloadGroupLaunch { pub host_aliases: BTreeSet, } -/// One named runtime-neutral OCI unit in a workload group. -#[derive(Clone, Debug, PartialEq, Eq)] +/// One named runtime-neutral OCI unit in a workload group. Its environment may carry secrets, so +/// its `Debug` form names the environment variables without their values. +#[derive(Clone, PartialEq, Eq)] pub struct WorkloadUnitLaunch { /// Exact immutable image identity selected and retained by the owning controller. pub image: String, @@ -264,6 +266,26 @@ pub struct WorkloadUnitLaunch { pub ready_launch_gate: Option, } +impl fmt::Debug for WorkloadUnitLaunch { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("WorkloadUnitLaunch") + .field("image", &self.image) + .field("command", &self.command) + .field("arguments", &self.arguments) + .field("environment", &self.environment.keys().collect::>()) + .field( + "reconfigurable_environment", + &self.reconfigurable_environment, + ) + .field("storage", &self.storage) + .field("ports", &self.ports) + .field("dependencies", &self.dependencies) + .field("ready_launch_gate", &self.ready_launch_gate) + .finish() + } +} + /// One condition on another named unit in the same group. #[derive(Clone, Debug, PartialEq, Eq)] pub struct WorkloadUnitDependency { diff --git a/catalog/applications/comfyui.yaml b/catalog/applications/comfyui.yaml index be1cf7ec..4e499a82 100644 --- a/catalog/applications/comfyui.yaml +++ b/catalog/applications/comfyui.yaml @@ -5,21 +5,22 @@ applicationId: "comfyui" displayName: "ComfyUI" description: "Node-based image and video generation served through an HTTP prompt API." notes: |- - Downloaded models live in storage shared by every ComfyUI deployment on the node, so they survive removing a deployment. Custom nodes, inputs, generated images, and user state stay with the deployment. The editor's missing-models dialog downloads public models from Hugging Face and Civitai on the server; models that need a login are placed by hand. - Deployments on one node share their models, so one deployment can place a model file, including formats that run code when loaded, that another deployment loads. Each deployment already runs as root without a login, so this grants no new authority. + By default, downloaded models are shared by every ComfyUI deployment on the node and survive removing a deployment; choose "This deployment" in Advanced to delete them at teardown. Custom nodes, inputs, generated images, and user state stay with the deployment. The editor's missing-models dialog downloads public models from Hugging Face and Civitai on the server; models that need a login are placed by hand. + With shared models, one deployment can place a model file, including formats that run code when loaded, that another deployment loads. Each already runs as root without a login, so this grants no new authority. The app has no login. Anyone who can reach its published port can run workflows that load any custom node of the deployment, which run as root with the container's network and GPU access. The only access control is the address the node publishes applications on, plus the network in front of it, so publish ComfyUI on a trusted network only. version: "0.42.5" graph: parameters: - - advanced: true - definition: - parameterId: "vram-mode" - displayName: "VRAM mode" - description: "How ComfyUI keeps models in GPU memory. Auto loads and offloads on demand; modes other than auto need a GPU target." - stringEnum: - values: ["auto", "highvram", "gpu-only", "novram"] - defaultValue: "auto" + - parameterId: "vram-mode" + displayName: "VRAM mode" + visibility: "PARAMETER_VISIBILITY_ADVANCED" + description: "How ComfyUI keeps models in GPU memory. Leave it on auto to load and offload on demand; use another mode to force models to stay on the GPU or to save GPU memory, which needs a GPU target." + # The modes `apps/comfyui/entrypoint.sh` maps to ComfyUI's `--highvram`, `--gpu-only` and + # `--novram` flags; `auto` passes none. + stringEnum: + values: ["auto", "highvram", "gpu-only", "novram"] + defaultValue: "auto" services: - serviceName: "comfyui" # The image and its digests are pinned by `apps/comfyui/docker-image.nix`. @@ -48,13 +49,14 @@ graph: - bindingName: "data" mountPath: "/data" kind: "STORAGE_KIND_DATA" - scope: "STORAGE_SCOPE_DEPLOYMENT" - # Downloaded models, nested over the base directory's models folder. Every ComfyUI - # deployment on the node shares this storage, and it outlives each deployment. + placements: ["STORAGE_PLACEMENT_DEPLOYMENT"] + # Downloaded models, nested over the base directory's models folder. By default every + # ComfyUI deployment on the node shares this storage and it outlives each deployment; the + # operator may keep it with the deployment instead. - bindingName: "models" mountPath: "/data/models" kind: "STORAGE_KIND_DATA" - scope: "STORAGE_SCOPE_APPLICATION" + placements: ["STORAGE_PLACEMENT_APPLICATION", "STORAGE_PLACEMENT_DEPLOYMENT"] readiness: httpGet: portName: "http" diff --git a/catalog/applications/hermes.yaml b/catalog/applications/hermes.yaml index 70114612..c1320790 100644 --- a/catalog/applications/hermes.yaml +++ b/catalog/applications/hermes.yaml @@ -5,10 +5,25 @@ displayName: "Hermes" description: "Autonomous AI agent with tools, memory, and a web dashboard that has a chat page." version: "0.2.1" notes: |- - Dashboard login: username admin, password dotcore. Every deployment uses the same login, so it - doesn't protect the dashboard from anyone who can reach it. Chat with the agent on the dashboard's - Chat page. + Log in to the dashboard with the username and password in this deployment's sidebar. Chat with + the agent on the dashboard's Chat page. graph: + parameters: + - parameterId: "dashboard-username" + displayName: "Dashboard username" + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "The username for logging in to the Hermes dashboard." + text: + minimumLength: 1 + maximumLength: 1024 + defaultValue: "admin" + - parameterId: "dashboard-password" + displayName: "Dashboard password" + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "The password for logging in to the Hermes dashboard. Leave it empty to generate a random one, which you can read in the deployment's sidebar." + secret: + generatedLength: 32 + minimumLength: 1 services: - serviceName: "gateway" artifact: @@ -38,10 +53,7 @@ graph: literalValue: "0.0.0.0" - bindingName: "dashboard-login-username" environmentVariable: "HERMES_DASHBOARD_BASIC_AUTH_USERNAME" - literalValue: "admin" - - bindingName: "dashboard-login-password" - environmentVariable: "HERMES_DASHBOARD_BASIC_AUTH_PASSWORD" - literalValue: "dotcore" + parameterId: "dashboard-username" - bindingName: "agent-api-enabled" environmentVariable: "API_SERVER_ENABLED" literalValue: "false" @@ -54,6 +66,11 @@ graph: input: inferenceRoute: projection: "APPLICATION_INFERENCE_ROUTE_PROJECTION_ROUTE_NAME" + secretBindings: + - bindingName: "dashboard-login-password" + environmentVariable: "HERMES_DASHBOARD_BASIC_AUTH_PASSWORD" + parameterId: "dashboard-password" + form: "APPLICATION_SECRET_FORM_PLAIN" readiness: httpGet: portName: "gateway" diff --git a/catalog/applications/n8n.yaml b/catalog/applications/n8n.yaml index 89a47d90..28b0c5a2 100644 --- a/catalog/applications/n8n.yaml +++ b/catalog/applications/n8n.yaml @@ -1,16 +1,33 @@ # Immutable n8n graph running the unmodified upstream image. The command seeds # one OpenAI-compatible credential that points at the configured Inference # endpoint, then starts n8n. The owner account is created from environment -# values with a fixed POC login (admin@dotcore.local / admin). +# values: the email and password come from parameters, and n8n takes the password as a bcrypt hash. applicationId: "n8n" displayName: "n8n" description: "Workflow automation with an Inference endpoint credential already configured." notes: |- - Sign in with admin@dotcore.local / admin (fixed POC login). + Sign in with the email and password in this deployment's sidebar. A Dotcore Inference credential points at this Site's Inference endpoint; pick it in OpenAI nodes. OpenAI node "Message a model": under Options, add Store and turn it off. Dotcore does not keep responses. version: "2.43.5" graph: + parameters: + - parameterId: "owner-email" + displayName: "Owner email" + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "The email address for logging in to n8n as its owner." + # Email addresses run from `a@b` (3 characters) to 254 characters (RFC 5321 path limit). + text: + minimumLength: 3 + maximumLength: 254 + defaultValue: "admin@dotcore.local" + - parameterId: "owner-password" + displayName: "Owner password" + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "The password for logging in to n8n as its owner. Leave it empty to generate a random one, which you can read in the deployment's sidebar." + secret: + generatedLength: 32 + minimumLength: 1 services: - serviceName: "web" # The image and its digests are pinned by `apps/n8n/docker-image.nix`. @@ -51,17 +68,13 @@ graph: literalValue: "true" - bindingName: "owner-email" environmentVariable: "N8N_INSTANCE_OWNER_EMAIL" - literalValue: "admin@dotcore.local" + parameterId: "owner-email" - bindingName: "owner-first-name" environmentVariable: "N8N_INSTANCE_OWNER_FIRST_NAME" literalValue: "Admin" - bindingName: "owner-last-name" environmentVariable: "N8N_INSTANCE_OWNER_LAST_NAME" literalValue: "Dotcore" - # bcrypt (cost 10) of the fixed POC password "admin"; not a secret. - - bindingName: "owner-password-hash" - environmentVariable: "N8N_INSTANCE_OWNER_PASSWORD_HASH" - literalValue: "$2a$10$lb16z8Ce.cqruLzF5coJ1OsaP9zYd7ZZbBgjj6it/cWwnRcKWcjPm" - bindingName: "secure-cookie" environmentVariable: "N8N_SECURE_COOKIE" literalValue: "false" @@ -90,11 +103,17 @@ graph: - bindingName: "editor-base-url" environmentVariable: "N8N_EDITOR_BASE_URL" resolvedValueName: "published-endpoint-url" + secretBindings: + # n8n accepts only a bcrypt hash of the owner password. + - bindingName: "owner-password-hash" + environmentVariable: "N8N_INSTANCE_OWNER_PASSWORD_HASH" + parameterId: "owner-password" + form: "APPLICATION_SECRET_FORM_BCRYPT" storageBindings: - bindingName: "data" mountPath: "/home/node/.n8n" kind: "STORAGE_KIND_DATA" - scope: "STORAGE_SCOPE_DEPLOYMENT" + placements: ["STORAGE_PLACEMENT_DEPLOYMENT"] readiness: httpGet: portName: "http" diff --git a/catalog/applications/open-webui.yaml b/catalog/applications/open-webui.yaml index fc316abc..5ce43f28 100644 --- a/catalog/applications/open-webui.yaml +++ b/catalog/applications/open-webui.yaml @@ -4,7 +4,33 @@ applicationId: "open-webui" displayName: "Open WebUI" description: "Self-hosted chat interface for the models on this site." version: "0.11.4" +notes: |- + Login is off by default: opening the app lands in the chat. When "Require login" is on, sign in + with the admin email and password in this deployment's sidebar. graph: + parameters: + - parameterId: "require-login" + displayName: "Require login" + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "Ask for an email and password to use Open WebUI. The admin email and password below apply only when it is on." + boolean: + defaultValue: false + - parameterId: "admin-email" + displayName: "Admin email" + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "The email address for logging in to Open WebUI as its admin. Applies only when login is required." + # Email addresses run from `a@b` (3 characters) to 254 characters (RFC 5321 path limit). + text: + minimumLength: 3 + maximumLength: 254 + defaultValue: "admin@dotcore.local" + - parameterId: "admin-password" + displayName: "Admin password" + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "The initial password for logging in to Open WebUI as its admin. Leave it empty to generate a random one, which you can read in the deployment's sidebar. Applies only when login is required. If you change the password inside Open WebUI, the one shown here is no longer valid." + secret: + generatedLength: 32 + minimumLength: 1 services: - serviceName: "web" # The image and its digests are pinned by `apps/open-webui/docker-image.nix`. @@ -24,7 +50,14 @@ graph: configurationBindings: - bindingName: "webui-auth" environmentVariable: "WEBUI_AUTH" + parameterId: "require-login" + - bindingName: "signup" + environmentVariable: "ENABLE_SIGNUP" literalValue: "false" + - bindingName: "admin-email" + environmentVariable: "WEBUI_ADMIN_EMAIL" + parameterId: "admin-email" + enabledByParameterId: "require-login" - bindingName: "persistent-config" environmentVariable: "ENABLE_PERSISTENT_CONFIG" literalValue: "false" @@ -40,11 +73,17 @@ graph: - bindingName: "openai-api-base-url" environmentVariable: "OPENAI_API_BASE_URL" resolvedValueName: "configured-inference-openai-base-url" + secretBindings: + - bindingName: "admin-password" + environmentVariable: "WEBUI_ADMIN_PASSWORD" + parameterId: "admin-password" + enabledByParameterId: "require-login" + form: "APPLICATION_SECRET_FORM_PLAIN" storageBindings: - bindingName: "data" mountPath: "/app/backend/data" kind: "STORAGE_KIND_DATA" - scope: "STORAGE_SCOPE_DEPLOYMENT" + placements: ["STORAGE_PLACEMENT_DEPLOYMENT"] readiness: httpGet: portName: "http" diff --git a/catalog/engines/llama-cpp-macos.yaml b/catalog/engines/llama-cpp-macos.yaml index 0e96f028..7b1cbbd6 100644 --- a/catalog/engines/llama-cpp-macos.yaml +++ b/catalog/engines/llama-cpp-macos.yaml @@ -11,8 +11,8 @@ parameterSchema: # Shared parameters: the engine adapter in backend/core/src/model/engine/ maps them to flags. - parameterId: "total-context" displayName: "Total context" - exposed: true - description: "Context memory the server holds from the start and shares among the requests it generates at once. Each active request stores its own tokens in it, even when requests start with the same prompt. When it fills, the server frees idle requests first, then fails the active ones." + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "Memory the model holds for all requests at once; each active request stores its tokens in it, even when prompts start the same. Raise it to serve longer or more simultaneous conversations; when it fills, idle requests are dropped first, then active ones fail." unsignedInteger: minimum: "1024" maximum: "16777216" @@ -20,16 +20,16 @@ parameterSchema: defaultValue: "32768" - parameterId: "context-per-request" displayName: "Context per request" - exposed: true - description: "Largest context one request may use from the total context. It reserves nothing: requests take memory from the total context as they grow. A new request reuses the start of an earlier prompt from an idle request or the prompt cache, but not from requests running at the same time." + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "Largest context one request may use. It reserves nothing: requests draw from the total context as they grow. Lower it to stop one long request from filling the total context." unsignedInteger: minimum: "1024" maximum: "16777216" unit: "tokens" - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true - description: "Requests the server generates at the same time; further requests wait in a queue. All of them share the total context, so each added request only adds its own small state." + visibility: "PARAMETER_VISIBILITY_ADVANCED" + description: "Requests generated at once; more wait in a queue. They share the total context, so raising it adds little memory; raise it when several people or agents use the model together." unsignedInteger: minimum: "1" maximum: "64" @@ -37,8 +37,8 @@ parameterSchema: defaultValue: "4" - parameterId: "prompt-cache-memory-mib" displayName: "Prompt cache memory" - exposed: true - description: "Memory kept to reuse repeated prompts, such as agent system prompts, without processing them again. Idle conversations move here and free their accelerator memory. It uses host memory, which unified-memory machines share with the model; 0 turns the cache off." + visibility: "PARAMETER_VISIBILITY_ADVANCED" + description: "Host memory used to reuse repeated prompts, such as agent system prompts, without processing them again. Raise it if the same long prompts come back often; on unified-memory machines it competes with the model, and 0 turns it off." unsignedInteger: minimum: "0" maximum: "65536" diff --git a/catalog/engines/llama-cpp.yaml b/catalog/engines/llama-cpp.yaml index 1420bc33..762c7ea1 100644 --- a/catalog/engines/llama-cpp.yaml +++ b/catalog/engines/llama-cpp.yaml @@ -10,8 +10,8 @@ parameterSchema: # Shared parameters: the engine adapter in backend/core/src/model/engine/ maps them to flags. - parameterId: "total-context" displayName: "Total context" - exposed: true - description: "Context memory the server holds from the start and shares among the requests it generates at once. Each active request stores its own tokens in it, even when requests start with the same prompt. When it fills, the server frees idle requests first, then fails the active ones." + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "Memory the model holds for all requests at once; each active request stores its tokens in it, even when prompts start the same. Raise it to serve longer or more simultaneous conversations; when it fills, idle requests are dropped first, then active ones fail." unsignedInteger: minimum: "1024" maximum: "16777216" @@ -19,16 +19,16 @@ parameterSchema: defaultValue: "32768" - parameterId: "context-per-request" displayName: "Context per request" - exposed: true - description: "Largest context one request may use from the total context. It reserves nothing: requests take memory from the total context as they grow. A new request reuses the start of an earlier prompt from an idle request or the prompt cache, but not from requests running at the same time." + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "Largest context one request may use. It reserves nothing: requests draw from the total context as they grow. Lower it to stop one long request from filling the total context." unsignedInteger: minimum: "1024" maximum: "16777216" unit: "tokens" - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true - description: "Requests the server generates at the same time; further requests wait in a queue. All of them share the total context, so each added request only adds its own small state." + visibility: "PARAMETER_VISIBILITY_ADVANCED" + description: "Requests generated at once; more wait in a queue. They share the total context, so raising it adds little memory; raise it when several people or agents use the model together." unsignedInteger: minimum: "1" maximum: "64" @@ -36,8 +36,8 @@ parameterSchema: defaultValue: "4" - parameterId: "prompt-cache-memory-mib" displayName: "Prompt cache memory" - exposed: true - description: "Memory kept to reuse repeated prompts, such as agent system prompts, without processing them again. Idle conversations move here and free their accelerator memory. It uses host memory, which unified-memory machines share with the model; 0 turns the cache off." + visibility: "PARAMETER_VISIBILITY_ADVANCED" + description: "Host memory used to reuse repeated prompts, such as agent system prompts, without processing them again. Raise it if the same long prompts come back often; on unified-memory machines it competes with the model, and 0 turns it off." unsignedInteger: minimum: "0" maximum: "65536" diff --git a/catalog/engines/ninfer.yaml b/catalog/engines/ninfer.yaml index db8acab2..1a62bdb8 100644 --- a/catalog/engines/ninfer.yaml +++ b/catalog/engines/ninfer.yaml @@ -10,8 +10,8 @@ parameterSchema: # Shared parameters: the engine adapter in backend/core/src/model/engine/ maps them to flags. - parameterId: "total-context" displayName: "Total context" - exposed: true - description: "Context memory the server holds from the start and shares among the requests it generates at once. Requests that start with the same prompt share its memory instead of holding their own copy. When it fills, the server first moves cached prompts to host memory, then pauses the newest requests until memory frees up." + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "Memory the model holds for all requests at once; requests that start with the same prompt share it. Raise it to serve longer or more simultaneous conversations; when it fills, cached prompts move to host memory, then the newest requests pause." unsignedInteger: minimum: "1024" maximum: "2097152" @@ -19,16 +19,17 @@ parameterSchema: defaultValue: "32768" - parameterId: "context-per-request" displayName: "Context per request" - exposed: true - description: "Largest context one request may use from the total context. It reserves nothing: requests take memory from the total context as they grow. A new request reuses the memory of a matching prompt start, from a running request, an idle conversation or the host context." + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "Largest context one request may use. It reserves nothing: requests draw from the total context as they grow. Lower it to stop one long request from filling the total context." unsignedInteger: minimum: "1024" maximum: "262144" unit: "tokens" - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true - description: "Requests the server generates at the same time; further requests wait in a queue, and the server refuses new ones once the queue is full. Each added request also adds its own state on the accelerator." + visibility: "PARAMETER_VISIBILITY_ADVANCED" + description: "Requests generated at once; more wait in a queue, and new ones are refused when the queue is full. Each added request uses some extra accelerator memory, so raise it only when several callers use the model together." + # ninfer's `--max-concurrency` accepts 1 to 8 (docs/startup-configuration.md at 13ebb92). unsignedInteger: minimum: "1" maximum: "8" @@ -36,8 +37,8 @@ parameterSchema: defaultValue: "4" - parameterId: "host-context-mib" displayName: "Host context memory" - exposed: true - description: "Host memory where idle conversations and paused requests are kept, so they resume without processing their prompt again. 0 turns it off." + visibility: "PARAMETER_VISIBILITY_ADVANCED" + description: "Host memory that keeps idle conversations and paused requests so they resume without processing their prompt again. Raise it for many long conversations that come back; 0 turns it off." unsignedInteger: minimum: "0" maximum: "1048576" diff --git a/catalog/engines/strata.yaml b/catalog/engines/strata.yaml index 007e2364..3b32622b 100644 --- a/catalog/engines/strata.yaml +++ b/catalog/engines/strata.yaml @@ -10,8 +10,8 @@ parameterSchema: # Shared parameters: the engine adapter in backend/core/src/model/engine/ maps them to flags. - parameterId: "total-context" displayName: "Total context" - exposed: true - description: "Context memory the server holds from the start. Each parallel request keeps its own part of it, as large as the context per request, so it is always the parallel requests times the context per request." + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "Memory the model holds for all requests at once. Each parallel request gets its own share as large as the context per request, so it is always the parallel requests times the context per request; change those two instead." unsignedInteger: minimum: "1024" maximum: "2097152" @@ -19,16 +19,17 @@ parameterSchema: defaultValue: "32768" - parameterId: "context-per-request" displayName: "Context per request" - exposed: true - description: "Largest context one request may use. The server holds this much context for each parallel request from the start, whether a request uses it or not." + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "Largest context one request may use. The server sets this much aside for every parallel request from the start, used or not, so keep it as small as your longest request needs." unsignedInteger: minimum: "1024" maximum: "262144" unit: "tokens" - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true - description: "Requests the server generates at the same time; further requests wait for a free one. Each one holds its own context per request." + visibility: "PARAMETER_VISIBILITY_ADVANCED" + description: "Requests generated at once; more wait for a free one. Each holds its own context per request, so raising it multiplies the memory used; raise it only when several callers use the model together." + # Strata's wrapper accepts 1 to 8 parallel requests (`MAX_PARALLEL` in `engines/strata/start.py`). unsignedInteger: minimum: "1" maximum: "8" @@ -36,8 +37,8 @@ parameterSchema: defaultValue: "1" - parameterId: "expert-vram-mib" displayName: "Expert accelerator memory" - exposed: true - description: "Accelerator memory the model's experts may use; the experts that don't fit stay in host memory. The model's sizing sets it unless you do." + visibility: "PARAMETER_VISIBILITY_ADVANCED" + description: "Accelerator memory the model's experts may use; experts that don't fit stay in host memory and run slower. The model's sizing sets it; raise it to speed up generation if memory allows, or lower it to free accelerator memory." unsignedInteger: minimum: "1" maximum: "1048576" @@ -48,7 +49,7 @@ storageBindings: - bindingName: "data" mountPath: "/data" kind: "STORAGE_KIND_CACHE" - scope: "STORAGE_SCOPE_DEPLOYMENT" + placements: ["STORAGE_PLACEMENT_DEPLOYMENT"] artifacts: - artifactId: "linux-amd64-cuda" nix: diff --git a/catalog/engines/vllm.yaml b/catalog/engines/vllm.yaml index 7a0d09e9..02e266fe 100644 --- a/catalog/engines/vllm.yaml +++ b/catalog/engines/vllm.yaml @@ -10,8 +10,8 @@ parameterSchema: # Shared parameters: the engine adapter in backend/core/src/model/engine/ maps them to flags. - parameterId: "total-context" displayName: "Total context" - exposed: true - description: "Context memory the server holds from the start and shares among the requests it generates at once. Requests that start with the same prompt share its memory instead of holding their own copy. When it fills, the server pauses the newest requests and processes them again once memory frees up." + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "Memory the model holds for all requests at once; requests that start with the same prompt share it. Raise it to serve longer or more simultaneous conversations; when it fills, the newest requests pause and are processed again once memory frees up." unsignedInteger: minimum: "1024" maximum: "16777216" @@ -19,16 +19,16 @@ parameterSchema: defaultValue: "32768" - parameterId: "context-per-request" displayName: "Context per request" - exposed: true - description: "Largest context one request may use from the total context. It reserves nothing: requests take memory from the total context as they grow. A new request reuses the memory of a matching prompt start, from a running request, a finished one still held in the total context, or the KV offloading memory." + visibility: "PARAMETER_VISIBILITY_BASIC" + description: "Largest context one request may use. It reserves nothing: requests draw from the total context as they grow. Lower it to stop one long request from filling the total context." unsignedInteger: minimum: "1024" maximum: "1048576" unit: "tokens" - parameterId: "parallel-requests" displayName: "Parallel requests" - exposed: true - description: "Requests the server generates at the same time; further requests wait in a queue. They share the total context, and each one also holds model state of its own, so every added request adds memory." + visibility: "PARAMETER_VISIBILITY_ADVANCED" + description: "Requests generated at once; more wait in a queue. Each one also holds model state of its own, so every added request uses more memory; raise it when several callers use the model together." unsignedInteger: minimum: "1" maximum: "1024" @@ -36,32 +36,32 @@ parameterSchema: defaultValue: "4" - parameterId: "maximum-batched-tokens" displayName: "Maximum batched tokens" - exposed: false - description: "Maximum tokens processed across one scheduled batch." + visibility: "PARAMETER_VISIBILITY_HIDDEN" + description: "Most tokens processed in one step across all requests. Higher values read long prompts faster but use more memory; lower them if the model runs out of memory while reading prompts." unsignedInteger: minimum: "1" maximum: "1048576" unit: "tokens" - parameterId: "maximum-cudagraph-capture-size" displayName: "Maximum CUDA graph capture size" - exposed: false - description: "Largest sequence batch captured in a CUDA graph." + visibility: "PARAMETER_VISIBILITY_HIDDEN" + description: "Largest group of requests whose GPU work is recorded ahead of time for faster generation. Lower it to save a little memory and start-up time if you never run that many requests at once." unsignedInteger: minimum: "1" maximum: "1024" unit: "sequences" - parameterId: "kv-cache-memory-bytes" displayName: "KV cache memory" - exposed: false - description: "Accelerator memory reserved for the total context, in bytes." + visibility: "PARAMETER_VISIBILITY_HIDDEN" + description: "Accelerator memory set aside for the total context. The model's sizing sets it from the total context, so change it only to override that estimate." unsignedInteger: minimum: "1" maximum: "1099511627776" unit: "bytes" - parameterId: "kv-offloading-size-gib" displayName: "KV offloading size" - exposed: true - description: "Host memory where context evicted from the total context is kept, so a later request with the same prompt start reuses it instead of processing it again, in GiB. 0 turns it off." + visibility: "PARAMETER_VISIBILITY_ADVANCED" + description: "Host memory that keeps context evicted from the total context, so a later request with the same prompt start reuses it instead of processing it again. Raise it for long prompts that repeat; 0 turns it off." unsignedInteger: minimum: "0" maximum: "1024" @@ -75,7 +75,7 @@ storageBindings: - bindingName: "engine-cache" mountPath: "/var/cache/dotcore-engine" kind: "STORAGE_KIND_CACHE" - scope: "STORAGE_SCOPE_ENGINE_ARTIFACT" + placements: ["STORAGE_PLACEMENT_ENGINE_ARTIFACT"] environment: - { name: "VLLM_CACHE_ROOT", relativePath: "vllm" } - { name: "TRITON_CACHE_DIR", relativePath: "triton" } diff --git a/catalog/models/qwen3-8-27b.yaml b/catalog/models/qwen3-8-27b.yaml index 45bc446f..6c2922e9 100644 --- a/catalog/models/qwen3-8-27b.yaml +++ b/catalog/models/qwen3-8-27b.yaml @@ -507,6 +507,7 @@ configurations: values: - parameterId: "enabled" boolean: true + # ninfer accepts 1 to 5 MTP draft tokens (docs/startup-configuration.md at 13ebb92); 0 turns it off. - optionId: "mtp" parameters: - parameterId: "token-count" diff --git a/docs/specs/README.md b/docs/specs/README.md index 38bb3da4..e591a0ec 100644 --- a/docs/specs/README.md +++ b/docs/specs/README.md @@ -213,8 +213,8 @@ it, accepts it, and drives reality toward it. Observations flow back to the owne - Every deployment, model or app, has an immutable identity that's never reused. Its display name can change and isn't unique. Its lifecycle generations are never reused. - A paused deployment keeps its identity, configuration and storage, and releases its workload - resources and serving eligibility. Teardown removes the deployment and deletes the storage scoped - to it alone. + resources and serving eligibility. Teardown removes the deployment and deletes the storage placed + with it alone. ### Deploying a model diff --git a/docs/specs/access.md b/docs/specs/access.md index 69998fbb..a05cf596 100644 --- a/docs/specs/access.md +++ b/docs/specs/access.md @@ -114,7 +114,14 @@ flowchart TB ### Secrets -- Stored as references, never as durable plaintext values. +- Platform and provider credentials are stored as references, never as durable plaintext values. +- A deployment's secret parameters, such as an app login or an engine key, belong to that + deployment. Its controller generates or accepts them at creation, keeps them with the + deployment, and gives each workload only the value or derived form it needs. +- Clients show secret parameters masked until asked. Logs, diagnostics and telemetry never carry + them. +- Current trade-offs: a deployment's secret values sit in plain text in the site's state file and + are visible through the container runtime's inspect output. The management API has no login. - Verification and serving each receive only the credential capability they need. - Private provider credentials stay apart from inference client credentials and public connection information. diff --git a/docs/specs/apps.md b/docs/specs/apps.md index 3441174a..ccabf1b0 100644 --- a/docs/specs/apps.md +++ b/docs/specs/apps.md @@ -11,7 +11,8 @@ Code: `backend/core/src/application/` (apps controller). - Deploy a configured catalog app on a target you choose, on a managed machine, and inspect its retained configuration. - Pause, resume and tear it down. Teardown stays available while paused. -- Set the app's typed parameters, basic or advanced, when you create a deployment. +- Set the app's typed parameters, basic or advanced, when you create a deployment. Secret + parameters, such as an app login, are generated unless you supply them. - Link the app to an inference route when it declares that dependency, so it uses the site's models. - Inspect its lifecycle, readiness, access information, resource use and workload diagnostics. - Run GPU apps, such as ComfyUI, on Linux with NVIDIA GPUs, and see their GPUs and measured GPU @@ -112,7 +113,12 @@ What an application deployment keeps from creation: its configured GPU set. - The exact, immutable launch identity of each service. - The resolved typed parameters: catalog defaults, then operator overrides at creation, projected - into service configuration. + into service configuration. A binding enabled by a boolean parameter is left out when that + parameter is off ([catalog](catalog.md#app-graph)). +- Its secret parameters: generated at creation unless you supply them, and fixed from then on. + Redeploy to change one. A secret binding hands a service the value in plain form, or in a bcrypt + form computed once at creation. A secret is still generated and kept when its binding is left + out. Access rules: [access](access.md). - The inference base URL at which the hosting node reaches the site manager, composed at creation. - Its own published URL and a site-unique published host port, allocated at creation. The port belongs to the deployment, not to its snapshot or launch plan. @@ -120,10 +126,13 @@ What an application deployment keeps from creation: ### App storage -- Storage attachments are scoped to one deployment or to every deployment of the app on a node. The - apps controller owns the attachments and their scope, never their content. -- Storage shared across the app's deployments on a node outlives teardown. Machine cleanup removes it - once unused ([storage](storage.md#machine-cleanup)), and site reset removes it too. +- A storage binding declares its allowed placements in order, and the first is the default. An app + binding may allow this deployment, shared by the app (every deployment of the app on a node), or + both. You pick one placement per binding at creation. It is fixed: redeploy to change it. +- The apps controller owns the attachments and their placement, never their content. +- Teardown deletes the deployment's own data. Storage shared across the app's deployments on a node + outlives teardown. Machine cleanup removes it once unused + ([storage](storage.md#machine-cleanup)), and site reset removes it too. - App content belongs to you and the app. ## Components @@ -136,7 +145,7 @@ What an application deployment keeps from creation: policy and status. - Own app configuration resolution and endpoint projection, including the site inference base URL and optional route links. -- Own typed app parameter resolution. +- Own typed app parameter resolution, including generating secret parameters at creation. - Keep the target's GPU set in the snapshot at creation and pass that whole set to every unit at each launch. - Before compiling any launch, recheck the kept GPU set against the target's current configured @@ -173,6 +182,7 @@ What an application deployment keeps from creation: ## Future direction - Protected app access and explicit administrator custom images. +- Named storage locations declared per machine, which add choices to a binding's placements. - Multiple approved apps with versions, configuration, updates, rollback, removal and app data policy. - Staged updates and rollback under a mature version policy. diff --git a/docs/specs/catalog.md b/docs/specs/catalog.md index e9aea477..31311f08 100644 --- a/docs/specs/catalog.md +++ b/docs/specs/catalog.md @@ -97,6 +97,8 @@ option effects itself. omits and the engine release doesn't default. - Sizing equations must be linear in the total context, so the largest fitting total context follows from the estimates. +- Repository checks reject a non-empty literal bound to a credential-like variable name in the + master catalog; such values come from secret parameters. - Repository checks keep the master catalog pointing only at Dotcore's own repository. The backend accepts any locked source in site and development catalogs. @@ -129,14 +131,16 @@ A preset of defaults for one engine configuration. - Supplies defaults, never ongoing deployment authority. - Supplies no default for total context or context per request. +- May override the default and the visibility of its engine's parameters. It may hide total context + and context per request, never make them advanced. - Carries no platform. The hardware class follows from the exact placement, so one profile serves every hardware class its configuration supports. ### Engine release - Declares the shared engine parameters (total context, context per request, parallel requests), - exposed, with a default on total context and parallel requests and none on context per request. - Each release sets its own bounds, defaults and descriptions. + with fixed visibilities, a default on total context and parallel requests and none on context per + request. Each release sets its own bounds, defaults and descriptions. - May declare parameter defaults within their bounds. A default on a parameter that sizing derives is rejected. @@ -169,9 +173,16 @@ minimums. ### App graph - Carries the app's portable artifact, configuration, storage, secret, dependency, readiness and - endpoint requirements, and its typed deployment parameters, shown as basic or advanced. -- Engine releases and apps declare the storage they need. Deployments keep that declaration - ([runtime](runtime.md)). + endpoint requirements, and its typed deployment parameters, shown as basic or advanced, never + hidden. +- A configuration or secret binding may be enabled by a boolean parameter of the same graph. When + that parameter is off, the binding never reaches the service, not even as an empty value. +- Each parameter restriction (length bounds, ranges, accepted values) comes from what the app, + engine or model requires, and the catalog names its source. +- Engine releases and apps declare the storage they need. Each storage binding lists its allowed + placements in order, the first being the default. An app binding may allow this deployment or + shared by the app ([apps](apps.md)); an engine binding allows exactly one. Deployments keep that + declaration and the chosen placement ([runtime](runtime.md)). ## Components @@ -182,7 +193,9 @@ The artifact manager also owns artifact bytes ([storage](storage.md)). #### Responsibilities - Own the immutable catalog declarations and their lifecycle. -- Own the shape of the shared engine parameters. +- Own the typed parameter definition engine releases and apps share, with its basic, advanced or + hidden visibility, and its types (boolean, integer, text and secret), and the shape of the shared + engine parameters. Apps use any type; engines use boolean, integer and secret. - Validate catalogs as described above. #### Boundaries diff --git a/docs/specs/models.md b/docs/specs/models.md index 4919f96e..9941b9c9 100644 --- a/docs/specs/models.md +++ b/docs/specs/models.md @@ -58,7 +58,7 @@ flowchart TB FACTS["Target platform, runtime binding, observed facts"] --> SEL SEL --> MERGE subgraph MERGE["Each engine parameter: first value that applies"] - S1["Supplied value"] --> S2["Profile default"] --> S3["Engine release default"] --> S4["Sizing derivation"] + S1["Supplied value"] --> S2["Profile default"] --> S3["Engine release default"] --> S4["Sizing derivation"] --> S5["Generated secret"] end MERGE --> SNAP["Deployment snapshot: values, derived values, estimates"] ``` @@ -72,7 +72,11 @@ keeps the result as the deployment's snapshot. hardware classes the configuration supports. - A local request that omits its engine configuration takes the selected profile's default binding. - Engine parameter values merge per parameter: the supplied value, then the profile default, then - the engine release default, then the sizing derivation, for every supplied set. + the engine release default, then the sizing derivation, then, for a secret, a generated value, + for every supplied set. +- A secret engine parameter has no catalog default and sizing never reads it. Only creation + generates a missing one, after sizing; previews, estimates and placement need no secret value, + and a preview carries none. - An omitted context per request becomes the smaller of the maximum context and the total context, kept as a derived value. - It evaluates the declared sizing for the configuration, the selected hardware class and the @@ -191,6 +195,9 @@ never redefine it. - The portable launch plan. - The engine-neutral sizing result when sizing is declared, and which engine parameter values were derived rather than supplied. +- The engine parameter values, with which secrets were generated, and the selected profile's engine + parameter defaults and visibility overrides, so clients read every value and its origin from the + deployment alone. - The selected target's memory architecture. Engine adapters derive launch settings and non-blocking advisories from it. - The selected target's device memory total, or its host memory total on unified targets. Engine diff --git a/docs/specs/storage.md b/docs/specs/storage.md index ef706cd0..8e702381 100644 --- a/docs/specs/storage.md +++ b/docs/specs/storage.md @@ -202,6 +202,8 @@ Operator-triggered cleanup of one machine's disk. loader, and never removes an image a container uses. - It removes shared workload storage, from any site on the machine, that no container mounts and no workload has used for a retention period. +- Workload storage is attached by placement: teardown deletes deployment storage, and storage shared + by the app is kept for this cleanup ([apps](apps.md)). ### Workload resource records diff --git a/frontend/src/components/parameters/parameter-view.ts b/frontend/src/components/parameters/parameter-view.ts index 3ed8b17f..d7a88671 100644 --- a/frontend/src/components/parameters/parameter-view.ts +++ b/frontend/src/components/parameters/parameter-view.ts @@ -2,8 +2,8 @@ import { formatQuantity } from "@/lib/format"; /** One typed parameter as the shared form and details render it, independent of its owner. */ export interface ParameterView { - /** A short line under the field; absent when the label and value say enough. */ - hint?: string; + /** What the parameter does, shown in the field's help; absent when the label says enough. */ + description?: string; id: string; label: string; type: ParameterViewType; @@ -12,7 +12,9 @@ export interface ParameterView { /** The value type and constraints of a parameter view. */ export type ParameterViewType = | { kind: "boolean" } + | { kind: "secret"; minimumLength: number } | { kind: "stringEnum"; values: readonly string[] } + | { kind: "text"; maximumLength: number; minimumLength: number } | { kind: "unsignedInteger"; maximum: bigint; minimum: bigint; unit: string }; /** A typed parameter value, matched to its view by type. */ @@ -29,6 +31,17 @@ export function parameterDisplayValue( case "unsignedInteger": return typeof value === "bigint" ? formatQuantity(value, parameter.type.unit) : undefined; case "stringEnum": + case "text": + case "secret": return typeof value === "string" ? value : undefined; } } + +/** Why a typed text or secret value is refused whatever the parameter's bounds, if it is. */ +export function textValueError(label: string, input: string): string | undefined { + if (input.includes("=")) return `${label} can't contain "=".`; + // eslint-disable-next-line no-control-regex + if (/[\u0000-\u001f\u007f-\u009f]/.test(input)) + return `${label} can't contain control characters.`; + return undefined; +} diff --git a/frontend/src/components/parameters/parameters.test.tsx b/frontend/src/components/parameters/parameters.test.tsx index 7d5d1259..5700581e 100644 --- a/frontend/src/components/parameters/parameters.test.tsx +++ b/frontend/src/components/parameters/parameters.test.tsx @@ -2,7 +2,7 @@ import { render, screen } from "@testing-library/react"; import { describe, expect, it } from "vitest"; import type { ParameterView } from "./parameter-view"; -import { ParametersForm } from "./parameters"; +import { ParametersDetails, ParametersForm } from "./parameters"; describe("ParametersForm", () => { it("shows a number field's unit after its input, and nothing for a unit-less one", () => { @@ -38,9 +38,67 @@ describe("ParametersForm", () => { const cache = screen.getByRole("textbox", { name: "Prompt cache memory" }); expect(cache).toHaveValue("2048"); expect(cache.parentElement?.textContent).toBe("MiB"); - expect(cache).toHaveAccessibleDescription("MiB"); + expect(cache).not.toHaveAccessibleDescription(); const slots = screen.getByRole("textbox", { name: "Slots" }); expect(slots.parentElement?.textContent).toBe("Slots"); expect(slots).not.toHaveAccessibleDescription(); }); + + it("shows a secret's generated-at-deploy notice as a muted italic placeholder, not a value", () => { + const parameters: ParameterView[] = [ + { id: "pass", label: "Password", type: { kind: "secret", minimumLength: 1 } }, + { id: "user", label: "Username", type: { kind: "text", maximumLength: 8, minimumLength: 0 } }, + ]; + render( + {}} + onChange={() => {}} + parameters={parameters} + />, + ); + + const secret = screen.getByRole("textbox", { name: "Password" }); + expect(secret).toHaveAttribute("placeholder", "Generated at deploy"); + expect(secret).toHaveValue(""); + expect(secret).toHaveClass("placeholder:italic", "placeholder:text-muted-foreground"); + expect(screen.getByRole("textbox", { name: "Username" })).not.toHaveAttribute("placeholder"); + }); +}); + +describe("ParametersDetails", () => { + const parameters: ParameterView[] = [ + { id: "user", label: "Username", type: { kind: "text", maximumLength: 8, minimumLength: 0 } }, + { id: "note", label: "Note", type: { kind: "text", maximumLength: 8, minimumLength: 0 } }, + { id: "pass", label: "Password", type: { kind: "secret", minimumLength: 12 } }, + ]; + + it("shows values without tags, empty text as Empty and a masked secret", () => { + render( +
+ +
, + ); + + const rows = screen + .getAllByRole("term") + .map((term) => [term.textContent, term.nextElementSibling?.textContent]); + expect(rows).toEqual([ + ["Username", "admin"], + ["Note", "Empty"], + ["Password", "••••••••"], + ]); + }); }); diff --git a/frontend/src/components/parameters/parameters.tsx b/frontend/src/components/parameters/parameters.tsx index 48faaae7..e5869171 100644 --- a/frontend/src/components/parameters/parameters.tsx +++ b/frontend/src/components/parameters/parameters.tsx @@ -1,9 +1,10 @@ import type { RefObject } from "react"; import { Detail } from "@/components/ui/detail-list"; -import { FieldError } from "@/components/ui/field-error"; +import { Field } from "@/components/ui/field"; import { Input } from "@/components/ui/input"; import { NativeSelect, NativeSelectOption } from "@/components/ui/native-select"; +import { SecretValue } from "@/components/ui/secret-value"; import { Switch } from "@/components/ui/switch"; import { @@ -37,105 +38,127 @@ export function ParametersForm({ parameters, }: ParametersFormProps) { return parameters.map((parameter) => { - const error = errors.get(parameter.id); const controlId = `${idPrefix}-${parameter.id}`; - const hintId = `${controlId}-description`; - const errorId = `${controlId}-error`; - const unitId = `${controlId}-unit`; const unit = parameter.type.kind === "unsignedInteger" ? parameter.type.unit : ""; - const describedBy = - [parameter.hint ? hintId : "", unit ? unitId : "", error ? errorId : ""] - .filter(Boolean) - .join(" ") || undefined; + const field = { + controlId, + error: errors.get(parameter.id), + help: parameter.description || undefined, + label: parameter.label, + }; const register = (control: HTMLElement | null) => { inputRefs.current.set(parameter.id, control); }; - const hint = parameter.hint ? ( - - {parameter.hint} - - ) : null; - const fieldError = error ? {error} : null; if (parameter.type.kind === "boolean") { return ( -
-
- + + {(control) => ( onBlur(parameter.id)} onCheckedChange={(checked) => onChange(parameter.id, String(checked))} ref={register} /> -
- {hint} - {fieldError} -
+ )} + ); } if (parameter.type.kind === "stringEnum") { + const { values } = parameter.type; return ( -
- - onBlur(parameter.id)} - onChange={(event) => onChange(parameter.id, event.target.value)} - ref={register} - value={inputs.get(parameter.id) ?? ""} - > - {parameter.type.values.map((value) => ( - - {value} - - ))} - - {hint} - {fieldError} -
+ + {(control) => ( + onBlur(parameter.id)} + onChange={(event) => onChange(parameter.id, event.target.value)} + ref={register} + value={inputs.get(parameter.id) ?? ""} + > + {values.map((value) => ( + + {value} + + ))} + + )} + ); } - const input = ( - onBlur(parameter.id)} - onChange={(event) => onChange(parameter.id, event.target.value)} - ref={register} - value={inputs.get(parameter.id) ?? ""} - /> - ); return ( -
- - {unit ? ( -
- {input} - - {unit} - -
- ) : ( - input - )} - {hint} - {fieldError} -
+ + ) : parameter.type.kind === "text" || parameter.type.kind === "secret" ? ( + + ) : null + } + key={parameter.id} + > + {(control) => { + const input = ( + onBlur(parameter.id)} + onChange={(event) => onChange(parameter.id, event.target.value)} + ref={register} + value={inputs.get(parameter.id) ?? ""} + /> + ); + return unit ? ( +
+ {input} + {unit} +
+ ) : ( + input + ); + }} + ); }); } -/** Detail rows for retained parameter values, in view order; a missing or mistyped value is skipped. */ +/** The accepted length of a text or secret parameter. */ +function LengthFact({ type }: { type: { maximumLength?: number; minimumLength: number } }) { + return ( +

+ {type.maximumLength === undefined + ? `At least ${type.minimumLength} characters` + : `${type.minimumLength}–${type.maximumLength} characters`} +

+ ); +} + +/** The accepted range of a number parameter, with its unit when it has one. */ +function RangeFact({ type }: { type: { maximum: bigint; minimum: bigint; unit: string } }) { + return ( +

+ Range {`${type.minimum}–${type.maximum}`} + {type.unit ? ` ${type.unit}` : ""} +

+ ); +} + +/** + * Detail rows for retained parameter values, in view order; a missing or mistyped value is skipped. + * A secret is masked until revealed and empty text reads as Empty. + */ export function ParametersDetails({ parameters, values, @@ -144,9 +167,31 @@ export function ParametersDetails({ values: ReadonlyMap; }) { return parameters.map((parameter) => { - const displayValue = parameterDisplayValue(parameter, values.get(parameter.id)); - return displayValue === undefined ? null : ( - + const value = values.get(parameter.id); + const displayValue = parameterDisplayValue(parameter, value); + if (displayValue === undefined) return null; + + if (parameter.type.kind === "secret") { + return ( + } + /> + ); + } + return ( + Empty + ) : ( + displayValue + ) + } + /> ); }); } diff --git a/frontend/src/components/ui/combobox.test.tsx b/frontend/src/components/ui/combobox.test.tsx index f422c211..3748b2fe 100644 --- a/frontend/src/components/ui/combobox.test.tsx +++ b/frontend/src/components/ui/combobox.test.tsx @@ -38,7 +38,7 @@ function SingleCombobox({ loading={loading} loadingMessageVisible={loadingMessageVisible} searchable={searchable} - description="Choose one deployment." + help="Choose one deployment." /> {value ?? "none"} @@ -93,6 +93,36 @@ function MultipleCombobox({ } describe("Combobox", () => { + it("describes the input with its help and shows it in a popover beside the label", async () => { + const user = setupUser(); + render(); + + expect(screen.getByRole("combobox", { name: "Deployment" })).toHaveAccessibleDescription( + "Choose one deployment.", + ); + await user.click(screen.getByRole("button", { name: "About Deployment" })); + expect(await screen.findByRole("dialog", { name: "About Deployment" })).toHaveTextContent( + "Choose one deployment.", + ); + }); + + it("shows its status under the input and in the input's description", () => { + render( + {}} + options={options} + status="Loading deployments…" + value={null} + />, + ); + + expect(screen.getByText("Loading deployments…")).toBeVisible(); + expect(screen.getByRole("combobox", { name: "Deployment" })).toHaveAccessibleDescription( + "Loading deployments…", + ); + }); + it("filters options by label and stable ID", async () => { const user = setupUser(); render(); diff --git a/frontend/src/components/ui/combobox.tsx b/frontend/src/components/ui/combobox.tsx index 12ff9d66..0563af0a 100644 --- a/frontend/src/components/ui/combobox.tsx +++ b/frontend/src/components/ui/combobox.tsx @@ -2,6 +2,7 @@ import { Combobox as ComboboxPrimitive } from "@base-ui/react/combobox"; import { CheckIcon, ChevronDownIcon, XIcon } from "lucide-react"; import * as React from "react"; +import { FieldHelp } from "@/components/ui/field"; import { useInspectorFloatingLayerRegistration } from "@/components/ui/inspector-shell"; import { ResourceMarker } from "@/components/ui/resource-marker"; import { StatusDot } from "@/components/ui/status-badge"; @@ -31,9 +32,10 @@ type ComboboxSharedProps = { options: readonly ComboboxOption[]; ariaInvalid?: boolean; className?: string; - description?: string; disabled?: boolean; errorDescriptionId?: string; + /** What the field does, shown in a help popover beside the label. */ + help?: string; /** Keeps the label for assistive technology only, where the surrounding layout names the field. */ hideLabel?: boolean; inputRef?: React.Ref; @@ -41,6 +43,8 @@ type ComboboxSharedProps = { loadingMessageVisible?: boolean; onBlur?: React.FocusEventHandler; placeholder?: string; + /** The field's current state, such as that its options are loading, shown under the input. */ + status?: string; }; type SingleComboboxProps = ComboboxSharedProps & { @@ -77,9 +81,9 @@ function SingleCombobox({ ariaInvalid, className, clearable = true, - description, disabled, errorDescriptionId, + help, hideLabel = false, inputRef, label, @@ -90,12 +94,18 @@ function SingleCombobox({ options, placeholder, searchable = true, + status, value, }: SingleComboboxProps) { const descriptionId = React.useId(); const inputId = React.useId(); const labelId = React.useId(); - const describedBy = [description ? descriptionId : undefined, errorDescriptionId] + const statusId = React.useId(); + const describedBy = [ + help ? descriptionId : undefined, + status ? statusId : undefined, + errorDescriptionId, + ] .filter(Boolean) .join(" "); const registerFloatingLayer = useInspectorFloatingLayerRegistration(); @@ -111,7 +121,14 @@ function SingleCombobox({ value={value} >
-
@@ -152,9 +169,9 @@ function MultipleCombobox({ ariaInvalid, className, closeOnSelect = false, - description, disabled, errorDescriptionId, + help, hideLabel = false, inputRef, label, @@ -164,12 +181,18 @@ function MultipleCombobox({ onValueChange, options, placeholder = "Search options", + status, value, }: MultipleComboboxProps) { const descriptionId = React.useId(); const inputId = React.useId(); const labelId = React.useId(); - const describedBy = [description ? descriptionId : undefined, errorDescriptionId] + const statusId = React.useId(); + const describedBy = [ + help ? descriptionId : undefined, + status ? statusId : undefined, + errorDescriptionId, + ] .filter(Boolean) .join(" "); const optionLabel = createOptionLabel(options); @@ -206,7 +229,14 @@ function MultipleCombobox({ value={value} >
-
@@ -275,17 +305,21 @@ function MultipleCombobox({ * technology, label included, and only a label reference survives that hiding. */ function ComboboxLabel({ + descriptionId, + help, hidden, id, inputId, label, }: { + descriptionId: string; + help?: string; hidden: boolean; id: string; inputId: string; label: string; }) { - return ( + const labelElement = (