diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 00000000000..56cae91c577 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,48 @@ +# DAOS: guidance for Copilot + +DAOS is a distributed object store: Go control plane (`src/control`), C data plane +(`daos_engine`, `src/engine` + server modules) and C client library (`libdaos`, `src/client` + +`cli*.c` in pool/container/object). + +## Use the knowledge base first + +Before searching `src/` broadly, read the relevant files in [docs/kb](../docs/kb/README.md): + +- [docs/kb/architecture.md](../docs/kb/architecture.md): processes, channels (gRPC / dRPC / CaRT), + engine xstreams, module index. +- [docs/kb/concepts.md](../docs/kb/concepts.md): pool, container, object, epoch, DTX, target, ... mapped to code. +- Module cards in [docs/kb/modules](../docs/kb/modules): vos, dtx, object, pool, rebuild, engine, + control (Go control plane). +- Flow traces in [docs/kb/flows](../docs/kb/flows): write path, pool create, rebuild. +- Generated lookups: + - "who handles RPC X?": [docs/kb/generated/rpcs.md](../docs/kb/generated/rpcs.md) + - "who handles dRPC method X / who calls it from Go?": [docs/kb/generated/drpc.md](../docs/kb/generated/drpc.md) + - "what does module X depend on / who uses it?": [docs/kb/generated/modules.md](../docs/kb/generated/modules.md) + +KB references use `path::symbol`; jump to the file and search for the symbol (no line numbers). +If a KB statement contradicts the code, trust the code and fix the KB. + +## Code conventions + +- C style: `.clang-format`; function return type on its own line, name at column 0 (so + `^name(` finds definitions). +- Errors are negative `-DER_*` codes (`src/include/daos_errno.h`); log with `D_ERROR`/`DL_ERROR` + etc. from `src/include/gurt/debug.h`; goto-based cleanup (`D_GOTO(out, rc = ...)`). +- Memory: `D_ALLOC*`/`D_FREE` (never raw malloc/free in DAOS code). +- Engine code runs in Argobots ULTs: never block an xstream (no sleeping syscalls, no pthread + waits); yield or use ABT primitives. Never yield inside a `umem`/VOS transaction. +- RPCs are declared with X-macro lists (`*_PROTO_*_RPC_LIST`) and `CRT_RPC_DECLARE`; changing a + wire format requires a protocol version bump. +- Go code is under `src/control`; ignore `src/control/vendor`. + +## Build and test + +- Build: `scons --jobs=$(nproc) install` (see [docs/dev/development.md](../docs/dev/development.md)). +- C unit tests: `utils/run_utest.py` (config `utils/utest.yaml`); Go: `src/control/run_go_tests.sh`. +- Functional tests: `src/tests/ftest` (Avocado, `launch.py`). + +## Keeping the KB correct + +After renaming/moving code referenced by the KB, run `utils/kb/kb.py check` (regenerates nothing, +fails on stale generated files or broken references); fix with `utils/kb/kb.py extract` and by +editing the cards. diff --git a/.github/instructions/control.instructions.md b/.github/instructions/control.instructions.md new file mode 100644 index 00000000000..72e11158760 --- /dev/null +++ b/.github/instructions/control.instructions.md @@ -0,0 +1,20 @@ +--- +applyTo: "src/control/**,src/proto/**" +description: "Go control plane (daos_server, daos_agent, dmg) context and rules" +--- +# Working in the control plane + +Read [docs/kb/modules/control.md](../../docs/kb/modules/control.md) first; dRPC method mapping is +in [docs/kb/generated/drpc.md](../../docs/kb/generated/drpc.md) and an end-to-end example in +[docs/kb/flows/pool-create.md](../../docs/kb/flows/pool-create.md). + +- Never edit `src/control/vendor` or generated protobuf code (`*.pb.go`, `*.pb-c.[ch]`); + change `src/proto` and regenerate. +- MS state changes go through Raft (`system/raft.Database`); leader-only handlers call + `checkLeaderRequest`. +- A new dRPC method needs matching changes in `src/include/daos/drpc_modules.h`, + `src/control/lib/daos/drpc.go`, the Go caller and the engine handler (`src/mgmt/srv.c`), + then `utils/kb/kb.py extract`. +- Follow existing patterns: `dmg` command -> `lib/control` function -> `server/mgmt_*.go` + handler; add `_test.go` coverage with the existing mocks. +- Tests: `src/control/run_go_tests.sh`. diff --git a/.github/instructions/dtx.instructions.md b/.github/instructions/dtx.instructions.md new file mode 100644 index 00000000000..1e6ac42b5d4 --- /dev/null +++ b/.github/instructions/dtx.instructions.md @@ -0,0 +1,19 @@ +--- +applyTo: "src/dtx/**,src/include/daos_srv/dtx_srv.h,src/include/daos/dtx.h" +description: "DTX (distributed transaction) context and rules" +--- +# Working in DTX + +Read [docs/kb/modules/dtx.md](../../docs/kb/modules/dtx.md) first; the leader/non-leader sequence +is in [docs/kb/flows/write-path.md](../../docs/kb/flows/write-path.md). + +- Leader: `dtx_leader_begin()` -> `dtx_leader_exec_ops()` -> `dtx_leader_end()`. + Non-leader: `dtx_begin()` -> local work -> `dtx_end()`. +- Commit is normally asynchronous (CoS cache + `dtx_batched_commit` ULT); only `dth_sync` commits + inline. Do not assume a DTX is committed when the client gets its reply. +- `-DER_INPROGRESS` means a reader hit a prepared DTX: refresh (`dtx_refresh()`) or retry; never + treat it as a hard failure. +- Resends must be detected (`dtx_handle_resend()`) so updates are never applied twice. +- Persistent DTX state lives in VOS (`src/vos/vos_dtx.c`); keep the two in sync when changing + states (`src/include/daos/dtx.h`). +- Unit tests: `src/dtx/tests`; integration: `src/tests/suite/daos_dist_tx.c`. diff --git a/.github/instructions/engine.instructions.md b/.github/instructions/engine.instructions.md new file mode 100644 index 00000000000..7e308a57c3f --- /dev/null +++ b/.github/instructions/engine.instructions.md @@ -0,0 +1,17 @@ +--- +applyTo: "src/engine/**,src/include/daos_srv/daos_engine.h,src/include/daos_srv/iv.h" +description: "daos_engine runtime (xstreams, ULTs, scheduler, module loading, dRPC server) context and rules" +--- +# Working in the engine runtime + +Read [docs/kb/modules/engine.md](../../docs/kb/modules/engine.md) first; module/RPC tables are in +[docs/kb/generated/modules.md](../../docs/kb/generated/modules.md) and +[docs/kb/generated/drpc.md](../../docs/kb/generated/drpc.md). + +- Everything runs in Argobots ULTs: never block an xstream (no `sleep()`, blocking syscalls, + pthread waits); use `dss_sleep()`, ABT primitives, `dss_ult_execute()` or `dss_chore_register()`. +- Respect xstream roles: VOS work on the target main xstream, pool/rdb/dRPC work on xs 0. +- Startup ordering matters: modules are initialised in `dss_module_init_all()` before xstreams + exist; `sm_setup` runs later from `dss_module_setup_all()` after the rank is set up. +- Changes to `struct dss_module` or scheduler request types affect every server module. +- Tests: `src/engine/tests`. diff --git a/.github/instructions/kb.instructions.md b/.github/instructions/kb.instructions.md new file mode 100644 index 00000000000..4201e146b81 --- /dev/null +++ b/.github/instructions/kb.instructions.md @@ -0,0 +1,17 @@ +--- +applyTo: "docs/kb/**,utils/kb/**" +description: "How to maintain the DAOS knowledge base" +--- +# Maintaining the DAOS knowledge base + +- Never hand-edit `docs/kb/generated/*`; run `utils/kb/kb.py extract`. +- Every claim in a card must be checkable in code. Reference code as `::` or + `()`; no line numbers. `utils/kb/kb.py lint` verifies them. +- Cards start with frontmatter (`id`, `kind`, `depends_on`, `used_by`, `related`; lists in + `[a, b]` form) that `extract` merges into `generated/graph.json`. +- Keep cards under ~200 lines with sections: Purpose, File map, Entry points, Key structures, + Threading, Invariants and gotchas, Tests, Where to look for. Flows are ordered hop lists plus a + Mermaid sequence diagram. +- When adding a card, also add a scoped `.github/instructions/.instructions.md` that + points at it, and list it in `docs/kb/README.md`. +- Run `utils/kb/kb.py check` before committing. diff --git a/.github/instructions/object.instructions.md b/.github/instructions/object.instructions.md new file mode 100644 index 00000000000..bdc12f2d312 --- /dev/null +++ b/.github/instructions/object.instructions.md @@ -0,0 +1,19 @@ +--- +applyTo: "src/object/**,src/include/daos_srv/object.h,src/include/daos/object.h" +description: "Object layer (client and server I/O) context and rules" +--- +# Working in the object layer + +Read [docs/kb/modules/object.md](../../docs/kb/modules/object.md) first; end-to-end I/O is in +[docs/kb/flows/write-path.md](../../docs/kb/flows/write-path.md); opcode -> handler table in +[docs/kb/generated/rpcs.md](../../docs/kb/generated/rpcs.md). + +- `cli_*.c` run in libdaos (client tasks on the TSE scheduler); `srv_*.c` run in `daos_engine` + on target xstreams. Don't mix assumptions between the two. +- Server handlers must validate the request with `obj_ioc_begin()` (handles, capabilities, pool + map version) and return `-DER_STALE` for old maps; the client refreshes and retries. +- Modifications go through DTX (`dtx_leader_begin()` on the leader shard, + `ds_obj_tgt_update_handler()` on others). Don't write to VOS outside a DTX handle. +- RPC formats: `src/object/obj_rpc.h` (`DAOS_ISEQ_*`/`DAOS_OSEQ_*` + `CRT_RPC_DECLARE`). Any wire + change requires bumping `DAOS_OBJ_VERSION` and keeping the previous format working. +- Tests: `src/object/tests`, `src/tests/suite/daos_obj.c`, `src/tests/suite/daos_obj_ec.c`. diff --git a/.github/instructions/pool.instructions.md b/.github/instructions/pool.instructions.md new file mode 100644 index 00000000000..f120bcda2d7 --- /dev/null +++ b/.github/instructions/pool.instructions.md @@ -0,0 +1,18 @@ +--- +applyTo: "src/pool/**,src/include/daos_srv/pool.h,src/include/daos/pool.h,src/include/daos/pool_map.h,src/common/pool_map.c" +description: "Pool, pool service and pool map context and rules" +--- +# Working in the pool module + +Read [docs/kb/modules/pool.md](../../docs/kb/modules/pool.md) first; creation end-to-end is in +[docs/kb/flows/pool-create.md](../../docs/kb/flows/pool-create.md); opcode -> handler table in +[docs/kb/generated/rpcs.md](../../docs/kb/generated/rpcs.md). + +- Pool-service code (`src/pool/srv_pool.c`) runs on the system xstream of replica ranks, only does + work as leader, and changes metadata only inside an rdb transaction on the current term. +- Per-target state is `ds_pool_child`; target work must run on the target xstream (use collective + helpers), never inline on xs 0. +- Pool map versions are monotonic; every map change must be distributed (IV) and may need a + rebuild (`ds_rebuild_schedule()`). +- rdb layout changes need a version bump in `src/pool/srv_layout.h` and upgrade handling. +- Tests: `src/tests/suite/daos_pool.c`, `src/tests/suite/daos_rebuild.c`, `src/tests/ftest/pool`. diff --git a/.github/instructions/rebuild.instructions.md b/.github/instructions/rebuild.instructions.md new file mode 100644 index 00000000000..48c797cf38b --- /dev/null +++ b/.github/instructions/rebuild.instructions.md @@ -0,0 +1,17 @@ +--- +applyTo: "src/rebuild/**,src/include/daos_srv/rebuild.h,src/object/srv_obj_migrate.c" +description: "Rebuild and object migration context and rules" +--- +# Working in rebuild / migration + +Read [docs/kb/modules/rebuild.md](../../docs/kb/modules/rebuild.md) first; the end-to-end trace +(failure -> exclude -> rebuild -> reclaim) is [docs/kb/flows/rebuild.md](../../docs/kb/flows/rebuild.md); +placement and target states are covered in [docs/kb/modules/pool.md](../../docs/kb/modules/pool.md). + +- Rebuild is leader-driven (pool service leader, xs 0) and executed on every engine; scan and + migration run on target xstreams. Keep that split. +- All map-driven movement uses `RB_OP_REBUILD`; behaviour differs by target state in the pool + map, not by op code. Successful rebuilds are followed by `RB_OP_RECLAIM`. +- Scanning must wait for DTX resync; migration must fetch at stable epochs. +- Progress is reported through rebuild IV; never block the leader waiting on targets. +- Tests: `src/tests/suite/daos_rebuild.c` and friends, `src/tests/ftest/rebuild`. diff --git a/.github/instructions/vos.instructions.md b/.github/instructions/vos.instructions.md new file mode 100644 index 00000000000..0fa14f17d1f --- /dev/null +++ b/.github/instructions/vos.instructions.md @@ -0,0 +1,20 @@ +--- +applyTo: "src/vos/**,src/include/daos_srv/vos.h,src/include/daos_srv/vos_types.h" +description: "VOS (Versioning Object Store) context and rules" +--- +# Working in VOS + +Read [docs/kb/modules/vos.md](../../docs/kb/modules/vos.md) first; for how VOS is reached from the +network see [docs/kb/flows/write-path.md](../../docs/kb/flows/write-path.md). + +- VOS is per target and single-threaded per target xstream; it has no locking across xstreams and + no network code. Never call it from the system xstream for a target's pool. +- Never yield (ABT yield, blocking I/O, RPC wait) inside a `umem` transaction + (`vos_tx_begin()` ... `vos_tx_end()`). +- Everything is epoch-versioned (MVCC); punch is a tombstone recorded in the ilog. Do not update + in place. +- `*_begin()` reserves resources; always pair with `*_end()` passing the error so reservations are + cancelled. +- Any change to `src/vos/vos_layout.h` is an on-media format change: bump the layout version and + handle upgrade of existing pools. +- Unit tests: `src/vos/tests` (`vos_tests`); add a test with each behaviour change. diff --git a/docs/kb/README.md b/docs/kb/README.md new file mode 100644 index 00000000000..8175ecdf7de --- /dev/null +++ b/docs/kb/README.md @@ -0,0 +1,52 @@ +# DAOS Knowledge Base + +A compact, link-checked map of the DAOS code base, written for LLM agents (and humans) who need to +find their way around `src/` quickly. Prefer these files over broad searches; follow the paths and +`file::symbol` references they contain to jump straight to code. + +## How to use + +1. Start with [architecture.md](architecture.md) (processes, layers, channels, module index) and + [concepts.md](concepts.md) (vocabulary mapped to code). +2. For a module, open its card in [modules/](modules/). Cards list purpose, file map, entry points, + key structs, threading, invariants, and where to look for common tasks. +3. For cross-module behaviour, open a flow trace in [flows/](flows/). Each is an ordered call chain + with `file::function` at every hop. +4. For exhaustive facts, use the generated tables (regenerated from source, never hand-edited): + - [generated/rpcs.md](generated/rpcs.md): every CaRT RPC opcode -> handler -> file. + - [generated/drpc.md](generated/drpc.md): every dRPC method -> Go caller -> engine handler. + - [generated/modules.md](generated/modules.md): engine modules and include-derived dependencies. + - [generated/graph.json](generated/graph.json): the same facts plus card metadata as a node/edge + graph. + +## Contents + +| Kind | File | Covers | +|---|---|---| +| Overview | [architecture.md](architecture.md) | Processes, layering, communication, module index | +| Overview | [concepts.md](concepts.md) | Pool, container, object, epoch, DTX, target, xstream, ... | +| Module | [modules/vos.md](modules/vos.md) | Versioning Object Store (per-target persistent store) | +| Module | [modules/dtx.md](modules/dtx.md) | Distributed transactions | +| Module | [modules/object.md](modules/object.md) | Object client/server I/O, EC, migration | +| Module | [modules/pool.md](modules/pool.md) | Pool client, pool service, pool map | +| Module | [modules/rebuild.md](modules/rebuild.md) | Rebuild, reclaim and object migration | +| Module | [modules/engine.md](modules/engine.md) | `daos_engine` runtime: startup, xstreams, ULTs, scheduler, dRPC server, IV | +| Module | [modules/control.md](modules/control.md) | Go control plane: `daos_server`, MS, `dmg`, `daos_agent` | +| Flow | [flows/write-path.md](flows/write-path.md) | `dfs_write()` down to SCM/NVMe | +| Flow | [flows/pool-create.md](flows/pool-create.md) | `dmg pool create` through control plane, engine, rsvc, rdb | +| Flow | [flows/rebuild.md](flows/rebuild.md) | Rank failure / `dmg pool exclude` -> pool map update -> rebuild -> reclaim | +| Flow | [flows/drive-replace.md](flows/drive-replace.md) | NVMe replacement and engine recovery -> device and pool verification | + +## Maintenance + +- `utils/kb/kb.py extract` regenerates `generated/`. +- `utils/kb/kb.py check` fails if `generated/` is stale or if any path, link, or symbol referenced + by the KB or by `.github/` instructions no longer exists. Run it after renaming code. +- `utils/kb/benchmark/questions.md` holds benchmark questions with verified answers for + measuring the KB (baseline vs. KB runs); its anchors are linted too. +- Conventions for hand-written files: + - Reference code as `::` (checked: the file must contain the symbol) or + `()` (checked to exist somewhere under `src/`). Never cite line numbers. + - Keep each card under ~200 lines; link instead of copying. + - Cards start with frontmatter (`id`, `kind`, `depends_on`, `used_by`, `related`) which + `kb.py extract` merges into `generated/graph.json`. diff --git a/docs/kb/architecture.md b/docs/kb/architecture.md new file mode 100644 index 00000000000..279a399346e --- /dev/null +++ b/docs/kb/architecture.md @@ -0,0 +1,111 @@ +--- +id: architecture +kind: overview +related: [concepts] +--- +# DAOS architecture map + +Longer prose: [src/README.md](../../src/README.md), [docs/overview/architecture.md](../overview/architecture.md), +[docs/overview/storage.md](../overview/storage.md), [docs/overview/transaction.md](../overview/transaction.md). + +## Processes + +| Process | Language | Source | Role | +|---|---|---|---| +| `daos_server` | Go | `src/control/cmd/daos_server` | Control plane: config, storage provisioning, starts and monitors engines, management service (MS) with its own Raft DB, gRPC server | +| `daos_engine` | C | `src/engine` | Data plane: one per socket/rank; loads the server modules (pool, cont, obj, dtx, rdb, ...), serves CaRT RPCs | +| `daos_agent` | Go | `src/control/cmd/daos_agent` | Client-node daemon: attach info and credential signing for libdaos via dRPC | +| `dmg` | Go | `src/control/cmd/dmg` | Admin CLI, talks gRPC to `daos_server` | +| `daos` | Go + cgo | `src/control/cmd/daos` | User CLI, uses libdaos | +| libdaos / libdfs | C | `src/client`, `src/object`, `src/pool`, `src/container` (`cli*.c`) | Client library linked into applications | +| `dfuse` | C | `src/client/dfuse` | FUSE daemon exposing DFS as a POSIX mount | + +## Communication channels + +```mermaid +flowchart LR + dmg -- gRPC/TLS --> server[daos_server] + server -- dRPC unix socket --> engine[daos_engine] + engine -- dRPC notify --> server + app[app + libdaos] -- dRPC --> agent[daos_agent] + agent -- gRPC --> server + app -- CaRT RPC + bulk over fabric --> engine + engine -- CaRT RPC / collective RPC / IV / SWIM --> engine +``` + +- **gRPC**: protobuf definitions in `src/proto` (Go stubs under `src/control/common/proto`). +- **dRPC**: protobuf over Unix domain socket. Method IDs: `src/include/daos/drpc_modules.h`. + Full method -> handler table: [generated/drpc.md](generated/drpc.md). Engine side dispatch: + `src/mgmt/srv.c`; dRPC server runs as a ULT (`src/engine/drpc_listener.c`). +- **CaRT**: RPC/bulk/collective transport over Mercury (`src/cart`). Each module declares its + opcodes in an X-macro list (e.g. `src/object/obj_rpc.h::OBJ_PROTO_CLI_RPC_LIST`). Full table: + [generated/rpcs.md](generated/rpcs.md). + +## Engine internals (`src/engine`) + +- Entry: `src/engine/init.c::main` -> `src/engine/init.c::server_init` -> module load + (`src/engine/module.c::dss_module_load`) and xstream setup (`src/engine/srv.c::dss_srv_init`). +- Modules register a `struct dss_module` (RPC handlers, init/setup, per-xstream keys). List: + [generated/modules.md](generated/modules.md). +- Concurrency is Argobots: execution streams (xstreams, OS threads pinned to cores) run + user-level threads (ULTs). `src/include/daos_srv/daos_engine.h::dss_xs_type`: + - system xstreams: xs 0 (`DSS_XS_SYS`: pool service, rdb, dRPC handlers), SWIM, dRPC listener; + - one main xstream per target (`DSS_XS_VOS`): owns that target's VOS pool, handles I/O RPCs; + - optional helper/offload xstreams (`DSS_XS_IOFW`, `DSS_XS_OFFLOAD`) for I/O forwarding, EC, + checksum. +- Create ULTs with `dss_ult_create()` or the scheduler-aware `sched_create_ult()`; offload to a + helper with `dss_chore_register()`. +- CaRT RPCs are dispatched to the right xstream by `src/engine/srv.c::dss_rpc_hdlr`. + +## Server-side layering + +```mermaid +flowchart TB + subgraph services[Replicated services - system xstream] + pool_svc[pool service] --> rsvc --> rdb[rdb / Raft] + cont_svc[container service] --> rsvc + mgmt + end + subgraph io[Per-target I/O - target xstreams] + obj[object srv] --> dtx + obj --> vos + dtx --> vos + rebuild --> obj + end + vos --> bio --> spdk[(NVMe via SPDK)] + vos --> vea[vea allocator] + vos --> umem[(SCM / DRAM via umem: dav, ad_mem)] + rdb --> vos +``` + +## Module index + +Card = hand-written guide in [modules/](modules/). Dependencies are in +[generated/modules.md](generated/modules.md). + +| Module | Side | What it is | Card | +|---|---|---|---| +| `src/gurt` | common | Utilities: logging (`D_*`), hash tables, lists, errno, telemetry | | +| `src/cart` | common | Collective And RPC Transport: RPC, bulk, collective RPCs, IV, SWIM | | +| `src/common` | common | btree, `pool_map.c`, checksums, ACLs, task scheduler `tse.c`, `dav`/`dav_v2`/`ad_mem` allocators | | +| `src/client` | client | Public APIs: `api` (daos_*), `array`, `kv`, `dfs`, `dfuse`, `pydaos`, `java`, `ds3` | | +| `src/placement` | both | Object layout algorithms (jump map, ring map): `pl_obj_place()` | | +| `src/object` | both | Object I/O client (`cli_*`) and server (`srv_*`), EC, migration | [object](modules/object.md) | +| `src/pool` | both | Pool client, pool service, pool map, per-target pool state | [pool](modules/pool.md) | +| `src/container` | both | Container client, container service, per-target container child, aggregation scheduling | | +| `src/dtx` | server | Distributed transactions (2PC, commit-on-share, resync) | [dtx](modules/dtx.md) | +| `src/vos` | server | Versioning Object Store: per-target MVCC persistent store | [vos](modules/vos.md) | +| `src/bio` | server | Blob I/O: NVMe via SPDK blobstore, SMD, WAL for MD-on-SSD | | +| `src/vea` | server | Versioned extent allocator for NVMe space | | +| `src/rdb` | server | Raft-replicated KV database | | +| `src/rsvc` | server | Replicated service framework on rdb (leader election, start/stop) | | +| `src/rebuild` | server | Data rebuild after target exclude / reintegration / extend | [rebuild](modules/rebuild.md) | +| `src/mgmt` | server | Engine-side management: dRPC handlers, target create/destroy | | +| `src/engine` | server | `daos_engine` main, xstreams/ULTs, module loader, dRPC server | [engine](modules/engine.md) | +| `src/security` | both | Credential validation, ACL enforcement | | +| `src/chk` | server | Catastrophic recovery checker (`dmg check`) | | +| `src/pipeline` | both | Server-side filter/aggregate pipeline API | | +| `src/control` | control | All Go: `daos_server`, `daos_agent`, `dmg`, `daos`, control API lib | [control](modules/control.md) | +| `src/proto` | control | Protobuf definitions for gRPC and dRPC | | +| `src/utils` | tools | `ddb`, `dlck`, `daos_metrics`, `self_test`, `crt_launch` | | +| `src/tests` | tests | `ftest` (Avocado functional), `suite` (`daos_test`), perf tools | | diff --git a/docs/kb/concepts.md b/docs/kb/concepts.md new file mode 100644 index 00000000000..dfbf566539f --- /dev/null +++ b/docs/kb/concepts.md @@ -0,0 +1,62 @@ +--- +id: concepts +kind: overview +related: [architecture] +--- +# DAOS concepts mapped to code + +Definitions: [docs/overview/terminology.md](../overview/terminology.md), +[docs/overview/storage.md](../overview/storage.md). This page maps each concept to where it lives in code. + +## Storage model + +| Concept | Meaning | Code anchors | +|---|---|---| +| System | Set of `daos_server`s sharing a fabric and a management service | `src/control/system` | +| Rank | One `daos_engine` instance in the system (`d_rank_t`) | `src/include/daos/pool_map.h::pool_domain` | +| Target | Unit of storage and concurrency inside an engine: one main xstream, one VOS pool file per pool | `src/include/daos/pool_map.h::pool_target` | +| Pool | Storage reservation across a set of targets; has a pool map and a replicated pool service | `src/include/daos_srv/pool.h::ds_pool`, `src/pool/cli.c::dc_pool_connect` | +| Pool map | Versioned tree of fault domains -> ranks -> targets with per-component state (`PO_COMP_ST_UP`, `UPIN`, `DOWN`, `DOWNOUT`, `DRAIN`, `NEW`) | `src/common/pool_map.c`, `src/include/daos/pool_map.h::pool_map` | +| Pool service | Raft-replicated metadata service for a pool (leader on one replica); also hosts container service | `src/pool/srv_pool.c::ds_pool_svc_start`, [modules/pool.md](modules/pool.md) | +| Container | Object namespace inside a pool; metadata in the pool service rdb; per-target state in `ds_cont_child` | `src/include/daos_srv/container.h::ds_cont_child`, `src/container/srv_container.c` | +| Object | 128-bit OID (`daos_obj_id_t`); object class encodes redundancy (RP/EC) and sharding | `src/include/daos_obj_class.h`, `src/object/obj_class.c` | +| Shard | One placement unit of an object, `daos_unit_oid_t` = OID + shard index, lives on one target | `src/include/daos/object.h::daos_unit_oid_t` | +| Redundancy group | Set of shards that replicate or EC-protect the same dkeys | `src/object/obj_layout.c` | +| dkey / akey | Distribution key (selects redundancy group, unit of co-location) / attribute key under a dkey | `src/include/daos_obj.h` | +| Value | Single value (SV, atomic blob) or array value (EV, extents of records) | `src/include/daos_obj.h::daos_iod_type_t` | +| Layout | OID + pool map -> shard placement, computed client- and server-side | `src/placement/pl_map.c::pl_obj_place` | + +## Versioning, consistency, transactions + +| Concept | Meaning | Code anchors | +|---|---|---| +| Epoch | 64-bit HLC timestamp; every update is versioned by epoch (MVCC) | `src/include/daos_types.h::daos_epoch_t`, `src/gurt/hlc.c` | +| Snapshot | Persistent epoch that aggregation must preserve | `src/container/srv_container.c` | +| DTX | Distributed transaction used for every replicated/EC modification; leader + non-leaders | [modules/dtx.md](modules/dtx.md) | +| CoS | Commit-on-Share: committable DTXs cached and piggybacked on later RPCs instead of extra commit RPCs | `src/dtx/dtx_cos.c` | +| Incarnation log (ilog) | Per object/dkey/akey log of create/punch epochs | `src/vos/ilog.c` | +| Punch | Logical delete at an epoch | `src/vos/vos_obj.c::vos_obj_punch` | +| Aggregation | Merges/removes old versions below the oldest snapshot (VOS) and computes EC parity (EC aggregation) | `src/vos/vos_aggregate.c::vos_aggregate`, `src/object/srv_ec_aggregate.c` | + +## Runtime and services + +| Concept | Meaning | Code anchors | +|---|---|---| +| Xstream / ULT | Argobots OS-thread / user-level thread. Never block an xstream; yield instead | `src/engine/ult.c::dss_ult_create`, `src/engine/sched.c` | +| rsvc | Generic replicated-service framework (used by the pool service, which also hosts the container service) on rdb | `src/rsvc/srv.c::ds_rsvc_start` | +| rdb | Raft-based replicated KV store backed by VOS | `src/rdb/rdb.c::rdb_create`, `src/rdb/rdb_tx.c` | +| IV | Incast variables: tree-based propagation/caching of values (pool map, props, handles) across ranks | `src/cart/crt_iv.c`, `src/pool/srv_iv.c`, `src/engine/server_iv.c` | +| SWIM | Gossip-based failure detection between engines | `src/cart/swim` | +| Rebuild | Re-protect data after pool map change (exclude, drain, reintegrate, extend) via object migration | `src/rebuild/srv.c`, `src/object/srv_obj_migrate.c` | +| Handle | Pool/container handles carry capabilities; global handles shareable via `l2g`/`g2l` | `src/pool/cli.c::dc_pool_local2global` | +| Task (client) | Every client API call is a `tse_task_t` in an event-driven scheduler | `src/common/tse.c` | +| Event | Async completion object for client calls | `src/client/api/event.c` | + +## Media + +| Concept | Meaning | Code anchors | +|---|---|---| +| SCM / pmem | Byte-addressable metadata (and small values) via PMDK-like allocators behind `umem` | `src/common/mem.c`, `src/common/dav`, `src/common/dav_v2`, `src/common/ad_mem.c` | +| MD-on-SSD | Metadata kept in DRAM and persisted via WAL + checkpoint on NVMe | `src/bio/bio_wal.c`, `src/pool/srv_pool_chkpt.c` | +| NVMe | Bulk data via SPDK blobstore; space allocated by VEA | `src/bio`, `src/vea/vea_api.c::vea_reserve` | +| SMD | Server metadata: which device holds which target/pool blob | `src/bio/smd` | diff --git a/docs/kb/flows/drive-replace.md b/docs/kb/flows/drive-replace.md new file mode 100644 index 00000000000..8e56fff81f8 --- /dev/null +++ b/docs/kb/flows/drive-replace.md @@ -0,0 +1,232 @@ +--- +id: flow-drive-replace +kind: flow +related: [control, pool, rebuild] +--- +# Flow: replace a drive and verify recovery + +This flow distinguishes device-level NVMe replacement from offline engine/rank storage recovery. +`dmg storage format --replace` is not a single-device replacement command: `--replace` reuses a +rank identity, and a format request can format all NVMe devices configured for an engine that +needs its superblock. Treat the engine's local pool data as lost and recover it through rebuild. +Device state and pool-target state are different: `dmg storage query list-devices` verifies the +device, while pool queries verify target membership and rebuild progress. See +[SSD management](../../admin/administration.md#ssd-management), +[pool reintegration](../../admin/pool_operations.md#reintegration), and the +[rebuild flow](rebuild.md). + +## Online NVMe replacement + +Use this path for a regular NVMe SSD that supports online hotplug. SysXS failures terminate the +engine and use the offline path below. The SSD management commands are dispatched through +`src/control/cmd/dmg/storage_query.go::nvmeReplaceCmd.Execute` and +`src/control/server/ctl_smd_rpc.go::singleDevSmdManage`. + +```mermaid +flowchart TD + A[Query device UUID, rank, targets, state] --> B{Device already EVICTED?} + B -- No --> C[Mark faulty, unless auto-evicted] + C --> D[Old device NORMAL to EVICTED; affected targets rebuild] + B -- Yes --> D + D --> E[Hot-remove old SSD] + E --> F[Insert replacement SSD] + F --> G{VMD enabled?} + G -- No --> H[Bind new SSD to userspace driver with nvme-rebind] + G -- Yes --> I[New SSD appears as NEW] + H --> I + I --> J[Run storage replace nvme with old and new UUIDs] + J --> K[Verify new UUID NORMAL; old UUID remains EVICTED until unplugged] + K --> L[Check affected pool rebuild and target state] + L --> M{Target is still excluded and should return?} + M -- Yes --> N[Run pool reintegrate for pool, rank, and target index] + M -- No --> O[No pool reintegrate needed] + N --> P[Verify rebuild completes and target becomes UPIN] + O --> Q[Recovery complete] + P --> Q +``` + +1. Record the old device UUID, its rank, and target indices: + + ```bash + dmg -l storage query list-devices + ``` + + Before a fault, the old device is normally `NORMAL`. If it is already `EVICTED`, do not issue + `set nvme-faulty` again. `dmg storage query list-devices --show-evicted` can list evicted + devices only. +2. If the device has not already been evicted automatically, mark it faulty: + + ```bash + dmg storage set nvme-faulty --host= --uuid= + ``` + + The device changes from `NORMAL` to `EVICTED`; DAOS starts the faulty-device reaction, which + rebuilds the targets on that SSD. Check the old UUID with `dmg storage query list-devices`. +3. Hot-remove the old SSD and insert the replacement. The old device remains `EVICTED` until it + is unplugged. The replacement is expected to appear as `NEW` before it is assigned for use. + Query again to confirm both UUIDs and host/rank association. +4. If VMD is not enabled, bind the inserted SSD to a userspace driver, then replace the device: + + ```bash + dmg storage nvme-rebind -a -l + dmg storage replace nvme --host= \ + --old-uuid= --new-uuid= + ``` + + On success, the new device transitions `NEW` -> `NORMAL`; the old device remains `EVICTED` + until physically unplugged. Verify device state and health: + + ```bash + dmg -l storage query list-devices --health + ``` + + The new UUID should be `NORMAL` and show the expected rank and target IDs. The old UUID should + be absent after unplugging (or remain `EVICTED` if still present). +5. Verify storage and pool recovery separately: + + ```bash + dmg -l storage query list-pools --uuid= --verbose + dmg pool query + ``` + + `list-pools` shows the pool-to-rank/target association; `pool query` reports pool target and + rebuild status. A `NORMAL` device alone does not prove that a pool target has returned to the + healthy `UPIN` state. If a target remains excluded and should return to this pool, reintegrate + only the affected target(s): + + ```bash + dmg pool reintegrate --ranks= --target-idx= + ``` + + Verify rebuild completion and that the target is `UPIN` in the pool map. Do not use + `dmg system reintegrate` for a device-only hot replacement unless the whole rank needs to be + returned to all of its pools. + +## Single-drive swap while the node is powered off + +A powered-off swap is not the same event sequence as a live hotplug. A post-boot +`dmg storage replace nvme` is only viable if the engine starts and can see both the old device's +fault record and the new device. `bio_replace_dev()` requires the old SMD device to be `FAULTY`, +its blobstore to be out of service, and the new UUID to be discoverable but not already in SMD. +See `src/bio/bio_device.c::bio_replace_dev` and +`src/bio/bio_device.c::replace_dev`. + +```mermaid +flowchart TD + A[Power off node and replace one regular SSD] --> B[Power on and start daos_server] + B --> C{Engine starts and joins?} + C -- No --> X[Stop; use the supported offline engine recovery procedure] + C -- Yes --> D[Query old and new device UUIDs] + D --> E{Old UUID is EVICTED/FAULTY; new UUID is NEW and discoverable?} + E -- No --> Y[Stop; do not format or replace speculatively] + E -- Yes --> F{VMD enabled?} + F -- No --> G[Apply the documented non-VMD rebind step] + F -- Yes --> H[Run storage replace nvme] + G --> H + H --> I[Verify new UUID NORMAL and pool targets recover] +``` + +1. After starting `daos_server`, verify rank membership and inspect device states: + + ```bash + dmg system query -v + dmg -l storage query list-devices + ``` + + Continue only if the engine is available and the state is suitable for device replacement: the + old UUID must remain in SMD as faulty/evicted and the new UUID must be visible as an unassigned + device (`NEW`). The device-replace handler also requires the old blobstore to be out of service. +2. If VMD is disabled, use the documented rebind step to expose the new SSD to DAOS: + + ```bash + dmg storage nvme-rebind -a -l + ``` + + Local documentation notes that full non-VMD hotplug capability is not production-supported in + the described release; confirm support for the deployed DAOS version and platform before using + this path. See `src/bio/README.md`. +3. If the engine can discover both devices in the required states, issue the device-level replace: + + ```bash + dmg storage replace nvme --host= \ + --old-uuid= --new-uuid= + ``` + + Verify the new device is `NORMAL` and check each affected pool with `dmg pool query` until + rebuild/reintegration completes and the targets are healthy (`UPIN`). Device replacement + triggers automatic reintegration in the normal hotplug path, but the code notes a restart can + miss that automatic trigger. If targets remain excluded, handle their pool reintegration + explicitly and verify completion. + +Do not run `dmg storage format --replace` as a substitute for replacing a single SSD. That command +is for engine/rank storage recovery, and `--replace` selects rank identity rather than a device. +If the engine cannot start or the required SMD state is unavailable, stop and follow the +failure-specific offline recovery procedure; do not infer that formatting is safe for the other +drives or their pool data. + +## Offline engine or rank recovery + +Use the device-level `dmg storage replace nvme` path when the engine is available and the device +replacement procedure is supported. It replaces the SMD device mapping and is the specific +single-device operation. The documented procedure assumes online hotplug; do not assume it applies +to a powered-off swap unless the platform's recovery procedure supports it. + +The following offline procedure is for engine/rank storage recovery when formatting is required, +such as a SysXS failure or loss of engine metadata. It is not a way to ask DAOS to replace only one +NVMe device. The `--replace` option preserves the old rank identity; it does not limit formatting to +the failed device. When the engine needs a new superblock, `formatEngineBdevs()` formats the NVMe +tiers configured for that engine. Local data on those devices, including healthy devices in the same +engine, must be treated as lost and reconstructed by rebuild. Preserve metadata for other engines +on a multi-engine host. See `src/control/server/instance_storage_rpc.go::formatEngineBdevs`. + +```mermaid +flowchart TD + A[Verify rank and failed device] --> B[Stop daos_server and replace hardware] + B --> C[Perform failure-specific storage repair] + C --> D[Start daos_server and format engine storage with --replace] + D --> E{Engine rejoins with original rank?} + E -- No --> X[Stop and resolve format or rank issue] + E -- Yes --> F[Verify system rank Joined] + F --> G[Verify replacement device NORMAL and healthy] + G --> H[Run system reintegrate for rank] + H --> I[Verify pool rebuild completes and targets become UPIN] +``` + +1. Verify the affected rank and device before stopping the server: + + ```bash + dmg system query -v + dmg -l storage query list-devices + ``` + + The engine/rank may be `Errored` or `Excluded`; the device query identifies the failed device, + its role, and whether it is `EVICTED` or `UNPLUGGED`. +2. Stop `daos_server`, replace the device, and perform the failure-specific repair. For documented + SysXS recovery, remove only the failed engine's superblock as described in + [SysXS device failure recovery](../../admin/administration.md#sysxs-device-failure-recovery). +3. Restart `daos_server`, then format the recovered engine using its prior rank. This can reset + every NVMe device configured for that engine, not just the failed SSD; only proceed when this + broader data loss and rebuild are intended: + + ```bash + dmg storage format --replace -l + ``` + + `--replace` preserves the old rank identity; it is not a device selector. Before format + completes, the new device may be `NEW`. After format, verify the expected devices are `NORMAL` + and healthy, and verify that the engine joined with the original rank: + + ```bash + dmg system query -v + dmg -l storage query list-devices --health + ``` + +4. Reintegrate the returned rank into all pools and verify the resulting rebuild: + + ```bash + dmg system reintegrate --ranks= + dmg pool query + ``` + + The pool targets transition through `UP` while reintegration/rebuild is pending and become + `UPIN` when healthy. Confirm rebuild completion for every affected pool. \ No newline at end of file diff --git a/docs/kb/flows/pool-create.md b/docs/kb/flows/pool-create.md new file mode 100644 index 00000000000..6e0e8be06b3 --- /dev/null +++ b/docs/kb/flows/pool-create.md @@ -0,0 +1,85 @@ +--- +id: flow-pool-create +kind: flow +related: [pool, mgmt, rsvc, rdb, vos, control] +--- +# Flow: `dmg pool create` + +Spans three processes and both RPC worlds: gRPC (dmg -> daos_server), dRPC (daos_server -> +daos_engine) and CaRT (engine <-> engine). Modules: [pool](../modules/pool.md), +[vos](../modules/vos.md); dRPC table: [generated/drpc.md](../generated/drpc.md). + +```mermaid +sequenceDiagram + participant D as dmg (Go) + participant S as daos_server MS leader (Go) + participant E as daos_engine (local to MS leader) + participant T as all pool ranks (engines) + participant R as pool-service replicas (engines) + D->>S: gRPC MgmtSvc.PoolCreate + S->>S: pick ranks, fault domains, sizes; sysdb.AddPoolService (state Creating, Raft) + S->>E: dRPC DRPC_METHOD_MGMT_POOL_CREATE (207) + E->>T: CaRT MGMT_TGT_CREATE (collective) + T->>T: vos_pool_create per target + E->>R: CaRT RSVC_START (collective) -> rdb_create + E->>R: CaRT POOL_CREATE to new leader + R->>R: ds_pool_create_handler: init_pool_metadata in rdb tx + E-->>S: PoolCreateResp (svc replicas, target ranks) + S->>S: ps.State = Ready; sysdb.UpdatePoolService + S-->>D: PoolCreateResp +``` + +## dmg (admin node) + +1. `src/control/cmd/dmg/pool.go::poolCreateCmd.Execute`: parses flags (size/ratio/ranks/props), + builds `control.PoolCreateReq`. +2. `src/control/lib/control/pool.go::PoolCreate` -> `src/control/lib/control/pool.go::poolCreateGenPBReq` + converts to `mgmtpb.PoolCreateReq` (`src/proto/mgmt/pool.proto::PoolCreateReq`) and invokes gRPC + `MgmtSvc.PoolCreate` (`src/proto/mgmt/mgmt.proto`) through `InvokeUnaryRPC`, which routes MS + requests to the management-service leader. + +## daos_server, management service leader + +3. `src/control/server/mgmt_pool.go::mgmtSvc.PoolCreate`: leader check, serialised request. +4. `src/control/server/mgmt_pool.go::mgmtSvc.poolCreate`: + - `poolCreateAddSystemProps`, takes a pool lock (`sysdb.TakePoolLock`); if the pool already + exists and is not Ready, returns `TryAgain`; + - chooses ranks (all joined ranks, a random subset for `--nranks`, or the explicit list; + non-joined ranks become `UnavailableRanks` / DOWNOUT), compresses the fault-domain tree, + computes per-tier sizes (`src/control/server/mgmt_pool.go::calculateCreateStorage`); + - records the pool in the MS Raft DB with state Creating: + `src/control/system/raft/database.go::Database.AddPoolService` -> + `src/control/system/raft/raft.go::Database.submitPoolUpdate`; + - `svc.harness.CallDrpc(ctx, daos.MethodPoolCreate, req)` (`src/control/server/harness.go::CallDrpc`) + to a local engine; on failure destroys the pool and removes the MS entry; + - on success sets `ps.Replicas`, `PoolServiceStateReady`, `sysdb.UpdatePoolService`. + +## daos_engine that received the dRPC + +5. `src/mgmt/srv.c` dispatches `DRPC_METHOD_MGMT_POOL_CREATE` to + `src/mgmt/srv_drpc.c::ds_mgmt_drpc_pool_create` (unpacks `Mgmt__PoolCreateReq`, props, ACL). +6. `src/mgmt/srv_pool.c::ds_mgmt_create_pool`: + 1. `src/mgmt/srv_pool.c::ds_mgmt_tgt_pool_create_ranks` sends collective `MGMT_TGT_CREATE` + to all active ranks. Each rank runs `src/mgmt/srv_target.c::ds_mgmt_hdlr_tgt_create` + (preallocates files, `tgt_create_preallocate`) and, per target, + `src/mgmt/srv_target.c::tgt_vos_create_one` -> `src/vos/vos_pool.c::vos_pool_create`. + 2. `src/mgmt/srv_pool.c::ds_mgmt_pool_svc_create` -> + `src/pool/srv_pool.c::ds_pool_svc_dist_create`: builds the initial pool map buffer + (`pool_buf`, version 1), chooses service replicas from the redundancy factor, then: + - `src/rsvc/srv.c::ds_rsvc_dist_start` broadcasts `RSVC_START`; each replica runs + `src/rsvc/srv.c::ds_rsvc_start_handler` -> `src/rsvc/srv.c::ds_rsvc_start` -> `start()` -> + `src/rdb/rdb.c::rdb_create` (new Raft DB, backed by VOS) and starts Raft; + - sends `POOL_CREATE` to the elected leader via `rsvc_client`. + 3. On the pool-service leader `src/pool/srv_pool.c::ds_pool_create_handler` opens an rdb + transaction; `src/pool/srv_pool.c::init_pool_metadata` writes the pool map and properties, + `ds_cont_init_metadata()` initialises the container-service KVSs; `rdb_tx_commit()`. + 4. Failures roll back by destroying the targets created in step 1. +7. Response (service replicas, target ranks, tier sizes) returns via dRPC to step 4. + +## Related flows + +- Pool destroy mirrors this: `DRPC_METHOD_MGMT_POOL_DESTROY` -> `ds_mgmt_drpc_pool_destroy()`. +- After create, the pool service leader steps up (`src/pool/srv_pool.c::pool_svc_step_up_cb`) and + distributes the map to targets via IV (`ds_pool_iv_map_update()`). +- Pool connect from a client: `dc_pool_connect()` -> `POOL_CONNECT` -> + `src/pool/srv_pool.c::ds_pool_connect_handler`. diff --git a/docs/kb/flows/rebuild.md b/docs/kb/flows/rebuild.md new file mode 100644 index 00000000000..3a4a5105324 --- /dev/null +++ b/docs/kb/flows/rebuild.md @@ -0,0 +1,120 @@ +--- +id: flow-rebuild +kind: flow +related: [rebuild, pool, object, placement, dtx, control, engine] +--- +# Flow: target failure -> exclude -> rebuild -> reclaim + +Scenario: an engine (rank) dies, or an admin runs `dmg pool exclude`. The pool service excludes +the rank's targets, and the remaining engines restore redundancy. Module details: +[rebuild](../modules/rebuild.md), [pool](../modules/pool.md), [object](../modules/object.md), +[control](../modules/control.md). + +```mermaid +sequenceDiagram + participant E as Surviving engines (CaRT/SWIM) + participant MS as daos_server MS leader + participant PS as Pool service leader (xs 0) + participant T as Pool engines (xs 0 + targets) + participant D as Destination targets + alt automatic + E->>PS: CRT_EVT_DEAD (SWIM or group update) -> ds_pool_crt_event_cb + MS-->>E: (engine died -> member state, GROUP_UPDATE dRPC) + else manual + MS->>PS: dmg pool exclude -> dRPC POOL_EXCLUDE -> POOL_EXCLUDE RPC + end + PS->>PS: pool_svc_update_map: targets -> DOWN, rdb commit, IV map distribution + PS->>PS: ds_rebuild_schedule (RB_OP_REBUILD, ~5s merge delay) + PS->>T: REBUILD_OBJECTS_SCAN + T->>D: DAOS_OBJ_RPC_MIGRATE (objects that lost a shard) + D->>D: fetch from surviving replicas / EC reconstruct -> VOS + T-->>PS: rebuild IV progress (scan_done, pull_done) + PS->>PS: ds_pool_tgt_finish_rebuild: DOWN -> DOWNOUT + PS->>T: RB_OP_RECLAIM (scan again, delete stale data) +``` + +## 1. Failure is noticed + +Automatic path (no admin action): + +1. SWIM in every engine (`src/cart/swim`) marks the dead rank; CaRT raises a `CRT_EVT_DEAD` event. + Separately, `daos_server` sees the engine process exit + (`src/control/server/instance_exec.go::EngineInstance.handleExit`, RAS `RASEngineDied`), the MS + updates the member state (`src/control/system/membership.go::Membership.OnEvent`), and MS group + updates (`src/control/server/mgmt_system.go::mgmtSvc.doGroupUpdate`, dRPC + `DRPC_METHOD_MGMT_GROUP_UPDATE`) remove the rank from the engines' primary group, which also + surfaces as a dead event (`CRT_EVS_GRPMOD`). +2. On the pool service leader, `src/pool/srv_pool.c::ds_pool_crt_event_cb` -> `queue_event()`; + the `src/pool/srv_pool.c::events_handler` ULT -> `src/pool/srv_pool.c::handle_event` -> + `src/pool/srv_pool.c::pool_svc_exclude_ranks` builds the rank list and calls + `pool_svc_update_map()` with `MAP_EXCLUDE` and source `MUS_SWIM`. + +Manual path: + +1. `dmg pool exclude` -> `src/control/server/mgmt_pool.go::mgmtSvc.PoolExclude` -> dRPC + `DRPC_METHOD_MGMT_POOL_EXCLUDE` -> `src/mgmt/srv_drpc.c::ds_mgmt_drpc_pool_exclude` -> + `src/mgmt/srv_pool.c::ds_mgmt_pool_target_update_state` -> + `src/pool/srv_cli.c::dsc_pool_svc_update_target_state` (POOL_EXCLUDE RPC to the leader). +2. `src/pool/srv_pool.c::ds_pool_update_handler` -> `pool_update_handler()` -> + `pool_svc_update_map()`. Drain (`PO_COMP_ST_DRAIN`) and reintegrate (`PO_COMP_ST_UP`) take + the same route with a different target state. + +## 2. Pool map update (pool service leader, xs 0) + +3. `src/pool/srv_pool.c::pool_svc_update_map` -> `src/pool/srv_pool.c::pool_svc_update_map_internal`: + - for SWIM-sourced excludes, checks the pool property `DAOS_SELF_HEAL_AUTO_EXCLUDE` and the + system self-heal property (`DS_MGMT_SELF_HEAL_POOL_EXCLUDE`); if disabled nothing changes; + - `src/pool/srv_pool_map.c::ds_pool_map_tgts_update` moves targets to `DOWN` and bumps the map + version; written in an rdb transaction (`rdb_tx_commit()`); + - `ds_rsvc_request_map_dist()` -> `src/pool/srv_pool.c::pool_svc_map_dist_cb` -> + `ds_pool_iv_map_update()` distributes the new map to all engines. Clients learn it lazily + (I/O with an old map version gets `-DER_STALE`, see [write-path.md](write-path.md)). +4. Still in `pool_svc_update_map`: rebuild is skipped if `DAOS_REBUILD=no`, if the system + self-heal disables pool rebuild for automatic events, or if + `src/include/daos_srv/pool.h::is_pool_rebuild_allowed` says no (pool `self_heal` lacks + `AUTO_REBUILD`/`DELAY_REBUILD` for automatic events, or rebuild was stopped by an admin). + Otherwise `src/rebuild/srv.c::ds_rebuild_schedule` queues an `RB_OP_REBUILD` task, with a + 5 s delay for exclude/drain/reint so multi-rank failures merge into one rebuild + (`DAOS_SELF_HEAL_DELAY_REBUILD` defers it until an admin starts it). + +## 3. Rebuild (see [modules/rebuild.md](../modules/rebuild.md) for every hop) + +5. Leader: `src/rebuild/srv.c::rebuild_ults` -> `src/rebuild/srv.c::rebuild_task_ult` -> + `src/rebuild/srv.c::rebuild_leader_start` -> `src/rebuild/srv.c::rebuild_scan_broadcast` + (collective `REBUILD_OBJECTS_SCAN`). +6. Every engine: `src/rebuild/scan.c::rebuild_tgt_scan_handler` -> `rebuild_scan_leader()` waits + for DTX resync, then `src/rebuild/scan.c::rebuild_scanner` on every target walks local objects + and asks placement (`src/placement/pl_map.c::pl_obj_find_rebuild`) which shards were on the + `DOWN` targets and where they now go. +7. Scanner sends object lists: `src/object/srv_obj_migrate.c::ds_object_migrate_send` + (`DAOS_OBJ_RPC_MIGRATE`). +8. Destination: `src/object/srv_obj_migrate.c::ds_obj_migrate_handler` -> `ds_migrate_object()` -> + `migrate_one_ult()` -> `src/object/srv_obj_migrate.c::migrate_dkey`: reads the data from + surviving shards with `dsc_obj_fetch()` (EC: reconstructs data/parity) and writes it with + `vos_obj_update()`. +9. Progress flows back via rebuild IV (`src/rebuild/rebuild_iv.c::rebuild_iv_update` -> + `src/rebuild/srv.c::rebuild_global_status_update` on the leader); `dmg pool query` shows it + through `src/rebuild/srv.c::ds_rebuild_query`. + +## 4. Completion + +10. `rebuild_task_ult` -> `src/pool/srv_pool.c::ds_pool_tgt_finish_rebuild`: `DOWN` -> `DOWNOUT` + (map version bump, distributed again). +11. `src/rebuild/srv.c::rebuild_task_complete_schedule` queues `RB_OP_RECLAIM`, which re-scans and + removes data that no longer belongs on each target. On failure it calls + `src/rebuild/srv.c::retry_rebuild_task` and may queue `RB_OP_FAIL_RECLAIM` before retrying. + +## Variations + +- **Drain**: targets go `DRAIN` (still in the map) while their shards are copied to other + targets; finish moves them to `DOWNOUT`. +- **Reintegrate**: `ds_mgmt_pool_target_update_state()` first recreates the target's VOS files + (`ds_mgmt_tgt_pool_create_ranks()`), targets go `UP`; rebuild copies data back; finish moves them + to `UPIN`; reclaim deletes stale copies elsewhere. Reintegration mode + `DAOS_REINT_MODE_NO_DATA_SYNC` skips data movement. +- **Extend**: `dmg pool extend` -> `src/pool/srv_pool.c::ds_pool_extend_handler` adds targets, + which move `NEW` -> `UP` (`MAP_EXTEND` in `src/pool/srv_pool_map.c`); rebuild moves the shards + whose placement changed; finish moves them to `UPIN`. +- State transitions for every map op are in `src/pool/srv_pool_map.c::ds_pool_map_tgts_update`. +- **Admin control**: `dmg pool rebuild stop|start` -> `ds_rebuild_admin_stop()` / + `ds_rebuild_admin_start()`. diff --git a/docs/kb/flows/write-path.md b/docs/kb/flows/write-path.md new file mode 100644 index 00000000000..505e047c2a2 --- /dev/null +++ b/docs/kb/flows/write-path.md @@ -0,0 +1,107 @@ +--- +id: flow-write-path +kind: flow +related: [object, dtx, vos, bio, vea, placement] +--- +# Flow: DFS write down to media + +Scenario: an application calls `dfs_write()` on a file whose object class is replicated (RP_*) or +erasure coded (EC_*). Modules: [object](../modules/object.md), [dtx](../modules/dtx.md), +[vos](../modules/vos.md). + +```mermaid +sequenceDiagram + participant App as App (libdfs/libdaos) + participant L as Leader target (engine) + participant N as Non-leader target(s) + participant M as VOS / bio / media + App->>App: dfs_write -> daos_array_write -> dc_array_io -> dc_obj_update_task + App->>L: DAOS_OBJ_RPC_UPDATE (one per redundancy group, bulk handle) + L->>L: ds_obj_rw_handler: obj_ioc_begin (map version, capas), dtx_leader_begin + par forward + L->>N: DAOS_OBJ_RPC_TGT_UPDATE (ds_obj_remote_update) + N->>N: ds_obj_tgt_update_handler: dtx_begin + N->>M: obj_local_rw: vos_update_begin, bulk pull, bio_iod_post, vos_update_end + N-->>L: reply + and local + L->>M: obj_local_rw (same as non-leader) + end + L->>L: dtx_leader_end: committable, CoS add (async) or dtx_commit (sync) + L-->>App: reply (epoch) +``` + +## Client side (`libdaos`, application threads + TSE scheduler) + +1. `src/client/dfs/io.c::dfs_write`: builds a `daos_array_iod_t` (offset/length) and calls + `daos_array_write()`. +2. `src/client/api/array.c::daos_array_write`: creates a task running `dc_array_write()`. +3. `src/client/array/dc_array.c::dc_array_write` -> `src/client/array/dc_array.c::dc_array_io`: + maps byte ranges to array chunks (one dkey per chunk) and creates `DAOS_OPC_OBJ_UPDATE` + sub-tasks with `daos_task_create()`. +4. Opcode table `src/client/api/init.c` maps `DAOS_OPC_OBJ_UPDATE` to + `src/object/cli_obj.c::dc_obj_update_task` (also reachable directly from + `src/client/api/object.c::daos_obj_update`). +5. `src/object/cli_obj.c::dc_obj_update`: checks map version, selects the redundancy group for + the dkey hash (`src/object/cli_obj.c::obj_update_shards_get`), EC reassembly if needed + (`src/object/cli_ec.c`), creates bulk handles for large payloads + (`src/object/cli_obj.c::obj_rw_bulk_prep`). +6. `src/object/cli_obj.c::obj_req_fanout`: with server-side dispatch (`ort_srv_disp`, the default + when group size > 1) sends **one RPC per redundancy group to the DTX leader shard**. +7. `src/object/cli_shard.c::dc_obj_shard_rw`: fills `obj_rw_in` (oid, dkey, iods, epoch, + `orw_map_ver`, DTX id, bulks) and sends `DAOS_OBJ_RPC_UPDATE`. + +## Leader target (`daos_engine`, target main xstream) + +8. `src/engine/srv.c::dss_rpc_hdlr` queues the RPC to the target xstream's scheduler. +9. `src/object/srv_obj.c::ds_obj_rw_handler`: `src/object/srv_obj.c::obj_ioc_begin` validates + pool/container handles, capabilities and pool map version (`-DER_STALE` on mismatch); resend + check (`dtx_handle_resend()`); then `src/dtx/dtx_common.c::dtx_leader_begin`. +10. `src/dtx/dtx_common.c::dtx_leader_exec_ops` runs `src/object/srv_obj.c::obj_tgt_update` for + every participant: + - remote shards: `src/object/srv_obj_remote.c::ds_obj_remote_update` sends + `DAOS_OBJ_RPC_TGT_UPDATE` (dispatched as a `dss_chore`); + - local shard (`idx == -1`): `src/object/srv_obj.c::obj_local_rw`. +11. `src/dtx/dtx_common.c::dtx_leader_end`: on success marks the DTX committable and either commits + synchronously (`dtx_commit()`) or adds it to the CoS cache (`dtx_cos_add()`) for batched + commit by `dtx_batched_commit`; on failure `dtx_abort()`. Replies to the client. + +## Non-leader targets + +12. `src/object/srv_obj.c::ds_obj_tgt_update_handler`: `src/dtx/dtx_common.c::dtx_begin`, then + `obj_local_rw()`, then `src/dtx/dtx_common.c::dtx_end` (entry stays *prepared* until the + leader commits). + +## Local I/O (any target, target main xstream) + +13. `src/object/srv_obj.c::obj_local_rw` -> `src/object/srv_obj.c::obj_local_rw_internal_wrap` -> + `src/object/srv_obj.c::obj_local_rw_internal`: + - checksum verification (`src/object/srv_csum.c`) and EC filtering; + - `src/vos/vos_io.c::vos_update_begin`: resolves the object/dkey/akey trees, checks DTX/MVCC + conflicts, **reserves** space: SCM via `src/vos/vos_io.c::vos_reserve_scm`, NVMe via + `src/vos/vos_io.c::vos_reserve_blocks` -> `src/vea/vea_api.c::vea_reserve`; + - `src/bio/bio_buffer.c::bio_iod_prep`: maps the reserved extents to DMA buffers; + - `src/object/srv_obj.c::obj_bulk_transfer`: RDMA-pulls client data into those buffers + (inline data is copied with `bio_iod_copy()`); + - `src/bio/bio_buffer.c::bio_iod_post` / `bio_iod_post_async`: writes NVMe data via SPDK; + - `src/vos/vos_io.c::vos_update_end`: within one VOS/umem transaction + (`src/vos/vos_common.c::vos_tx_begin` ... `src/vos/vos_common.c::vos_tx_end`) publishes the + reservations, inserts records into SV/EV trees, updates ilogs and registers each record with + the DTX entry (`src/vos/vos_dtx.c::vos_dtx_register_record`). The DTX entry itself is + attached in DRAM by `dtx_begin()`/`dtx_leader_begin()` and persisted by + `dtx_end()`/`dtx_leader_end()` (both via `src/vos/vos_dtx.c::vos_dtx_attach`). + +## Branches worth knowing + +- **Solo DTX** (single-replica object class): no forwarding; DTX is validated inside VOS. +- **EC**: client splits stripes into data + parity per shard; partial-stripe updates are + replicated to parity shards and later folded by EC aggregation (`src/object/srv_ec_aggregate.c`). +- **Small values** stay in SCM; larger array extents go to NVMe (media selection in `vos_io.c`). +- **Errors**: `-DER_STALE` (refresh pool map, retry), `-DER_INPROGRESS` (conflicting prepared + DTX; server tries `dtx_refresh()` first), `-DER_TX_RESTART` (MVCC conflict, restart with new + epoch). Client retry logic: `src/object/cli_obj.c::obj_retry_cb`. + +## Related reads (fetch path mirror) + +`dc_obj_fetch_task()` -> `ds_obj_rw_handler()` (fetch path: local `dtx_begin()` only, no +forwarding) -> `obj_local_rw()` -> `vos_fetch_begin()` -> `bio_iod_prep()` -> bulk PUT -> +`vos_fetch_end()`. diff --git a/docs/kb/generated/drpc.md b/docs/kb/generated/drpc.md new file mode 100644 index 00000000000..68ba53aa896 --- /dev/null +++ b/docs/kb/generated/drpc.md @@ -0,0 +1,66 @@ + + +# dRPC methods (daos_server <-> daos_engine) + +IDs from `src/include/daos/drpc_modules.h`; Go constants from `src/control/lib/daos/drpc.go`. 2xx are handled by the engine (`src/mgmt/srv.c`); 3xx are engine -> daos_server notifications handled in Go. + +| ID | Method | Go const | Go callers | Engine handler | Handler file | +|---|---|---|---|---|---| +| 101 | `DRPC_METHOD_SEC_AGENT_REQUEST_CREDS` | `MethodRequestCredentials` | `src/control/cmd/daos_agent/security_rpc.go` | | | +| 201 | `DRPC_METHOD_MGMT_KILL_RANK` | | | | | +| 202 | `DRPC_METHOD_MGMT_SET_RANK` | `MethodSetRank` | `src/control/server/instance.go` | `ds_mgmt_drpc_set_rank` | `src/mgmt/srv_drpc.c` | +| 206 | `DRPC_METHOD_MGMT_GET_ATTACH_INFO` | `MethodGetAttachInfo` | `src/control/cmd/daos_agent/mgmt_rpc.go` | | | +| 207 | `DRPC_METHOD_MGMT_POOL_CREATE` | `MethodPoolCreate` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_create` | `src/mgmt/srv_drpc.c` | +| 208 | `DRPC_METHOD_MGMT_POOL_DESTROY` | `MethodPoolDestroy` | `src/control/server/mgmt_fi.go`, `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_destroy` | `src/mgmt/srv_drpc.c` | +| 209 | `DRPC_METHOD_MGMT_SET_UP` | `MethodSetUp` | `src/control/server/instance.go` | `ds_mgmt_drpc_set_up` | `src/mgmt/srv_drpc.c` | +| 210 | `DRPC_METHOD_MGMT_BIO_HEALTH_QUERY` | `MethodBioHealth` | `src/control/server/instance_drpc.go` | `ds_mgmt_drpc_bio_health_query` | `src/mgmt/srv_drpc.c` | +| 211 | `DRPC_METHOD_MGMT_SMD_LIST_DEVS` | `MethodSmdDevs` | `src/control/server/instance_drpc.go` | `ds_mgmt_drpc_smd_list_devs` | `src/mgmt/srv_drpc.c` | +| 212 | `DRPC_METHOD_MGMT_SMD_LIST_POOLS` | `MethodSmdPools` | `src/control/server/ctl_smd_rpc.go` | `ds_mgmt_drpc_smd_list_pools` | `src/mgmt/srv_drpc.c` | +| 213 | `DRPC_METHOD_MGMT_POOL_GET_ACL` | `MethodPoolGetACL` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_get_acl` | `src/mgmt/srv_drpc.c` | +| 215 | `DRPC_METHOD_MGMT_POOL_OVERWRITE_ACL` | `MethodPoolOverwriteACL` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_overwrite_acl` | `src/mgmt/srv_drpc.c` | +| 216 | `DRPC_METHOD_MGMT_POOL_UPDATE_ACL` | `MethodPoolUpdateACL` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_update_acl` | `src/mgmt/srv_drpc.c` | +| 217 | `DRPC_METHOD_MGMT_POOL_DELETE_ACL` | `MethodPoolDeleteACL` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_delete_acl` | `src/mgmt/srv_drpc.c` | +| 218 | `DRPC_METHOD_MGMT_PREP_SHUTDOWN` | `MethodPrepShutdown` | `src/control/server/ctl_ranks_rpc.go`, `src/control/server/instance_drpc.go` | `ds_mgmt_drpc_prep_shutdown` | `src/mgmt/srv_drpc.c` | +| 220 | `DRPC_METHOD_MGMT_DEV_SET_FAULTY` | `MethodSetFaultyState` | `src/control/server/ctl_smd_rpc.go` | `ds_mgmt_drpc_dev_set_faulty` | `src/mgmt/srv_drpc.c` | +| 221 | `DRPC_METHOD_MGMT_DEV_REPLACE` | `MethodReplaceStorage` | `src/control/server/ctl_smd_rpc.go` | `ds_mgmt_drpc_dev_replace` | `src/mgmt/srv_drpc.c` | +| 222 | `DRPC_METHOD_MGMT_LIST_CONTAINERS` | `MethodListContainers` | `src/control/server/mgmt_cont.go`, `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_list_cont` | `src/mgmt/srv_drpc.c` | +| 223 | `DRPC_METHOD_MGMT_POOL_QUERY` | `MethodPoolQuery` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_query` | `src/mgmt/srv_drpc.c` | +| 224 | `DRPC_METHOD_MGMT_POOL_SET_PROP` | `MethodPoolSetProp` | `src/control/server/mgmt_fi.go`, `src/control/server/mgmt_pool.go`, `src/control/server/mgmt_system.go` | `ds_mgmt_drpc_pool_set_prop` | `src/mgmt/srv_drpc.c` | +| 225 | `DRPC_METHOD_MGMT_PING_RANK` | `MethodPingRank` | `src/control/server/instance_drpc.go` | `ds_mgmt_drpc_ping_rank` | `src/mgmt/srv_drpc.c` | +| 226 | `DRPC_METHOD_MGMT_POOL_REINT` | `MethodPoolReintegrate` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_reintegrate` | `src/mgmt/srv_drpc.c` | +| 227 | `DRPC_METHOD_MGMT_CONT_SET_OWNER` | `MethodContSetOwner` | `src/control/server/mgmt_cont.go` | `ds_mgmt_drpc_cont_set_owner` | `src/mgmt/srv_drpc.c` | +| 228 | `DRPC_METHOD_MGMT_POOL_EXCLUDE` | `MethodPoolExclude` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_exclude` | `src/mgmt/srv_drpc.c` | +| 229 | `DRPC_METHOD_MGMT_POOL_EXTEND` | `MethodPoolExtend` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_extend` | `src/mgmt/srv_drpc.c` | +| 230 | `DRPC_METHOD_MGMT_POOL_EVICT` | `MethodPoolEvict` | `src/control/server/mgmt_pool.go`, `src/control/server/mgmt_system.go` | `ds_mgmt_drpc_pool_evict` | `src/mgmt/srv_drpc.c` | +| 231 | `DRPC_METHOD_MGMT_POOL_DRAIN` | `MethodPoolDrain` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_drain` | `src/mgmt/srv_drpc.c` | +| 232 | `DRPC_METHOD_MGMT_GROUP_UPDATE` | `MethodGroupUpdate` | `src/control/server/mgmt_system.go` | `ds_mgmt_drpc_group_update` | `src/mgmt/srv_drpc.c` | +| 233 | `DRPC_METHOD_MGMT_NOTIFY_EXIT` | `MethodNotifyExit` | `src/control/cmd/daos_agent/mgmt_rpc.go`, `src/control/cmd/daos_agent/procmon.go` | | | +| 235 | `DRPC_METHOD_MGMT_NOTIFY_POOL_CONNECT` | `MethodNotifyPoolConnect` | `src/control/cmd/daos_agent/mgmt_rpc.go`, `src/control/cmd/daos_agent/procmon.go` | | | +| 236 | `DRPC_METHOD_MGMT_NOTIFY_POOL_DISCONNECT` | `MethodNotifyPoolDisconnect` | `src/control/cmd/daos_agent/mgmt_rpc.go`, `src/control/cmd/daos_agent/procmon.go` | | | +| 237 | `DRPC_METHOD_MGMT_POOL_GET_PROP` | `MethodPoolGetProp` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_get_prop` | `src/mgmt/srv_drpc.c` | +| 238 | `DRPC_METHOD_MGMT_SET_LOG_MASKS` | `MethodSetLogMasks` | `src/control/server/ctl_ranks_rpc.go` | `ds_mgmt_drpc_set_log_masks` | `src/mgmt/srv_drpc.c` | +| 239 | `DRPC_METHOD_MGMT_POOL_UPGRADE` | `MethodPoolUpgrade` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_upgrade` | `src/mgmt/srv_drpc.c` | +| 240 | `DRPC_METHOD_MGMT_POOL_QUERY_TARGETS` | `MethodPoolQueryTarget` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_query_targets` | `src/mgmt/srv_drpc.c` | +| 241 | `DRPC_METHOD_MGMT_LED_MANAGE` | `MethodLedManage` | `src/control/server/ctl_smd_rpc.go` | `ds_mgmt_drpc_dev_manage_led` | `src/mgmt/srv_drpc.c` | +| 242 | `DRPC_METHOD_MGMT_CHK_START` | `MethodCheckerStart` | `src/control/server/mgmt_check.go` | `ds_mgmt_drpc_check_start` | `src/mgmt/srv_drpc.c` | +| 243 | `DRPC_METHOD_MGMT_CHK_STOP` | `MethodCheckerStop` | `src/control/server/mgmt_check.go` | `ds_mgmt_drpc_check_stop` | `src/mgmt/srv_drpc.c` | +| 244 | `DRPC_METHOD_MGMT_CHK_QUERY` | `MethodCheckerQuery` | `src/control/server/mgmt_check.go` | `ds_mgmt_drpc_check_query` | `src/mgmt/srv_drpc.c` | +| 245 | `DRPC_METHOD_MGMT_CHK_PROP` | `MethodCheckerProp` | | `ds_mgmt_drpc_check_prop` | `src/mgmt/srv_drpc.c` | +| 246 | `DRPC_METHOD_MGMT_CHK_ACT` | `MethodCheckerAction` | `src/control/server/ctl_check.go` | `ds_mgmt_drpc_check_act` | `src/mgmt/srv_drpc.c` | +| 247 | `DRPC_METHOD_MGMT_SETUP_CLIENT_TELEM` | `MethodSetupClientTelemetry` | `src/control/cmd/daos_agent/mgmt_rpc.go` | | | +| 248 | `DRPC_METHOD_MGMT_GROUP_STATUS_GET` | `MethodGroupStatusGet` | `src/control/server/mgmt_system.go` | `ds_mgmt_drpc_get_group_status` | `src/mgmt/srv_drpc.c` | +| 249 | `DRPC_METHOD_MGMT_CHK_SET_POLICY` | `MethodCheckerSetPolicy` | `src/control/server/mgmt_check.go` | `ds_mgmt_drpc_check_set_policy` | `src/mgmt/srv_drpc.c` | +| 250 | `DRPC_METHOD_MGMT_POOL_REBUILD_START` | `MethodPoolRebuildStart` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_rebuild_start` | `src/mgmt/srv_drpc.c` | +| 251 | `DRPC_METHOD_MGMT_POOL_REBUILD_STOP` | `MethodPoolRebuildStop` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_rebuild_stop` | `src/mgmt/srv_drpc.c` | +| 252 | `DRPC_METHOD_MGMT_POOL_SELF_HEAL_EVAL` | `MethodPoolSelfHealEval` | `src/control/server/mgmt_pool.go` | `ds_mgmt_drpc_pool_self_heal_eval` | `src/mgmt/srv_drpc.c` | +| 301 | `DRPC_METHOD_SRV_NOTIFY_READY` | `MethodNotifyReady` | `src/control/server/mgmt_drpc.go` | | | +| 303 | `DRPC_METHOD_SRV_GET_POOL_SVC` | `MethodGetPoolServiceRanks` | `src/control/server/mgmt_drpc.go` | | | +| 304 | `DRPC_METHOD_SRV_CLUSTER_EVENT` | `MethodClusterEvent` | `src/control/server/mgmt_drpc.go` | | | +| 305 | `DRPC_METHOD_SRV_POOL_FIND_BYLABEL` | `MethodPoolFindByLabel` | `src/control/server/mgmt_drpc.go` | | | +| 306 | `DRPC_METHOD_CHK_LIST_POOL` | `MethodCheckerListPools` | `src/control/server/mgmt_drpc.go` | | | +| 307 | `DRPC_METHOD_CHK_REG_POOL` | `MethodCheckerRegisterPool` | `src/control/server/mgmt_drpc.go` | | | +| 308 | `DRPC_METHOD_CHK_DEREG_POOL` | `MethodCheckerDeregisterPool` | `src/control/server/mgmt_drpc.go` | | | +| 309 | `DRPC_METHOD_CHK_REPORT` | `MethodCheckerReport` | `src/control/server/mgmt_drpc.go` | | | +| 310 | `DRPC_METHOD_SRV_LIST_POOLS` | `MethodListPools` | `src/control/server/mgmt_drpc.go` | | | +| 311 | `DRPC_METHOD_SRV_GET_SYS_PROPS` | `MethodGetSysProps` | `src/control/server/mgmt_drpc.go` | | | +| 401 | `DRPC_METHOD_SEC_VALIDATE_CREDS` | `MethodValidateCredentials` | `src/control/server/security_rpc.go` | | | diff --git a/docs/kb/generated/graph.json b/docs/kb/generated/graph.json new file mode 100644 index 00000000000..064bd3174a0 --- /dev/null +++ b/docs/kb/generated/graph.json @@ -0,0 +1,3867 @@ +{ + "edges": [ + { + "dst": "concepts", + "from": "card", + "kind": "related", + "src": "architecture" + }, + { + "dst": "engine", + "kind": "includes", + "src": "bio", + "weight": 1 + }, + { + "dst": "container", + "kind": "includes", + "src": "cart", + "weight": 1 + }, + { + "dst": "mgmt", + "kind": "includes", + "src": "cart", + "weight": 1 + }, + { + "dst": "object", + "kind": "includes", + "src": "cart", + "weight": 1 + }, + { + "dst": "pipeline", + "kind": "includes", + "src": "cart", + "weight": 1 + }, + { + "dst": "pool", + "kind": "includes", + "src": "cart", + "weight": 1 + }, + { + "dst": "rpc:CHK_ACT", + "kind": "handles_rpc", + "src": "chk" + }, + { + "dst": "rpc:CHK_CONT_LIST", + "kind": "handles_rpc", + "src": "chk" + }, + { + "dst": "rpc:CHK_MARK", + "kind": "handles_rpc", + "src": "chk" + }, + { + "dst": "rpc:CHK_POOL_MBS", + "kind": "handles_rpc", + "src": "chk" + }, + { + "dst": "rpc:CHK_POOL_START", + "kind": "handles_rpc", + "src": "chk" + }, + { + "dst": "rpc:CHK_QUERY", + "kind": "handles_rpc", + "src": "chk" + }, + { + "dst": "rpc:CHK_REJOIN", + "kind": "handles_rpc", + "src": "chk" + }, + { + "dst": "rpc:CHK_REPORT", + "kind": "handles_rpc", + "src": "chk" + }, + { + "dst": "rpc:CHK_SET_POLICY", + "kind": "handles_rpc", + "src": "chk" + }, + { + "dst": "rpc:CHK_START", + "kind": "handles_rpc", + "src": "chk" + }, + { + "dst": "rpc:CHK_STOP", + "kind": "handles_rpc", + "src": "chk" + }, + { + "dst": "engine", + "kind": "includes", + "src": "chk", + "weight": 8 + }, + { + "dst": "mgmt", + "kind": "includes", + "src": "chk", + "weight": 3 + }, + { + "dst": "pool", + "kind": "includes", + "src": "chk", + "weight": 3 + }, + { + "dst": "rsvc", + "kind": "includes", + "src": "chk", + "weight": 2 + }, + { + "dst": "vos", + "kind": "includes", + "src": "chk", + "weight": 3 + }, + { + "dst": "bio", + "kind": "includes", + "src": "common", + "weight": 1 + }, + { + "dst": "architecture", + "from": "card", + "kind": "related", + "src": "concepts" + }, + { + "dst": "rpc:CONT_ACL_DELETE", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_ACL_UPDATE", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_ATTR_DEL", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_ATTR_GET", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_ATTR_LIST", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_ATTR_SET", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_CLOSE", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_CREATE", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_DESTROY", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_DESTROY_BYLABEL", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_EPOCH_AGGREGATE", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_OID_ALLOC", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_OPEN", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_OPEN_BYLABEL", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_PROP_SET", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_PROP_SET_BYLABEL", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_QUERY", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_SNAP_CREATE", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_SNAP_DESTROY", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_SNAP_LIST", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_SNAP_OIT_CREATE", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_SNAP_OIT_DESTROY", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_SNAP_OIT_OID_GET", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_TGT_DESTROY", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_TGT_EPOCH_AGGREGATE", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_TGT_QUERY", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "rpc:CONT_TGT_SNAPSHOT_NOTIFY", + "kind": "handles_rpc", + "src": "container" + }, + { + "dst": "dtx", + "kind": "includes", + "src": "container", + "weight": 1 + }, + { + "dst": "engine", + "kind": "includes", + "src": "container", + "weight": 7 + }, + { + "dst": "object", + "kind": "includes", + "src": "container", + "weight": 1 + }, + { + "dst": "pool", + "kind": "includes", + "src": "container", + "weight": 4 + }, + { + "dst": "rdb", + "kind": "includes", + "src": "container", + "weight": 4 + }, + { + "dst": "rsvc", + "kind": "includes", + "src": "container", + "weight": 1 + }, + { + "dst": "security", + "kind": "includes", + "src": "container", + "weight": 5 + }, + { + "dst": "vos", + "kind": "includes", + "src": "container", + "weight": 3 + }, + { + "dst": "engine", + "from": "card", + "kind": "depends_on", + "src": "control" + }, + { + "dst": "mgmt", + "from": "card", + "kind": "depends_on", + "src": "control" + }, + { + "dst": "proto", + "from": "card", + "kind": "depends_on", + "src": "control" + }, + { + "dst": "security", + "from": "card", + "kind": "depends_on", + "src": "control" + }, + { + "dst": "drpc", + "from": "card", + "kind": "related", + "src": "control" + }, + { + "dst": "engine", + "from": "card", + "kind": "related", + "src": "control" + }, + { + "dst": "flow-pool-create", + "from": "card", + "kind": "related", + "src": "control" + }, + { + "dst": "daos_agent", + "from": "card", + "kind": "used_by", + "src": "control" + }, + { + "dst": "daos_server", + "from": "card", + "kind": "used_by", + "src": "control" + }, + { + "dst": "dmg", + "from": "card", + "kind": "used_by", + "src": "control" + }, + { + "dst": "container", + "from": "card", + "kind": "depends_on", + "src": "dtx" + }, + { + "dst": "engine", + "from": "card", + "kind": "depends_on", + "src": "dtx" + }, + { + "dst": "placement", + "from": "card", + "kind": "depends_on", + "src": "dtx" + }, + { + "dst": "pool", + "from": "card", + "kind": "depends_on", + "src": "dtx" + }, + { + "dst": "vos", + "from": "card", + "kind": "depends_on", + "src": "dtx" + }, + { + "dst": "rpc:DTX_ABORT", + "kind": "handles_rpc", + "src": "dtx" + }, + { + "dst": "rpc:DTX_CHECK", + "kind": "handles_rpc", + "src": "dtx" + }, + { + "dst": "rpc:DTX_COLL_ABORT", + "kind": "handles_rpc", + "src": "dtx" + }, + { + "dst": "rpc:DTX_COLL_CHECK", + "kind": "handles_rpc", + "src": "dtx" + }, + { + "dst": "rpc:DTX_COLL_COMMIT", + "kind": "handles_rpc", + "src": "dtx" + }, + { + "dst": "rpc:DTX_COMMIT", + "kind": "handles_rpc", + "src": "dtx" + }, + { + "dst": "rpc:DTX_REFRESH", + "kind": "handles_rpc", + "src": "dtx" + }, + { + "dst": "container", + "kind": "includes", + "src": "dtx", + "weight": 5 + }, + { + "dst": "engine", + "kind": "includes", + "src": "dtx", + "weight": 5 + }, + { + "dst": "pool", + "kind": "includes", + "src": "dtx", + "weight": 1 + }, + { + "dst": "vos", + "kind": "includes", + "src": "dtx", + "weight": 6 + }, + { + "dst": "cos", + "from": "card", + "kind": "related", + "src": "dtx" + }, + { + "dst": "epoch", + "from": "card", + "kind": "related", + "src": "dtx" + }, + { + "dst": "object", + "from": "card", + "kind": "related", + "src": "dtx" + }, + { + "dst": "container", + "from": "card", + "kind": "used_by", + "src": "dtx" + }, + { + "dst": "object", + "from": "card", + "kind": "used_by", + "src": "dtx" + }, + { + "dst": "rebuild", + "from": "card", + "kind": "used_by", + "src": "dtx" + }, + { + "dst": "vos", + "from": "card", + "kind": "used_by", + "src": "dtx" + }, + { + "dst": "bio", + "from": "card", + "kind": "depends_on", + "src": "engine" + }, + { + "dst": "cart", + "from": "card", + "kind": "depends_on", + "src": "engine" + }, + { + "dst": "chk", + "from": "card", + "kind": "depends_on", + "src": "engine" + }, + { + "dst": "common", + "from": "card", + "kind": "depends_on", + "src": "engine" + }, + { + "dst": "gurt", + "from": "card", + "kind": "depends_on", + "src": "engine" + }, + { + "dst": "vos", + "from": "card", + "kind": "depends_on", + "src": "engine" + }, + { + "dst": "bio", + "kind": "includes", + "src": "engine", + "weight": 2 + }, + { + "dst": "chk", + "kind": "includes", + "src": "engine", + "weight": 1 + }, + { + "dst": "vos", + "kind": "includes", + "src": "engine", + "weight": 3 + }, + { + "dst": "control", + "from": "card", + "kind": "related", + "src": "engine" + }, + { + "dst": "drpc", + "from": "card", + "kind": "related", + "src": "engine" + }, + { + "dst": "iv", + "from": "card", + "kind": "related", + "src": "engine" + }, + { + "dst": "ult", + "from": "card", + "kind": "related", + "src": "engine" + }, + { + "dst": "xstream", + "from": "card", + "kind": "related", + "src": "engine" + }, + { + "dst": "bio", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "chk", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "container", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "dtx", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "mgmt", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "object", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "pipeline", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "pool", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "rdb", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "rebuild", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "rsvc", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "security", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "vos", + "from": "card", + "kind": "used_by", + "src": "engine" + }, + { + "dst": "control", + "from": "card", + "kind": "related", + "src": "flow-drive-replace" + }, + { + "dst": "pool", + "from": "card", + "kind": "related", + "src": "flow-drive-replace" + }, + { + "dst": "rebuild", + "from": "card", + "kind": "related", + "src": "flow-drive-replace" + }, + { + "dst": "control", + "from": "card", + "kind": "related", + "src": "flow-pool-create" + }, + { + "dst": "mgmt", + "from": "card", + "kind": "related", + "src": "flow-pool-create" + }, + { + "dst": "pool", + "from": "card", + "kind": "related", + "src": "flow-pool-create" + }, + { + "dst": "rdb", + "from": "card", + "kind": "related", + "src": "flow-pool-create" + }, + { + "dst": "rsvc", + "from": "card", + "kind": "related", + "src": "flow-pool-create" + }, + { + "dst": "vos", + "from": "card", + "kind": "related", + "src": "flow-pool-create" + }, + { + "dst": "control", + "from": "card", + "kind": "related", + "src": "flow-rebuild" + }, + { + "dst": "dtx", + "from": "card", + "kind": "related", + "src": "flow-rebuild" + }, + { + "dst": "engine", + "from": "card", + "kind": "related", + "src": "flow-rebuild" + }, + { + "dst": "object", + "from": "card", + "kind": "related", + "src": "flow-rebuild" + }, + { + "dst": "placement", + "from": "card", + "kind": "related", + "src": "flow-rebuild" + }, + { + "dst": "pool", + "from": "card", + "kind": "related", + "src": "flow-rebuild" + }, + { + "dst": "rebuild", + "from": "card", + "kind": "related", + "src": "flow-rebuild" + }, + { + "dst": "bio", + "from": "card", + "kind": "related", + "src": "flow-write-path" + }, + { + "dst": "dtx", + "from": "card", + "kind": "related", + "src": "flow-write-path" + }, + { + "dst": "object", + "from": "card", + "kind": "related", + "src": "flow-write-path" + }, + { + "dst": "placement", + "from": "card", + "kind": "related", + "src": "flow-write-path" + }, + { + "dst": "vea", + "from": "card", + "kind": "related", + "src": "flow-write-path" + }, + { + "dst": "vos", + "from": "card", + "kind": "related", + "src": "flow-write-path" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_BIO_HEALTH_QUERY", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_CHK_ACT", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_CHK_PROP", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_CHK_QUERY", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_CHK_SET_POLICY", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_CHK_START", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_CHK_STOP", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_CONT_SET_OWNER", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_DEV_REPLACE", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_DEV_SET_FAULTY", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_GROUP_STATUS_GET", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_GROUP_UPDATE", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_LED_MANAGE", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_LIST_CONTAINERS", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_PING_RANK", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_CREATE", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_DELETE_ACL", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_DESTROY", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_DRAIN", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_EVICT", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_EXCLUDE", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_EXTEND", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_GET_ACL", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_GET_PROP", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_OVERWRITE_ACL", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_QUERY", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_QUERY_TARGETS", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_REBUILD_START", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_REBUILD_STOP", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_REINT", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_SELF_HEAL_EVAL", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_SET_PROP", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_UPDATE_ACL", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_POOL_UPGRADE", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_PREP_SHUTDOWN", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_SET_LOG_MASKS", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_SET_RANK", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_SET_UP", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_SMD_LIST_DEVS", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "drpc:DRPC_METHOD_MGMT_SMD_LIST_POOLS", + "kind": "handles_drpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_GET_BS_STATE", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_MARK", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_PARAMS_SET", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_POOL_FIND", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_POOL_GET_SVCRANKS", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_POOL_LIST", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_PROFILE", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_SVC_RIP", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_TGT_CREATE", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_TGT_DESTROY", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_TGT_MAP_UPDATE", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_TGT_MARK", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_TGT_PARAMS_SET", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_TGT_PROFILE", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "rpc:MGMT_TGT_SHARD_DESTROY", + "kind": "handles_rpc", + "src": "mgmt" + }, + { + "dst": "bio", + "kind": "includes", + "src": "mgmt", + "weight": 5 + }, + { + "dst": "chk", + "kind": "includes", + "src": "mgmt", + "weight": 2 + }, + { + "dst": "container", + "kind": "includes", + "src": "mgmt", + "weight": 1 + }, + { + "dst": "engine", + "kind": "includes", + "src": "mgmt", + "weight": 4 + }, + { + "dst": "pool", + "kind": "includes", + "src": "mgmt", + "weight": 4 + }, + { + "dst": "rdb", + "kind": "includes", + "src": "mgmt", + "weight": 3 + }, + { + "dst": "rsvc", + "kind": "includes", + "src": "mgmt", + "weight": 3 + }, + { + "dst": "security", + "kind": "includes", + "src": "mgmt", + "weight": 1 + }, + { + "dst": "vos", + "kind": "includes", + "src": "mgmt", + "weight": 1 + }, + { + "dst": "bio", + "from": "card", + "kind": "depends_on", + "src": "object" + }, + { + "dst": "cart", + "from": "card", + "kind": "depends_on", + "src": "object" + }, + { + "dst": "container", + "from": "card", + "kind": "depends_on", + "src": "object" + }, + { + "dst": "dtx", + "from": "card", + "kind": "depends_on", + "src": "object" + }, + { + "dst": "engine", + "from": "card", + "kind": "depends_on", + "src": "object" + }, + { + "dst": "placement", + "from": "card", + "kind": "depends_on", + "src": "object" + }, + { + "dst": "pool", + "from": "card", + "kind": "depends_on", + "src": "object" + }, + { + "dst": "rebuild", + "from": "card", + "kind": "depends_on", + "src": "object" + }, + { + "dst": "security", + "from": "card", + "kind": "depends_on", + "src": "object" + }, + { + "dst": "vos", + "from": "card", + "kind": "depends_on", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_AKEY_RPC_ENUMERATE", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_DKEY_RPC_ENUMERATE", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RECX_RPC_ENUMERATE", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_COLL_PUNCH", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_COLL_QUERY", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_CPD", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_EC_AGGREGATE", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_EC_REPLICATE", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_ENUMERATE", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_FETCH", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_KEY2ANCHOR", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_MIGRATE", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_PUNCH", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_PUNCH_AKEYS", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_PUNCH_DKEYS", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_QUERY_KEY", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_SYNC", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_TGT_PUNCH", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_TGT_PUNCH_AKEYS", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_TGT_PUNCH_DKEYS", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_TGT_UPDATE", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "rpc:DAOS_OBJ_RPC_UPDATE", + "kind": "handles_rpc", + "src": "object" + }, + { + "dst": "bio", + "kind": "includes", + "src": "object", + "weight": 4 + }, + { + "dst": "container", + "kind": "includes", + "src": "object", + "weight": 4 + }, + { + "dst": "dtx", + "kind": "includes", + "src": "object", + "weight": 5 + }, + { + "dst": "engine", + "kind": "includes", + "src": "object", + "weight": 8 + }, + { + "dst": "pool", + "kind": "includes", + "src": "object", + "weight": 5 + }, + { + "dst": "rebuild", + "kind": "includes", + "src": "object", + "weight": 3 + }, + { + "dst": "security", + "kind": "includes", + "src": "object", + "weight": 1 + }, + { + "dst": "vos", + "kind": "includes", + "src": "object", + "weight": 6 + }, + { + "dst": "dtx", + "from": "card", + "kind": "related", + "src": "object" + }, + { + "dst": "ec", + "from": "card", + "kind": "related", + "src": "object" + }, + { + "dst": "placement", + "from": "card", + "kind": "related", + "src": "object" + }, + { + "dst": "vos", + "from": "card", + "kind": "related", + "src": "object" + }, + { + "dst": "client", + "from": "card", + "kind": "used_by", + "src": "object" + }, + { + "dst": "container", + "from": "card", + "kind": "used_by", + "src": "object" + }, + { + "dst": "pipeline", + "from": "card", + "kind": "used_by", + "src": "object" + }, + { + "dst": "rebuild", + "from": "card", + "kind": "used_by", + "src": "object" + }, + { + "dst": "rpc:DAOS_PIPELINE_RPC_RUN", + "kind": "handles_rpc", + "src": "pipeline" + }, + { + "dst": "bio", + "kind": "includes", + "src": "pipeline", + "weight": 1 + }, + { + "dst": "container", + "kind": "includes", + "src": "pipeline", + "weight": 1 + }, + { + "dst": "engine", + "kind": "includes", + "src": "pipeline", + "weight": 2 + }, + { + "dst": "vos", + "kind": "includes", + "src": "pipeline", + "weight": 1 + }, + { + "dst": "bio", + "from": "card", + "kind": "depends_on", + "src": "pool" + }, + { + "dst": "cart", + "from": "card", + "kind": "depends_on", + "src": "pool" + }, + { + "dst": "container", + "from": "card", + "kind": "depends_on", + "src": "pool" + }, + { + "dst": "engine", + "from": "card", + "kind": "depends_on", + "src": "pool" + }, + { + "dst": "mgmt", + "from": "card", + "kind": "depends_on", + "src": "pool" + }, + { + "dst": "object", + "from": "card", + "kind": "depends_on", + "src": "pool" + }, + { + "dst": "rdb", + "from": "card", + "kind": "depends_on", + "src": "pool" + }, + { + "dst": "rebuild", + "from": "card", + "kind": "depends_on", + "src": "pool" + }, + { + "dst": "rsvc", + "from": "card", + "kind": "depends_on", + "src": "pool" + }, + { + "dst": "security", + "from": "card", + "kind": "depends_on", + "src": "pool" + }, + { + "dst": "vos", + "from": "card", + "kind": "depends_on", + "src": "pool" + }, + { + "dst": "rpc:POOL_ACL_DELETE", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_ACL_UPDATE", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_ADD_IN", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_ADD_TGT", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_ATTR_DEL", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_ATTR_GET", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_ATTR_LIST", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_ATTR_SET", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_CONNECT", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_CREATE", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_DISCONNECT", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_DRAIN", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_EVAL_SELF_HEAL", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_EVICT", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_EXCLUDE", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_EXCLUDE_OUT", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_EXTEND", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_FILTER_CONT", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_LIST_CONT", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_PROP_GET", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_PROP_SET", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_QUERY", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_QUERY_INFO", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_RANKS_GET", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_REBUILD_START", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_REBUILD_STOP", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_RECOV_CONT", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_REINT", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_REPLICAS_ADD", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_REPLICAS_REMOVE", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_SVC_STOP", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_TGT_DISCARD", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_TGT_DISCONNECT", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_TGT_QUERY", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_TGT_QUERY_MAP", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_TGT_WARMUP", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "rpc:POOL_UPGRADE", + "kind": "handles_rpc", + "src": "pool" + }, + { + "dst": "bio", + "kind": "includes", + "src": "pool", + "weight": 2 + }, + { + "dst": "container", + "kind": "includes", + "src": "pool", + "weight": 3 + }, + { + "dst": "engine", + "kind": "includes", + "src": "pool", + "weight": 6 + }, + { + "dst": "mgmt", + "kind": "includes", + "src": "pool", + "weight": 4 + }, + { + "dst": "object", + "kind": "includes", + "src": "pool", + "weight": 2 + }, + { + "dst": "rdb", + "kind": "includes", + "src": "pool", + "weight": 4 + }, + { + "dst": "rebuild", + "kind": "includes", + "src": "pool", + "weight": 3 + }, + { + "dst": "security", + "kind": "includes", + "src": "pool", + "weight": 3 + }, + { + "dst": "vos", + "kind": "includes", + "src": "pool", + "weight": 3 + }, + { + "dst": "iv", + "from": "card", + "kind": "related", + "src": "pool" + }, + { + "dst": "pool_map", + "from": "card", + "kind": "related", + "src": "pool" + }, + { + "dst": "rdb", + "from": "card", + "kind": "related", + "src": "pool" + }, + { + "dst": "rsvc", + "from": "card", + "kind": "related", + "src": "pool" + }, + { + "dst": "chk", + "from": "card", + "kind": "used_by", + "src": "pool" + }, + { + "dst": "container", + "from": "card", + "kind": "used_by", + "src": "pool" + }, + { + "dst": "dtx", + "from": "card", + "kind": "used_by", + "src": "pool" + }, + { + "dst": "mgmt", + "from": "card", + "kind": "used_by", + "src": "pool" + }, + { + "dst": "object", + "from": "card", + "kind": "used_by", + "src": "pool" + }, + { + "dst": "rebuild", + "from": "card", + "kind": "used_by", + "src": "pool" + }, + { + "dst": "security", + "from": "card", + "kind": "used_by", + "src": "pool" + }, + { + "dst": "rpc:RDB_APPENDENTRIES", + "kind": "handles_rpc", + "src": "rdb" + }, + { + "dst": "rpc:RDB_INSTALLSNAPSHOT", + "kind": "handles_rpc", + "src": "rdb" + }, + { + "dst": "rpc:RDB_REQUESTVOTE", + "kind": "handles_rpc", + "src": "rdb" + }, + { + "dst": "dtx", + "kind": "includes", + "src": "rdb", + "weight": 1 + }, + { + "dst": "engine", + "kind": "includes", + "src": "rdb", + "weight": 5 + }, + { + "dst": "mgmt", + "kind": "includes", + "src": "rdb", + "weight": 1 + }, + { + "dst": "object", + "kind": "includes", + "src": "rdb", + "weight": 1 + }, + { + "dst": "vos", + "kind": "includes", + "src": "rdb", + "weight": 4 + }, + { + "dst": "container", + "from": "card", + "kind": "depends_on", + "src": "rebuild" + }, + { + "dst": "dtx", + "from": "card", + "kind": "depends_on", + "src": "rebuild" + }, + { + "dst": "engine", + "from": "card", + "kind": "depends_on", + "src": "rebuild" + }, + { + "dst": "mgmt", + "from": "card", + "kind": "depends_on", + "src": "rebuild" + }, + { + "dst": "object", + "from": "card", + "kind": "depends_on", + "src": "rebuild" + }, + { + "dst": "placement", + "from": "card", + "kind": "depends_on", + "src": "rebuild" + }, + { + "dst": "pool", + "from": "card", + "kind": "depends_on", + "src": "rebuild" + }, + { + "dst": "security", + "from": "card", + "kind": "depends_on", + "src": "rebuild" + }, + { + "dst": "vos", + "from": "card", + "kind": "depends_on", + "src": "rebuild" + }, + { + "dst": "rpc:REBUILD_OBJECTS_SCAN", + "kind": "handles_rpc", + "src": "rebuild" + }, + { + "dst": "container", + "kind": "includes", + "src": "rebuild", + "weight": 3 + }, + { + "dst": "dtx", + "kind": "includes", + "src": "rebuild", + "weight": 1 + }, + { + "dst": "engine", + "kind": "includes", + "src": "rebuild", + "weight": 5 + }, + { + "dst": "mgmt", + "kind": "includes", + "src": "rebuild", + "weight": 2 + }, + { + "dst": "object", + "kind": "includes", + "src": "rebuild", + "weight": 2 + }, + { + "dst": "pool", + "kind": "includes", + "src": "rebuild", + "weight": 3 + }, + { + "dst": "security", + "kind": "includes", + "src": "rebuild", + "weight": 1 + }, + { + "dst": "vos", + "kind": "includes", + "src": "rebuild", + "weight": 1 + }, + { + "dst": "dtx", + "from": "card", + "kind": "related", + "src": "rebuild" + }, + { + "dst": "iv", + "from": "card", + "kind": "related", + "src": "rebuild" + }, + { + "dst": "object", + "from": "card", + "kind": "related", + "src": "rebuild" + }, + { + "dst": "placement", + "from": "card", + "kind": "related", + "src": "rebuild" + }, + { + "dst": "pool", + "from": "card", + "kind": "related", + "src": "rebuild" + }, + { + "dst": "object", + "from": "card", + "kind": "used_by", + "src": "rebuild" + }, + { + "dst": "pool", + "from": "card", + "kind": "used_by", + "src": "rebuild" + }, + { + "dst": "rpc:RSVC_START", + "kind": "handles_rpc", + "src": "rsvc" + }, + { + "dst": "rpc:RSVC_STOP", + "kind": "handles_rpc", + "src": "rsvc" + }, + { + "dst": "engine", + "kind": "includes", + "src": "rsvc", + "weight": 2 + }, + { + "dst": "rdb", + "kind": "includes", + "src": "rsvc", + "weight": 1 + }, + { + "dst": "engine", + "kind": "includes", + "src": "security", + "weight": 2 + }, + { + "dst": "pool", + "kind": "includes", + "src": "security", + "weight": 1 + }, + { + "dst": "bio", + "kind": "includes", + "src": "utils", + "weight": 5 + }, + { + "dst": "chk", + "kind": "includes", + "src": "utils", + "weight": 2 + }, + { + "dst": "engine", + "kind": "includes", + "src": "utils", + "weight": 5 + }, + { + "dst": "mgmt", + "kind": "includes", + "src": "utils", + "weight": 4 + }, + { + "dst": "vos", + "kind": "includes", + "src": "utils", + "weight": 7 + }, + { + "dst": "bio", + "from": "card", + "kind": "depends_on", + "src": "vos" + }, + { + "dst": "common", + "from": "card", + "kind": "depends_on", + "src": "vos" + }, + { + "dst": "dtx", + "from": "card", + "kind": "depends_on", + "src": "vos" + }, + { + "dst": "vea", + "from": "card", + "kind": "depends_on", + "src": "vos" + }, + { + "dst": "bio", + "kind": "includes", + "src": "vos", + "weight": 4 + }, + { + "dst": "dtx", + "kind": "includes", + "src": "vos", + "weight": 2 + }, + { + "dst": "engine", + "kind": "includes", + "src": "vos", + "weight": 4 + }, + { + "dst": "object", + "kind": "includes", + "src": "vos", + "weight": 2 + }, + { + "dst": "vea", + "kind": "includes", + "src": "vos", + "weight": 1 + }, + { + "dst": "aggregation", + "from": "card", + "kind": "related", + "src": "vos" + }, + { + "dst": "dtx", + "from": "card", + "kind": "related", + "src": "vos" + }, + { + "dst": "epoch", + "from": "card", + "kind": "related", + "src": "vos" + }, + { + "dst": "ilog", + "from": "card", + "kind": "related", + "src": "vos" + }, + { + "dst": "chk", + "from": "card", + "kind": "used_by", + "src": "vos" + }, + { + "dst": "container", + "from": "card", + "kind": "used_by", + "src": "vos" + }, + { + "dst": "dtx", + "from": "card", + "kind": "used_by", + "src": "vos" + }, + { + "dst": "mgmt", + "from": "card", + "kind": "used_by", + "src": "vos" + }, + { + "dst": "object", + "from": "card", + "kind": "used_by", + "src": "vos" + }, + { + "dst": "pipeline", + "from": "card", + "kind": "used_by", + "src": "vos" + }, + { + "dst": "rdb", + "from": "card", + "kind": "used_by", + "src": "vos" + }, + { + "dst": "rebuild", + "from": "card", + "kind": "used_by", + "src": "vos" + } + ], + "nodes": [ + { + "id": "aggregation", + "kind": "concept" + }, + { + "card": "docs/kb/architecture.md", + "id": "architecture", + "kind": "overview" + }, + { + "id": "bio", + "kind": "module" + }, + { + "id": "cart", + "kind": "module" + }, + { + "engine_module": "chk", + "id": "chk", + "kind": "module" + }, + { + "id": "client", + "kind": "module" + }, + { + "id": "common", + "kind": "module" + }, + { + "card": "docs/kb/concepts.md", + "id": "concepts", + "kind": "overview" + }, + { + "engine_module": "cont", + "id": "container", + "kind": "module" + }, + { + "card": "docs/kb/modules/control.md", + "id": "control", + "kind": "module" + }, + { + "id": "cos", + "kind": "concept" + }, + { + "id": "daos_agent", + "kind": "module" + }, + { + "id": "daos_server", + "kind": "module" + }, + { + "id": "dmg", + "kind": "module" + }, + { + "id": "drpc", + "kind": "concept" + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_CHK_DEREG_POOL", + "kind": "drpc", + "method_id": 308 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_CHK_LIST_POOL", + "kind": "drpc", + "method_id": 306 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_CHK_REG_POOL", + "kind": "drpc", + "method_id": 307 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_CHK_REPORT", + "kind": "drpc", + "method_id": 309 + }, + { + "handler": "ds_mgmt_drpc_bio_health_query", + "id": "drpc:DRPC_METHOD_MGMT_BIO_HEALTH_QUERY", + "kind": "drpc", + "method_id": 210 + }, + { + "handler": "ds_mgmt_drpc_check_act", + "id": "drpc:DRPC_METHOD_MGMT_CHK_ACT", + "kind": "drpc", + "method_id": 246 + }, + { + "handler": "ds_mgmt_drpc_check_prop", + "id": "drpc:DRPC_METHOD_MGMT_CHK_PROP", + "kind": "drpc", + "method_id": 245 + }, + { + "handler": "ds_mgmt_drpc_check_query", + "id": "drpc:DRPC_METHOD_MGMT_CHK_QUERY", + "kind": "drpc", + "method_id": 244 + }, + { + "handler": "ds_mgmt_drpc_check_set_policy", + "id": "drpc:DRPC_METHOD_MGMT_CHK_SET_POLICY", + "kind": "drpc", + "method_id": 249 + }, + { + "handler": "ds_mgmt_drpc_check_start", + "id": "drpc:DRPC_METHOD_MGMT_CHK_START", + "kind": "drpc", + "method_id": 242 + }, + { + "handler": "ds_mgmt_drpc_check_stop", + "id": "drpc:DRPC_METHOD_MGMT_CHK_STOP", + "kind": "drpc", + "method_id": 243 + }, + { + "handler": "ds_mgmt_drpc_cont_set_owner", + "id": "drpc:DRPC_METHOD_MGMT_CONT_SET_OWNER", + "kind": "drpc", + "method_id": 227 + }, + { + "handler": "ds_mgmt_drpc_dev_replace", + "id": "drpc:DRPC_METHOD_MGMT_DEV_REPLACE", + "kind": "drpc", + "method_id": 221 + }, + { + "handler": "ds_mgmt_drpc_dev_set_faulty", + "id": "drpc:DRPC_METHOD_MGMT_DEV_SET_FAULTY", + "kind": "drpc", + "method_id": 220 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_MGMT_GET_ATTACH_INFO", + "kind": "drpc", + "method_id": 206 + }, + { + "handler": "ds_mgmt_drpc_get_group_status", + "id": "drpc:DRPC_METHOD_MGMT_GROUP_STATUS_GET", + "kind": "drpc", + "method_id": 248 + }, + { + "handler": "ds_mgmt_drpc_group_update", + "id": "drpc:DRPC_METHOD_MGMT_GROUP_UPDATE", + "kind": "drpc", + "method_id": 232 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_MGMT_KILL_RANK", + "kind": "drpc", + "method_id": 201 + }, + { + "handler": "ds_mgmt_drpc_dev_manage_led", + "id": "drpc:DRPC_METHOD_MGMT_LED_MANAGE", + "kind": "drpc", + "method_id": 241 + }, + { + "handler": "ds_mgmt_drpc_pool_list_cont", + "id": "drpc:DRPC_METHOD_MGMT_LIST_CONTAINERS", + "kind": "drpc", + "method_id": 222 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_MGMT_NOTIFY_EXIT", + "kind": "drpc", + "method_id": 233 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_MGMT_NOTIFY_POOL_CONNECT", + "kind": "drpc", + "method_id": 235 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_MGMT_NOTIFY_POOL_DISCONNECT", + "kind": "drpc", + "method_id": 236 + }, + { + "handler": "ds_mgmt_drpc_ping_rank", + "id": "drpc:DRPC_METHOD_MGMT_PING_RANK", + "kind": "drpc", + "method_id": 225 + }, + { + "handler": "ds_mgmt_drpc_pool_create", + "id": "drpc:DRPC_METHOD_MGMT_POOL_CREATE", + "kind": "drpc", + "method_id": 207 + }, + { + "handler": "ds_mgmt_drpc_pool_delete_acl", + "id": "drpc:DRPC_METHOD_MGMT_POOL_DELETE_ACL", + "kind": "drpc", + "method_id": 217 + }, + { + "handler": "ds_mgmt_drpc_pool_destroy", + "id": "drpc:DRPC_METHOD_MGMT_POOL_DESTROY", + "kind": "drpc", + "method_id": 208 + }, + { + "handler": "ds_mgmt_drpc_pool_drain", + "id": "drpc:DRPC_METHOD_MGMT_POOL_DRAIN", + "kind": "drpc", + "method_id": 231 + }, + { + "handler": "ds_mgmt_drpc_pool_evict", + "id": "drpc:DRPC_METHOD_MGMT_POOL_EVICT", + "kind": "drpc", + "method_id": 230 + }, + { + "handler": "ds_mgmt_drpc_pool_exclude", + "id": "drpc:DRPC_METHOD_MGMT_POOL_EXCLUDE", + "kind": "drpc", + "method_id": 228 + }, + { + "handler": "ds_mgmt_drpc_pool_extend", + "id": "drpc:DRPC_METHOD_MGMT_POOL_EXTEND", + "kind": "drpc", + "method_id": 229 + }, + { + "handler": "ds_mgmt_drpc_pool_get_acl", + "id": "drpc:DRPC_METHOD_MGMT_POOL_GET_ACL", + "kind": "drpc", + "method_id": 213 + }, + { + "handler": "ds_mgmt_drpc_pool_get_prop", + "id": "drpc:DRPC_METHOD_MGMT_POOL_GET_PROP", + "kind": "drpc", + "method_id": 237 + }, + { + "handler": "ds_mgmt_drpc_pool_overwrite_acl", + "id": "drpc:DRPC_METHOD_MGMT_POOL_OVERWRITE_ACL", + "kind": "drpc", + "method_id": 215 + }, + { + "handler": "ds_mgmt_drpc_pool_query", + "id": "drpc:DRPC_METHOD_MGMT_POOL_QUERY", + "kind": "drpc", + "method_id": 223 + }, + { + "handler": "ds_mgmt_drpc_pool_query_targets", + "id": "drpc:DRPC_METHOD_MGMT_POOL_QUERY_TARGETS", + "kind": "drpc", + "method_id": 240 + }, + { + "handler": "ds_mgmt_drpc_pool_rebuild_start", + "id": "drpc:DRPC_METHOD_MGMT_POOL_REBUILD_START", + "kind": "drpc", + "method_id": 250 + }, + { + "handler": "ds_mgmt_drpc_pool_rebuild_stop", + "id": "drpc:DRPC_METHOD_MGMT_POOL_REBUILD_STOP", + "kind": "drpc", + "method_id": 251 + }, + { + "handler": "ds_mgmt_drpc_pool_reintegrate", + "id": "drpc:DRPC_METHOD_MGMT_POOL_REINT", + "kind": "drpc", + "method_id": 226 + }, + { + "handler": "ds_mgmt_drpc_pool_self_heal_eval", + "id": "drpc:DRPC_METHOD_MGMT_POOL_SELF_HEAL_EVAL", + "kind": "drpc", + "method_id": 252 + }, + { + "handler": "ds_mgmt_drpc_pool_set_prop", + "id": "drpc:DRPC_METHOD_MGMT_POOL_SET_PROP", + "kind": "drpc", + "method_id": 224 + }, + { + "handler": "ds_mgmt_drpc_pool_update_acl", + "id": "drpc:DRPC_METHOD_MGMT_POOL_UPDATE_ACL", + "kind": "drpc", + "method_id": 216 + }, + { + "handler": "ds_mgmt_drpc_pool_upgrade", + "id": "drpc:DRPC_METHOD_MGMT_POOL_UPGRADE", + "kind": "drpc", + "method_id": 239 + }, + { + "handler": "ds_mgmt_drpc_prep_shutdown", + "id": "drpc:DRPC_METHOD_MGMT_PREP_SHUTDOWN", + "kind": "drpc", + "method_id": 218 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_MGMT_SETUP_CLIENT_TELEM", + "kind": "drpc", + "method_id": 247 + }, + { + "handler": "ds_mgmt_drpc_set_log_masks", + "id": "drpc:DRPC_METHOD_MGMT_SET_LOG_MASKS", + "kind": "drpc", + "method_id": 238 + }, + { + "handler": "ds_mgmt_drpc_set_rank", + "id": "drpc:DRPC_METHOD_MGMT_SET_RANK", + "kind": "drpc", + "method_id": 202 + }, + { + "handler": "ds_mgmt_drpc_set_up", + "id": "drpc:DRPC_METHOD_MGMT_SET_UP", + "kind": "drpc", + "method_id": 209 + }, + { + "handler": "ds_mgmt_drpc_smd_list_devs", + "id": "drpc:DRPC_METHOD_MGMT_SMD_LIST_DEVS", + "kind": "drpc", + "method_id": 211 + }, + { + "handler": "ds_mgmt_drpc_smd_list_pools", + "id": "drpc:DRPC_METHOD_MGMT_SMD_LIST_POOLS", + "kind": "drpc", + "method_id": 212 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_SEC_AGENT_REQUEST_CREDS", + "kind": "drpc", + "method_id": 101 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_SEC_VALIDATE_CREDS", + "kind": "drpc", + "method_id": 401 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_SRV_CLUSTER_EVENT", + "kind": "drpc", + "method_id": 304 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_SRV_GET_POOL_SVC", + "kind": "drpc", + "method_id": 303 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_SRV_GET_SYS_PROPS", + "kind": "drpc", + "method_id": 311 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_SRV_LIST_POOLS", + "kind": "drpc", + "method_id": 310 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_SRV_NOTIFY_READY", + "kind": "drpc", + "method_id": 301 + }, + { + "handler": "", + "id": "drpc:DRPC_METHOD_SRV_POOL_FIND_BYLABEL", + "kind": "drpc", + "method_id": 305 + }, + { + "card": "docs/kb/modules/dtx.md", + "engine_module": "dtx", + "id": "dtx", + "kind": "module" + }, + { + "id": "ec", + "kind": "concept" + }, + { + "card": "docs/kb/modules/engine.md", + "id": "engine", + "kind": "module" + }, + { + "id": "epoch", + "kind": "concept" + }, + { + "card": "docs/kb/flows/drive-replace.md", + "id": "flow-drive-replace", + "kind": "flow" + }, + { + "card": "docs/kb/flows/pool-create.md", + "id": "flow-pool-create", + "kind": "flow" + }, + { + "card": "docs/kb/flows/rebuild.md", + "id": "flow-rebuild", + "kind": "flow" + }, + { + "card": "docs/kb/flows/write-path.md", + "id": "flow-write-path", + "kind": "flow" + }, + { + "id": "gurt", + "kind": "module" + }, + { + "id": "ilog", + "kind": "concept" + }, + { + "id": "iv", + "kind": "concept" + }, + { + "engine_module": "mgmt", + "id": "mgmt", + "kind": "module" + }, + { + "card": "docs/kb/modules/object.md", + "engine_module": "obj", + "id": "object", + "kind": "module" + }, + { + "engine_module": "pipeline", + "id": "pipeline", + "kind": "module" + }, + { + "id": "placement", + "kind": "concept" + }, + { + "card": "docs/kb/modules/pool.md", + "engine_module": "pool", + "id": "pool", + "kind": "module" + }, + { + "id": "pool_map", + "kind": "concept" + }, + { + "id": "proto", + "kind": "module" + }, + { + "engine_module": "rdb", + "id": "rdb", + "kind": "module" + }, + { + "card": "docs/kb/modules/rebuild.md", + "engine_module": "rebuild", + "id": "rebuild", + "kind": "module" + }, + { + "direction": "server->server", + "handler": "ds_chk_act_hdlr", + "handler_files": [ + "src/chk/chk_srv.c" + ], + "id": "rpc:CHK_ACT", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_chk_cont_list_hdlr", + "handler_files": [ + "src/chk/chk_srv.c" + ], + "id": "rpc:CHK_CONT_LIST", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_chk_mark_hdlr", + "handler_files": [ + "src/chk/chk_srv.c" + ], + "id": "rpc:CHK_MARK", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_chk_pool_mbs_hdlr", + "handler_files": [ + "src/chk/chk_srv.c" + ], + "id": "rpc:CHK_POOL_MBS", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_chk_pool_start_hdlr", + "handler_files": [ + "src/chk/chk_srv.c" + ], + "id": "rpc:CHK_POOL_START", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_chk_query_hdlr", + "handler_files": [ + "src/chk/chk_srv.c" + ], + "id": "rpc:CHK_QUERY", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_chk_rejoin_hdlr", + "handler_files": [ + "src/chk/chk_srv.c" + ], + "id": "rpc:CHK_REJOIN", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_chk_report_hdlr", + "handler_files": [ + "src/chk/chk_srv.c" + ], + "id": "rpc:CHK_REPORT", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_chk_set_policy_hdlr", + "handler_files": [ + "src/chk/chk_srv.c" + ], + "id": "rpc:CHK_SET_POLICY", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_chk_start_hdlr", + "handler_files": [ + "src/chk/chk_srv.c" + ], + "id": "rpc:CHK_START", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_chk_stop_hdlr", + "handler_files": [ + "src/chk/chk_srv.c" + ], + "id": "rpc:CHK_STOP", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_ACL_DELETE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_ACL_UPDATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_ATTR_DEL", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_ATTR_GET", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_ATTR_LIST", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_ATTR_SET", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_CLOSE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_CREATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_DESTROY", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_DESTROY_BYLABEL", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_EPOCH_AGGREGATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_oid_alloc_handler", + "handler_files": [ + "src/container/srv_target.c" + ], + "id": "rpc:CONT_OID_ALLOC", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_OPEN", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_OPEN_BYLABEL", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_PROP_SET", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_cont_set_prop_srv_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_PROP_SET_BYLABEL", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_QUERY", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_SNAP_CREATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_SNAP_DESTROY", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_SNAP_LIST", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_SNAP_OIT_CREATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_SNAP_OIT_DESTROY", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_cont_op_handler", + "handler_files": [ + "src/container/srv_container.c" + ], + "id": "rpc:CONT_SNAP_OIT_OID_GET", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_cont_tgt_destroy_handler", + "handler_files": [ + "src/container/srv_target.c" + ], + "id": "rpc:CONT_TGT_DESTROY", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_cont_tgt_epoch_aggregate_handler", + "handler_files": [ + "src/container/srv_target.c" + ], + "id": "rpc:CONT_TGT_EPOCH_AGGREGATE", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_cont_tgt_query_handler", + "handler_files": [ + "src/container/srv_target.c" + ], + "id": "rpc:CONT_TGT_QUERY", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_cont_tgt_snapshot_notify_handler", + "handler_files": [ + "src/container/srv_target.c" + ], + "id": "rpc:CONT_TGT_SNAPSHOT_NOTIFY", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_enum_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_AKEY_RPC_ENUMERATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_enum_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_DKEY_RPC_ENUMERATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_enum_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RECX_RPC_ENUMERATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_coll_punch_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_COLL_PUNCH", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_coll_query_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_COLL_QUERY", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_cpd_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_CPD", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_ec_agg_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_EC_AGGREGATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_ec_rep_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_EC_REPLICATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_enum_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_ENUMERATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_rw_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_FETCH", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_key2anchor_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_KEY2ANCHOR", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_migrate_handler", + "handler_files": [ + "src/object/srv_obj_migrate.c" + ], + "id": "rpc:DAOS_OBJ_RPC_MIGRATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_punch_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_PUNCH", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_punch_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_PUNCH_AKEYS", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_punch_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_PUNCH_DKEYS", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_query_key_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_QUERY_KEY", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_sync_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_SYNC", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_tgt_punch_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_TGT_PUNCH", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_tgt_punch_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_TGT_PUNCH_AKEYS", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_tgt_punch_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_TGT_PUNCH_DKEYS", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_tgt_update_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_TGT_UPDATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_obj_rw_handler", + "handler_files": [ + "src/object/srv_obj.c" + ], + "id": "rpc:DAOS_OBJ_RPC_UPDATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pipeline_run_handler", + "handler_files": [ + "src/pipeline/srv_pipeline.c" + ], + "id": "rpc:DAOS_PIPELINE_RPC_RUN", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "dtx_handler", + "handler_files": [ + "src/dtx/dtx_srv.c" + ], + "id": "rpc:DTX_ABORT", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "dtx_handler", + "handler_files": [ + "src/dtx/dtx_srv.c" + ], + "id": "rpc:DTX_CHECK", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "dtx_coll_handler", + "handler_files": [ + "src/dtx/dtx_srv.c" + ], + "id": "rpc:DTX_COLL_ABORT", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "dtx_coll_handler", + "handler_files": [ + "src/dtx/dtx_srv.c" + ], + "id": "rpc:DTX_COLL_CHECK", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "dtx_coll_handler", + "handler_files": [ + "src/dtx/dtx_srv.c" + ], + "id": "rpc:DTX_COLL_COMMIT", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "dtx_handler", + "handler_files": [ + "src/dtx/dtx_srv.c" + ], + "id": "rpc:DTX_COMMIT", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "dtx_handler", + "handler_files": [ + "src/dtx/dtx_srv.c" + ], + "id": "rpc:DTX_REFRESH", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_mgmt_hdlr_get_bs_state", + "handler_files": [ + "src/mgmt/srv_query.c" + ], + "id": "rpc:MGMT_GET_BS_STATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_mgmt_mark_hdlr", + "handler_files": [ + "src/mgmt/srv.c" + ], + "id": "rpc:MGMT_MARK", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_mgmt_params_set_hdlr", + "handler_files": [ + "src/mgmt/srv.c" + ], + "id": "rpc:MGMT_PARAMS_SET", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_mgmt_pool_find_hdlr", + "handler_files": [ + "src/mgmt/srv.c" + ], + "id": "rpc:MGMT_POOL_FIND", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_mgmt_pool_get_svcranks_hdlr", + "handler_files": [ + "src/mgmt/srv.c" + ], + "id": "rpc:MGMT_POOL_GET_SVCRANKS", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_mgmt_pool_list_hdlr", + "handler_files": [ + "src/mgmt/srv.c" + ], + "id": "rpc:MGMT_POOL_LIST", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_mgmt_profile_hdlr", + "handler_files": [ + "src/mgmt/srv.c" + ], + "id": "rpc:MGMT_PROFILE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_mgmt_hdlr_svc_rip", + "handler_files": [ + "src/mgmt/srv.c" + ], + "id": "rpc:MGMT_SVC_RIP", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_mgmt_hdlr_tgt_create", + "handler_files": [ + "src/mgmt/srv_target.c" + ], + "id": "rpc:MGMT_TGT_CREATE", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_mgmt_hdlr_tgt_destroy", + "handler_files": [ + "src/mgmt/srv_target.c" + ], + "id": "rpc:MGMT_TGT_DESTROY", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_mgmt_hdlr_tgt_map_update", + "handler_files": [ + "src/mgmt/srv_target.c" + ], + "id": "rpc:MGMT_TGT_MAP_UPDATE", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_mgmt_tgt_mark_hdlr", + "handler_files": [ + "src/mgmt/srv_target.c" + ], + "id": "rpc:MGMT_TGT_MARK", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_mgmt_tgt_params_set_hdlr", + "handler_files": [ + "src/mgmt/srv_target.c" + ], + "id": "rpc:MGMT_TGT_PARAMS_SET", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_mgmt_tgt_profile_hdlr", + "handler_files": [ + "src/mgmt/srv_target.c" + ], + "id": "rpc:MGMT_TGT_PROFILE", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_mgmt_hdlr_tgt_shard_destroy", + "handler_files": [ + "src/mgmt/srv_target.c" + ], + "id": "rpc:MGMT_TGT_SHARD_DESTROY", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_acl_delete_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_ACL_DELETE", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_acl_update_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_ACL_UPDATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_update_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_ADD_IN", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_update_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_ADD_TGT", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_attr_del_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_ATTR_DEL", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_attr_get_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_ATTR_GET", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_attr_list_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_ATTR_LIST", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_attr_set_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_ATTR_SET", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_connect_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_CONNECT", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_create_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_CREATE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_disconnect_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_DISCONNECT", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_update_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_DRAIN", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_eval_self_heal_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_EVAL_SELF_HEAL", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_evict_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_EVICT", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_update_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_EXCLUDE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_update_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_EXCLUDE_OUT", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_extend_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_EXTEND", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_filter_cont_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_FILTER_CONT", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_list_cont_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_LIST_CONT", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_prop_get_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_PROP_GET", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_prop_set_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_PROP_SET", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_query_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_QUERY", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_query_info_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_QUERY_INFO", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_ranks_get_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_RANKS_GET", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_rebuild_start_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_REBUILD_START", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_rebuild_stop_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_REBUILD_STOP", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_recov_cont_handler", + "handler_files": [ + "src/pool/srv_target.c" + ], + "id": "rpc:POOL_RECOV_CONT", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_update_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_REINT", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_replicas_update_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_REPLICAS_ADD", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_replicas_update_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_REPLICAS_REMOVE", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_svc_stop_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_SVC_STOP", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_tgt_discard_handler", + "handler_files": [ + "src/pool/srv_target.c" + ], + "id": "rpc:POOL_TGT_DISCARD", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_tgt_disconnect_handler", + "handler_files": [ + "src/pool/srv_target.c" + ], + "id": "rpc:POOL_TGT_DISCONNECT", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_tgt_query_handler", + "handler_files": [ + "src/pool/srv_target.c" + ], + "id": "rpc:POOL_TGT_QUERY", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_tgt_query_map_handler", + "handler_files": [ + "src/pool/srv_target.c" + ], + "id": "rpc:POOL_TGT_QUERY_MAP", + "kind": "rpc" + }, + { + "direction": "client->server", + "handler": "ds_pool_tgt_warmup_handler", + "handler_files": [ + "src/pool/srv_target.c" + ], + "id": "rpc:POOL_TGT_WARMUP", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_pool_upgrade_handler", + "handler_files": [ + "src/pool/srv_pool.c" + ], + "id": "rpc:POOL_UPGRADE", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "rdb_appendentries_handler", + "handler_files": [ + "src/rdb/rdb_raft.c" + ], + "id": "rpc:RDB_APPENDENTRIES", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "rdb_installsnapshot_handler", + "handler_files": [ + "src/rdb/rdb_raft.c" + ], + "id": "rpc:RDB_INSTALLSNAPSHOT", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "rdb_requestvote_handler", + "handler_files": [ + "src/rdb/rdb_raft.c" + ], + "id": "rpc:RDB_REQUESTVOTE", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "rebuild_tgt_scan_handler", + "handler_files": [ + "src/rebuild/scan.c" + ], + "id": "rpc:REBUILD_OBJECTS_SCAN", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_rsvc_start_handler", + "handler_files": [ + "src/rsvc/srv.c" + ], + "id": "rpc:RSVC_START", + "kind": "rpc" + }, + { + "direction": "server->server", + "handler": "ds_rsvc_stop_handler", + "handler_files": [ + "src/rsvc/srv.c" + ], + "id": "rpc:RSVC_STOP", + "kind": "rpc" + }, + { + "engine_module": "rsvc", + "id": "rsvc", + "kind": "module" + }, + { + "engine_module": "security", + "id": "security", + "kind": "module" + }, + { + "id": "ult", + "kind": "concept" + }, + { + "id": "utils", + "kind": "module" + }, + { + "id": "vea", + "kind": "module" + }, + { + "card": "docs/kb/modules/vos.md", + "engine_module": "vos_srv", + "id": "vos", + "kind": "module" + }, + { + "id": "xstream", + "kind": "concept" + } + ] +} diff --git a/docs/kb/generated/modules.md b/docs/kb/generated/modules.md new file mode 100644 index 00000000000..5a2571809f2 --- /dev/null +++ b/docs/kb/generated/modules.md @@ -0,0 +1,66 @@ + + +# Engine modules and dependencies + +## Engine modules (`struct dss_module`) + +Loaded by `daos_engine` (see `src/engine/module.c`). + +| Name | Module ID | Symbol | Init | File | +|---|---|---|---|---| +| chk | `DAOS_CHK_MODULE` | `chk_module` | `ds_chk_init` | `src/chk/chk_srv.c` | +| cont | `DAOS_CONT_MODULE` | `cont_module` | `init` | `src/container/srv.c` | +| dtx | `DAOS_DTX_MODULE` | `dtx_module` | `dtx_init` | `src/dtx/dtx_srv.c` | +| mgmt | `DAOS_MGMT_MODULE` | `mgmt_module` | `ds_mgmt_init` | `src/mgmt/srv.c` | +| obj | `DAOS_OBJ_MODULE` | `obj_module` | `obj_mod_init` | `src/object/srv_mod.c` | +| pipeline | `DAOS_PIPELINE_MODULE` | `pipeline_module` | `pipeline_mod_init` | `src/pipeline/srv_mod.c` | +| pool | `DAOS_POOL_MODULE` | `pool_module` | `init` | `src/pool/srv.c` | +| rdb | `DAOS_RDB_MODULE` | `rdb_module` | `rdb_module_init` | `src/rdb/rdb_module.c` | +| rebuild | `DAOS_REBUILD_MODULE` | `rebuild_module` | `init` | `src/rebuild/srv.c` | +| rsvc | `DAOS_RSVC_MODULE` | `rsvc_module` | `rsvc_module_init` | `src/rsvc/srv.c` | +| security | `DAOS_SEC_MODULE` | `security_module` | `init` | `src/security/srv.c` | +| vos_srv | `DAOS_VOS_MODULE` | `vos_srv_module` | `vos_mod_init` | `src/vos/vos_common.c` | + +## Module dependencies (from `#include`) + +An edge A -> B means files in `src/A` include a `daos_srv/` header owned by B or a file under `src/B`. The number is how many files do so. + +| Module | Depends on | +|---|---| +| bio | engine (1) | +| cart | container (1), mgmt (1), object (1), pipeline (1), pool (1) | +| chk | engine (8), mgmt (3), pool (3), rsvc (2), vos (3) | +| common | bio (1) | +| container | dtx (1), engine (7), object (1), pool (4), rdb (4), rsvc (1), security (5), vos (3) | +| dtx | container (5), engine (5), pool (1), vos (6) | +| engine | bio (2), chk (1), vos (3) | +| mgmt | bio (5), chk (2), container (1), engine (4), pool (4), rdb (3), rsvc (3), security (1), vos (1) | +| object | bio (4), container (4), dtx (5), engine (8), pool (5), rebuild (3), security (1), vos (6) | +| pipeline | bio (1), container (1), engine (2), vos (1) | +| pool | bio (2), container (3), engine (6), mgmt (4), object (2), rdb (4), rebuild (3), security (3), vos (3) | +| rdb | dtx (1), engine (5), mgmt (1), object (1), vos (4) | +| rebuild | container (3), dtx (1), engine (5), mgmt (2), object (2), pool (3), security (1), vos (1) | +| rsvc | engine (2), rdb (1) | +| security | engine (2), pool (1) | +| utils | bio (5), chk (2), engine (5), mgmt (4), vos (7) | +| vos | bio (4), dtx (2), engine (4), object (2), vea (1) | + +## Reverse dependencies (used by) + +| Module | Used by | +|---|---| +| bio | common, engine, mgmt, object, pipeline, pool, utils, vos | +| chk | engine, mgmt, utils | +| container | cart, dtx, mgmt, object, pipeline, pool, rebuild | +| dtx | container, object, rdb, rebuild, vos | +| engine | bio, chk, container, dtx, mgmt, object, pipeline, pool, rdb, rebuild, rsvc, security, utils, vos | +| mgmt | cart, chk, pool, rdb, rebuild, utils | +| object | cart, container, pool, rdb, rebuild, vos | +| pipeline | cart | +| pool | cart, chk, container, dtx, mgmt, object, rebuild, security | +| rdb | container, mgmt, pool, rsvc | +| rebuild | object, pool | +| rsvc | chk, container, mgmt | +| security | container, mgmt, object, pool, rebuild | +| vea | vos | +| vos | chk, container, dtx, engine, mgmt, object, pipeline, pool, rdb, rebuild, utils | diff --git a/docs/kb/generated/rpcs.md b/docs/kb/generated/rpcs.md new file mode 100644 index 00000000000..cb6a6e625ec --- /dev/null +++ b/docs/kb/generated/rpcs.md @@ -0,0 +1,201 @@ + + +# CaRT RPC opcodes by module + +Source: `*_PROTO_{CLI,SRV}_RPC_LIST` X-macros. client->server RPCs are sent by libdaos; server->server RPCs are engine-internal (collective = has corpc ops). + +## chk + +Defined in `src/chk/chk_internal.h` + +| Opcode | Direction | Handler | Handler file | Collective | +|---|---|---|---|---| +| `CHK_START` | server->server | `ds_chk_start_hdlr` | `src/chk/chk_srv.c` | yes | +| `CHK_STOP` | server->server | `ds_chk_stop_hdlr` | `src/chk/chk_srv.c` | yes | +| `CHK_QUERY` | server->server | `ds_chk_query_hdlr` | `src/chk/chk_srv.c` | yes | +| `CHK_MARK` | server->server | `ds_chk_mark_hdlr` | `src/chk/chk_srv.c` | yes | +| `CHK_ACT` | server->server | `ds_chk_act_hdlr` | `src/chk/chk_srv.c` | | +| `CHK_CONT_LIST` | server->server | `ds_chk_cont_list_hdlr` | `src/chk/chk_srv.c` | yes | +| `CHK_POOL_START` | server->server | `ds_chk_pool_start_hdlr` | `src/chk/chk_srv.c` | yes | +| `CHK_POOL_MBS` | server->server | `ds_chk_pool_mbs_hdlr` | `src/chk/chk_srv.c` | | +| `CHK_REPORT` | server->server | `ds_chk_report_hdlr` | `src/chk/chk_srv.c` | | +| `CHK_REJOIN` | server->server | `ds_chk_rejoin_hdlr` | `src/chk/chk_srv.c` | | +| `CHK_SET_POLICY` | server->server | `ds_chk_set_policy_hdlr` | `src/chk/chk_srv.c` | yes | + +## container + +Defined in `src/container/rpc.h` + +| Opcode | Direction | Handler | Handler file | Collective | +|---|---|---|---|---| +| `CONT_CREATE` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_DESTROY` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_OPEN` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_CLOSE` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_QUERY` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_OID_ALLOC` | client->server | `ds_cont_oid_alloc_handler` | `src/container/srv_target.c` | | +| `CONT_ATTR_LIST` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_ATTR_GET` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_ATTR_SET` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_ATTR_DEL` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_EPOCH_AGGREGATE` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_SNAP_LIST` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_SNAP_CREATE` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_SNAP_DESTROY` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_PROP_SET` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_ACL_UPDATE` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_ACL_DELETE` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_OPEN_BYLABEL` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_DESTROY_BYLABEL` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_SNAP_OIT_OID_GET` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_SNAP_OIT_CREATE` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_SNAP_OIT_DESTROY` | client->server | `ds_cont_op_handler` | `src/container/srv_container.c` | | +| `CONT_TGT_DESTROY` | server->server | `ds_cont_tgt_destroy_handler` | `src/container/srv_target.c` | yes | +| `CONT_TGT_QUERY` | server->server | `ds_cont_tgt_query_handler` | `src/container/srv_target.c` | yes | +| `CONT_TGT_EPOCH_AGGREGATE` | server->server | `ds_cont_tgt_epoch_aggregate_handler` | `src/container/srv_target.c` | yes | +| `CONT_TGT_SNAPSHOT_NOTIFY` | server->server | `ds_cont_tgt_snapshot_notify_handler` | `src/container/srv_target.c` | yes | +| `CONT_PROP_SET_BYLABEL` | server->server | `ds_cont_set_prop_srv_handler` | `src/container/srv_container.c` | | + +## dtx + +Defined in `src/dtx/dtx_internal.h` + +| Opcode | Direction | Handler | Handler file | Collective | +|---|---|---|---|---| +| `DTX_COMMIT` | server->server | `dtx_handler` | `src/dtx/dtx_srv.c` | | +| `DTX_ABORT` | server->server | `dtx_handler` | `src/dtx/dtx_srv.c` | | +| `DTX_CHECK` | server->server | `dtx_handler` | `src/dtx/dtx_srv.c` | | +| `DTX_REFRESH` | server->server | `dtx_handler` | `src/dtx/dtx_srv.c` | | +| `DTX_COLL_COMMIT` | server->server | `dtx_coll_handler` | `src/dtx/dtx_srv.c` | yes | +| `DTX_COLL_ABORT` | server->server | `dtx_coll_handler` | `src/dtx/dtx_srv.c` | yes | +| `DTX_COLL_CHECK` | server->server | `dtx_coll_handler` | `src/dtx/dtx_srv.c` | yes | + +## mgmt + +Defined in `src/mgmt/rpc.h` + +| Opcode | Direction | Handler | Handler file | Collective | +|---|---|---|---|---| +| `MGMT_SVC_RIP` | client->server | `ds_mgmt_hdlr_svc_rip` | `src/mgmt/srv.c` | | +| `MGMT_PARAMS_SET` | client->server | `ds_mgmt_params_set_hdlr` | `src/mgmt/srv.c` | | +| `MGMT_PROFILE` | client->server | `ds_mgmt_profile_hdlr` | `src/mgmt/srv.c` | | +| `MGMT_POOL_GET_SVCRANKS` | client->server | `ds_mgmt_pool_get_svcranks_hdlr` | `src/mgmt/srv.c` | | +| `MGMT_POOL_FIND` | client->server | `ds_mgmt_pool_find_hdlr` | `src/mgmt/srv.c` | | +| `MGMT_POOL_LIST` | client->server | `ds_mgmt_pool_list_hdlr` | `src/mgmt/srv.c` | | +| `MGMT_MARK` | client->server | `ds_mgmt_mark_hdlr` | `src/mgmt/srv.c` | | +| `MGMT_GET_BS_STATE` | client->server | `ds_mgmt_hdlr_get_bs_state` | `src/mgmt/srv_query.c` | | +| `MGMT_TGT_CREATE` | server->server | `ds_mgmt_hdlr_tgt_create` | `src/mgmt/srv_target.c` | yes | +| `MGMT_TGT_DESTROY` | server->server | `ds_mgmt_hdlr_tgt_destroy` | `src/mgmt/srv_target.c` | yes | +| `MGMT_TGT_PARAMS_SET` | server->server | `ds_mgmt_tgt_params_set_hdlr` | `src/mgmt/srv_target.c` | | +| `MGMT_TGT_PROFILE` | server->server | `ds_mgmt_tgt_profile_hdlr` | `src/mgmt/srv_target.c` | | +| `MGMT_TGT_MAP_UPDATE` | server->server | `ds_mgmt_hdlr_tgt_map_update` | `src/mgmt/srv_target.c` | yes | +| `MGMT_TGT_MARK` | server->server | `ds_mgmt_tgt_mark_hdlr` | `src/mgmt/srv_target.c` | | +| `MGMT_TGT_SHARD_DESTROY` | server->server | `ds_mgmt_hdlr_tgt_shard_destroy` | `src/mgmt/srv_target.c` | | + +## object + +Defined in `src/object/obj_rpc.h` + +| Opcode | Direction | Handler | Handler file | Collective | +|---|---|---|---|---| +| `DAOS_OBJ_RPC_UPDATE` | client->server | `ds_obj_rw_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_FETCH` | client->server | `ds_obj_rw_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_DKEY_RPC_ENUMERATE` | client->server | `ds_obj_enum_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_AKEY_RPC_ENUMERATE` | client->server | `ds_obj_enum_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RECX_RPC_ENUMERATE` | client->server | `ds_obj_enum_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_ENUMERATE` | client->server | `ds_obj_enum_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_PUNCH` | client->server | `ds_obj_punch_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_PUNCH_DKEYS` | client->server | `ds_obj_punch_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_PUNCH_AKEYS` | client->server | `ds_obj_punch_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_QUERY_KEY` | client->server | `ds_obj_query_key_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_SYNC` | client->server | `ds_obj_sync_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_TGT_UPDATE` | client->server | `ds_obj_tgt_update_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_TGT_PUNCH` | client->server | `ds_obj_tgt_punch_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_TGT_PUNCH_DKEYS` | client->server | `ds_obj_tgt_punch_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_TGT_PUNCH_AKEYS` | client->server | `ds_obj_tgt_punch_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_MIGRATE` | client->server | `ds_obj_migrate_handler` | `src/object/srv_obj_migrate.c` | | +| `DAOS_OBJ_RPC_EC_AGGREGATE` | client->server | `ds_obj_ec_agg_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_EC_REPLICATE` | client->server | `ds_obj_ec_rep_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_CPD` | client->server | `ds_obj_cpd_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_KEY2ANCHOR` | client->server | `ds_obj_key2anchor_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_COLL_PUNCH` | client->server | `ds_obj_coll_punch_handler` | `src/object/srv_obj.c` | | +| `DAOS_OBJ_RPC_COLL_QUERY` | client->server | `ds_obj_coll_query_handler` | `src/object/srv_obj.c` | | + +## pipeline + +Defined in `src/pipeline/pipeline_rpc.h` + +| Opcode | Direction | Handler | Handler file | Collective | +|---|---|---|---|---| +| `DAOS_PIPELINE_RPC_RUN` | client->server | `ds_pipeline_run_handler` | `src/pipeline/srv_pipeline.c` | | + +## pool + +Defined in `src/pool/rpc.h` + +| Opcode | Direction | Handler | Handler file | Collective | +|---|---|---|---|---| +| `POOL_CREATE` | client->server | `ds_pool_create_handler` | `src/pool/srv_pool.c` | | +| `POOL_CONNECT` | client->server | `ds_pool_connect_handler` | `src/pool/srv_pool.c` | | +| `POOL_DISCONNECT` | client->server | `ds_pool_disconnect_handler` | `src/pool/srv_pool.c` | | +| `POOL_QUERY` | client->server | `ds_pool_query_handler` | `src/pool/srv_pool.c` | | +| `POOL_QUERY_INFO` | client->server | `ds_pool_query_info_handler` | `src/pool/srv_pool.c` | | +| `POOL_EXCLUDE` | client->server | `ds_pool_update_handler` | `src/pool/srv_pool.c` | | +| `POOL_DRAIN` | client->server | `ds_pool_update_handler` | `src/pool/srv_pool.c` | | +| `POOL_EXTEND` | client->server | `ds_pool_extend_handler` | `src/pool/srv_pool.c` | | +| `POOL_EVICT` | client->server | `ds_pool_evict_handler` | `src/pool/srv_pool.c` | | +| `POOL_REINT` | client->server | `ds_pool_update_handler` | `src/pool/srv_pool.c` | | +| `POOL_ADD_IN` | client->server | `ds_pool_update_handler` | `src/pool/srv_pool.c` | | +| `POOL_EXCLUDE_OUT` | client->server | `ds_pool_update_handler` | `src/pool/srv_pool.c` | | +| `POOL_SVC_STOP` | client->server | `ds_pool_svc_stop_handler` | `src/pool/srv_pool.c` | | +| `POOL_ATTR_LIST` | client->server | `ds_pool_attr_list_handler` | `src/pool/srv_pool.c` | | +| `POOL_ATTR_GET` | client->server | `ds_pool_attr_get_handler` | `src/pool/srv_pool.c` | | +| `POOL_ATTR_SET` | client->server | `ds_pool_attr_set_handler` | `src/pool/srv_pool.c` | | +| `POOL_ATTR_DEL` | client->server | `ds_pool_attr_del_handler` | `src/pool/srv_pool.c` | | +| `POOL_REPLICAS_ADD` | client->server | `ds_pool_replicas_update_handler` | `src/pool/srv_pool.c` | | +| `POOL_REPLICAS_REMOVE` | client->server | `ds_pool_replicas_update_handler` | `src/pool/srv_pool.c` | | +| `POOL_LIST_CONT` | client->server | `ds_pool_list_cont_handler` | `src/pool/srv_pool.c` | | +| `POOL_TGT_QUERY_MAP` | client->server | `ds_pool_tgt_query_map_handler` | `src/pool/srv_target.c` | | +| `POOL_FILTER_CONT` | client->server | `ds_pool_filter_cont_handler` | `src/pool/srv_pool.c` | | +| `POOL_TGT_WARMUP` | client->server | `ds_pool_tgt_warmup_handler` | `src/pool/srv_target.c` | | +| `POOL_TGT_DISCONNECT` | server->server | `ds_pool_tgt_disconnect_handler` | `src/pool/srv_target.c` | yes | +| `POOL_TGT_QUERY` | server->server | `ds_pool_tgt_query_handler` | `src/pool/srv_target.c` | yes | +| `POOL_PROP_GET` | server->server | `ds_pool_prop_get_handler` | `src/pool/srv_pool.c` | | +| `POOL_ADD_TGT` | server->server | `ds_pool_update_handler` | `src/pool/srv_pool.c` | | +| `POOL_PROP_SET` | server->server | `ds_pool_prop_set_handler` | `src/pool/srv_pool.c` | | +| `POOL_ACL_UPDATE` | server->server | `ds_pool_acl_update_handler` | `src/pool/srv_pool.c` | | +| `POOL_ACL_DELETE` | server->server | `ds_pool_acl_delete_handler` | `src/pool/srv_pool.c` | | +| `POOL_RANKS_GET` | server->server | `ds_pool_ranks_get_handler` | `src/pool/srv_pool.c` | | +| `POOL_UPGRADE` | server->server | `ds_pool_upgrade_handler` | `src/pool/srv_pool.c` | | +| `POOL_TGT_DISCARD` | server->server | `ds_pool_tgt_discard_handler` | `src/pool/srv_target.c` | | +| `POOL_REBUILD_STOP` | server->server | `ds_pool_rebuild_stop_handler` | `src/pool/srv_pool.c` | | +| `POOL_REBUILD_START` | server->server | `ds_pool_rebuild_start_handler` | `src/pool/srv_pool.c` | | +| `POOL_EVAL_SELF_HEAL` | server->server | `ds_pool_eval_self_heal_handler` | `src/pool/srv_pool.c` | | +| `POOL_RECOV_CONT` | server->server | `ds_pool_recov_cont_handler` | `src/pool/srv_target.c` | | + +## rdb + +Defined in `src/rdb/rdb_internal.h` + +| Opcode | Direction | Handler | Handler file | Collective | +|---|---|---|---|---| +| `RDB_REQUESTVOTE` | server->server | `rdb_requestvote_handler` | `src/rdb/rdb_raft.c` | | +| `RDB_APPENDENTRIES` | server->server | `rdb_appendentries_handler` | `src/rdb/rdb_raft.c` | | +| `RDB_INSTALLSNAPSHOT` | server->server | `rdb_installsnapshot_handler` | `src/rdb/rdb_raft.c` | | + +## rebuild + +Defined in `src/rebuild/rpc.h` + +| Opcode | Direction | Handler | Handler file | Collective | +|---|---|---|---|---| +| `REBUILD_OBJECTS_SCAN` | server->server | `rebuild_tgt_scan_handler` | `src/rebuild/scan.c` | yes | + +## rsvc + +Defined in `src/rsvc/rpc.h` + +| Opcode | Direction | Handler | Handler file | Collective | +|---|---|---|---|---| +| `RSVC_START` | server->server | `ds_rsvc_start_handler` | `src/rsvc/srv.c` | yes | +| `RSVC_STOP` | server->server | `ds_rsvc_stop_handler` | `src/rsvc/srv.c` | yes | diff --git a/docs/kb/modules/control.md b/docs/kb/modules/control.md new file mode 100644 index 00000000000..41bcc374dd2 --- /dev/null +++ b/docs/kb/modules/control.md @@ -0,0 +1,138 @@ +--- +id: control +kind: module +layer: control-plane (Go) +depends_on: [engine, mgmt, proto, security] +used_by: [dmg, daos_agent, daos_server] +related: [drpc, flow-pool-create, engine] +--- +# Control plane (`src/control`, Go) + +Overview: [src/control/README.md](../../../src/control/README.md), +[src/control/server/README.md](../../../src/control/server/README.md), +[src/control/drpc/README.md](../../../src/control/drpc/README.md), +[src/control/lib/control/README.md](../../../src/control/lib/control/README.md). +Protobufs: `src/proto` (Go stubs in `src/control/common/proto`). Never edit +`src/control/vendor`. + +## Purpose + +All management of a DAOS system: parse config, prepare/format SCM and NVMe, start and supervise +`daos_engine` processes, run the replicated **management service (MS)** (system membership, pool +service records, system-wide operations) and expose it over gRPC to `dmg`, `daos_agent` and +other servers. The control plane talks to its local engines over dRPC; it never touches pool +data directly. + +## Binaries (`src/control/cmd`) + +| Binary | Role | +|---|---| +| `src/control/cmd/daos_server` | Server daemon: config, storage, engine supervision, gRPC server, MS replica | +| `src/control/cmd/daos_agent` | Client-node daemon: dRPC server for libdaos (attach info, credential signing), caches system info | +| `src/control/cmd/dmg` | Admin CLI (go-flags commands -> `lib/control` calls) | +| `src/control/cmd/daos` | User CLI; uses libdaos through cgo | +| `src/control/cmd/daos_server_helper` | Privileged helper for storage operations (via `src/control/pbin`) | +| `src/control/cmd/ddb` | VOS debug tool front end | + +## Package map + +| Package | Role | +|---|---| +| `src/control/server` | `daos_server` core: startup (`server.go`), MS (`mgmt_*.go`), ctl service (`ctl_*.go`), engine harness/instances, dRPC modules for engine upcalls | +| `src/control/server/config` | `daos_server.yml` model and validation | +| `src/control/server/engine` | Engine config and process runner | +| `src/control/server/storage` | SCM/NVMe/metadata providers and formatting | +| `src/control/system` | Membership, members, pool service records, fault domains | +| `src/control/system/raft` | MS Raft database (hashicorp raft + bbolt): members, pools, system props | +| `src/control/lib/control` | Client API used by `dmg`/agent: request/response types, `InvokeUnaryRPC` | +| `src/control/lib/daos` | cgo bindings for DAOS constants (status codes, dRPC method IDs) | +| `src/control/drpc` | dRPC client/server, `Module` interface | +| `src/control/security` | Certificates, credentials, ACLs, gRPC authorization | +| `src/control/events` | RAS events and pub/sub, forwarding to the MS leader | +| `src/control/common/proto` | Generated protobuf Go code (`ctl`, `mgmt`, `srv`, `shared`, ...) | +| `src/control/lib/hardware`, `src/control/lib/ranklist`, `src/control/lib/spdk` | Topology/fabric discovery, rank sets, SPDK bindings | + +## Startup of `daos_server` + +1. `src/control/server/server.go::Start` -> `newServer` -> `createServices` (MS + ctl services), + `addEngines` (one `EngineInstance` per configured engine), `setupGrpc`, `start`. +2. `src/control/server/harness.go::EngineHarness.Start` runs each instance + (`src/control/server/instance_exec.go::EngineInstance.Run`): wait for storage + (format via `src/control/server/ctl_storage_rpc.go::ControlService.StorageFormat` if needed), + launch `daos_engine` with `src/control/server/engine/exec.go::Runner.Start`. +3. Engine reports ready over dRPC: `DRPC_METHOD_SRV_NOTIFY_READY` -> + `src/control/server/mgmt_drpc.go::srvModule.handleNotifyReady` -> + `src/control/server/instance_drpc.go::EngineInstance.NotifyDrpcReady`. +4. `src/control/server/instance_exec.go::EngineInstance.finishStartup` -> `handleReady`: + `determineRank` joins the system through gRPC + `src/control/server/mgmt_system.go::mgmtSvc.Join` on the MS leader, then + `src/control/server/instance.go::EngineInstance.SetupRank` sends dRPC `SET_RANK` and + `SET_UP` (engine side: [engine.md](engine.md)). + +## Management service (MS) + +- `src/control/server/mgmt_svc.go::mgmtSvc` implements gRPC `MgmtSvc` (`src/proto/mgmt/mgmt.proto`). + Only MS replicas (configured by `mgmt_svc_replicas` in `daos_server.yml`) run Raft; requests + that need the leader are checked with `src/control/server/mgmt_svc.go::mgmtSvc.checkLeaderRequest` + (`src/control/system/raft/database.go::Database.CheckLeader`). +- State lives in `src/control/system/raft/database.go::Database` and is changed only by + submitting Raft log entries (`src/control/system/raft/raft.go::Database.submitRaftUpdate`, + applied in `src/control/system/raft/raft.go::fsm.Apply`). +- Membership: `src/control/system/membership.go::Membership`; pool service records and states: + `src/control/system/pool.go`. +- Pool ops in `src/control/server/mgmt_pool.go` (`PoolCreate`, `PoolExclude`, `PoolDrain`, + `PoolReintegrate`, `PoolExtend`, `PoolRebuildStart`, ...) resolve the pool, then forward to an + engine with `src/control/server/harness.go::EngineHarness.CallDrpc`. See + [flows/pool-create.md](../flows/pool-create.md) and [generated/drpc.md](../generated/drpc.md). +- System ops in `src/control/server/mgmt_system.go` (`SystemQuery`, `SystemStop`, + `SystemStart`, `SystemExclude`, `SystemRebuildManage`, `GetAttachInfo`, `ClusterEvent`). + +## gRPC services + +| Service | Proto | Server files | Scope | +|---|---|---|---| +| `CtlSvc` | `src/proto/ctl` | `src/control/server/ctl_*.go` (`ControlService`) | Per-host: storage/network scan, storage format, SMD, ranks | +| `MgmtSvc` | `src/proto/mgmt` | `src/control/server/mgmt_*.go` (`mgmtSvc`) | System-wide via MS leader: pools, containers, system, checker | + +Client side: `src/control/lib/control/rpc.go::Client.InvokeUnaryRPC` sends a request either to a +list of hosts (ctl) or to the MS leader (mgmt; retried on leader changes). Multi-host responses +embed `src/control/lib/control/response.go::HostErrorsResp`; MS responses are unpacked with +`src/control/lib/control/response.go::convertMSResponse`. + +## dRPC + +- Go side of the protocol: `src/control/drpc` (`ClientConnection`, `DomainSocketServer`, + `Module`). Method IDs come from C via `src/control/lib/daos/drpc.go`. +- Server -> engine: `EngineHarness.CallDrpc` -> `src/control/server/instance_drpc.go::EngineInstance.CallDrpc`. +- Engine -> server: `src/control/server/mgmt_drpc.go::srvModule` (notify ready, cluster events, + pool service lookups), checker upcalls in `src/control/server/mgmt_drpc_checker.go`. +- libdaos -> agent: `src/control/cmd/daos_agent/mgmt_rpc.go::mgmtModule.handleGetAttachInfo` + (served from `src/control/cmd/daos_agent/infocache.go::InfoCache`) and + `src/control/cmd/daos_agent/security_rpc.go::SecurityModule` (signed credentials). + +## Invariants and gotchas + +- MS state changes must go through Raft (`Database` methods), never mutate in-memory copies. +- Leader-only RPCs must call `checkLeaderRequest`; clients rely on `lib/control` to find the leader. +- dRPC method/proto changes touch C and Go together: `src/include/daos/drpc_modules.h`, + `src/control/lib/daos/drpc.go`, `src/proto/*`, the engine handler in `src/mgmt`. Regenerate + protobuf code rather than editing `*.pb.go` / `*.pb-c.c`. +- Privileged operations go through `daos_server_helper` (`src/control/pbin`), not directly. +- Config examples: `utils/config/daos_server.yml`, `utils/config/daos_agent.yml`, + `utils/config/daos_control.yml`. + +## Tests + +- Unit: `*_test.go` next to the code; mocks in files such as `src/control/server/mocks.go` + and `src/control/system/raft/mocks.go`. Run `src/control/run_go_tests.sh`. +- Functional: `src/tests/ftest/control` and the rest of `src/tests/ftest`. + +## Where to look for... + +| Question | Look at | +|---|---| +| A `dmg` command's implementation | `src/control/cmd/dmg/.go` -> `src/control/lib/control/.go` -> `src/control/server/mgmt_.go` | +| What happens when an engine dies | `src/control/server/instance_exec.go::EngineInstance.handleExit`, `src/control/events` | +| How a rank joins | `src/control/server/mgmt_system.go::mgmtSvc.Join` | +| Storage format | `src/control/server/ctl_storage_rpc.go::ControlService.StorageFormat`, `src/control/server/storage` | +| Server config fields | `src/control/server/config/server.go::Server` | diff --git a/docs/kb/modules/dtx.md b/docs/kb/modules/dtx.md new file mode 100644 index 00000000000..4d331332013 --- /dev/null +++ b/docs/kb/modules/dtx.md @@ -0,0 +1,93 @@ +--- +id: dtx +kind: module +layer: server/transaction +depends_on: [vos, container, pool, engine, placement] +used_by: [object, container, rebuild, vos] +related: [epoch, cos, object] +--- +# DTX: distributed transactions (`src/dtx`) + +Design: [docs/overview/transaction.md](../../overview/transaction.md). Server API: +`src/include/daos_srv/dtx_srv.h`; shared types/states: `src/include/daos/dtx.h`. + +## Purpose + +Makes every modification that touches more than one shard (replicas, EC data + parity, multi-dkey +transactions) atomic. One shard is the **leader**; it executes locally and forwards to +**non-leaders**, collects results, then commits or aborts. Commit is normally **asynchronous**: +the DTX becomes *committable*, is cached in the Commit-on-Share (CoS) table and committed later in +batches (or piggybacked to non-leaders on the next RPC). Local persistent state lives in VOS +(`src/vos/vos_dtx.c`). + +## File map + +| File | Role | +|---|---| +| `src/dtx/dtx_common.c` | Handle lifecycle (`dtx_begin`/`dtx_end`, `dtx_leader_begin`/`dtx_leader_end`), forwarding (`dtx_leader_exec_ops`), batched-commit / aggregation / cleanup ULTs, resend detection | +| `src/dtx/dtx_rpc.c` | Sending DTX_COMMIT / ABORT / CHECK / REFRESH (`dtx_commit`, `dtx_abort`, `dtx_check`, `dtx_refresh`), collective variants | +| `src/dtx/dtx_srv.c` | Module registration (`dtx_module`), RPC handlers `dtx_handler`, `dtx_coll_handler`, starts `dtx_batched_commit` ULT on every target | +| `src/dtx/dtx_cos.c` | CoS cache: add committable DTX, fetch batches, piggyback lists | +| `src/dtx/dtx_resync.c` | Resolve in-doubt DTXs after a leader/pool-map change (`dtx_resync`) | +| `src/dtx/dtx_coll.c` | Collective DTX (one DTX spanning many targets, e.g. object-wide punch) | +| `src/dtx/dtx_internal.h` | RPC list (`DTX_PROTO_SRV_RPC_LIST`), thresholds, internal types | + +## Lifecycle on the write path + +``` +leader (target xstream) non-leader +dtx_leader_begin() +dtx_leader_exec_ops(dlh, obj_tgt_update, ...) --DAOS_OBJ_RPC_TGT_UPDATE--> ds_obj_tgt_update_handler + local: obj_local_rw() dtx_begin() + obj_local_rw() (state: prepared) + dtx_end() +dtx_leader_end() + all ok + sync -> vos_dtx_mark_committable + dtx_commit() now + all ok + async -> vos_dtx_mark_committable + dtx_cos_add() -> later batched commit / piggyback + failure -> dtx_abort() +``` + +- Leader side: `src/dtx/dtx_common.c::dtx_leader_begin`, `src/dtx/dtx_common.c::dtx_leader_exec_ops`, + `src/dtx/dtx_common.c::dtx_leader_end`. +- Non-leader side: `src/dtx/dtx_common.c::dtx_begin`, `src/dtx/dtx_common.c::dtx_end`. +- Async commit: `src/dtx/dtx_cos.c::dtx_cos_add`, `src/dtx/dtx_common.c::dtx_batched_commit`, + `src/dtx/dtx_rpc.c::dtx_commit`. Piggyback: `src/dtx/dtx_cos.c::dtx_cos_get_piggyback`. +- Solo DTX (single shard, e.g. non-replicated object): no forwarding; validated inside VOS. + +## States (`src/include/daos/dtx.h`) + +`DTX_ST_PREPARED` (locally applied, not yet decided) -> `DTX_ST_COMMITTABLE` (all participants +prepared; leader decided) -> `DTX_ST_COMMITTED`; or `DTX_ST_ABORTED`. Also `DTX_ST_CORRUPTED`, +and transient `DTX_ST_PREPARING` / `DTX_ST_COMMITTING` / `DTX_ST_ABORTING`. + +## Key structures + +- `src/include/daos_srv/dtx_srv.h::dtx_handle`: per-operation state (xid, epoch, flags, modified + records, CoS lists). `dth_solo`, `dth_sync` select solo / synchronous commit. +- `src/include/daos_srv/dtx_srv.h::dtx_leader_handle`: leader extension with sub-request status. +- `src/include/daos/dtx.h::dtx_id`, `src/include/daos/dtx.h::dtx_memberships`, + `src/include/daos_srv/vos_types.h::dtx_entry`. + +## Threading + +- `dtx_begin`/`dtx_leader_begin` run in the RPC handler ULT on the target xstream. +- Forwarding to non-leaders is done via `dss_chore_register()` (helper xstreams when configured). +- Per-target background ULTs: `dtx_batched_commit` (created with `dss_ult_create_all()`), + `dtx_aggregate`, `dtx_cleanup` (created via `sched_create_ult()`). +- Resync runs per container from `src/container/srv_target.c::ds_dtx_resync`. + +## Invariants and gotchas + +- A reader hitting a prepared-but-undecided DTX gets `-DER_INPROGRESS`; the server first tries + `dtx_refresh()` to ask the leader, then the client retries. +- Client resends are detected with `src/dtx/dtx_common.c::dtx_handle_resend`; never re-execute an + update whose DTX already exists. +- The leader builds its DTX entry before forwarding/bulk transfer to avoid races with fast + non-leaders (see comment in `src/object/srv_obj.c::obj_tgt_update`). +- Committable DTXs that fail to enter the CoS cache are recovered by the next resync. +- Pool map changes can change the leader; `dtx_resync()` resolves orphaned DTXs. + +## Tests + +- Unit: `src/dtx/tests` (`dtx_tests`). +- Integration: `src/tests/suite/daos_dist_tx.c`, `src/tests/suite/daos_obj.c`. diff --git a/docs/kb/modules/engine.md b/docs/kb/modules/engine.md new file mode 100644 index 00000000000..3e148c7b350 --- /dev/null +++ b/docs/kb/modules/engine.md @@ -0,0 +1,126 @@ +--- +id: engine +kind: module +layer: server/runtime +depends_on: [cart, gurt, bio, vos, chk, common] +used_by: [bio, chk, container, dtx, mgmt, object, pipeline, pool, rdb, rebuild, rsvc, security, vos] +related: [xstream, ult, iv, drpc, control] +--- +# Engine runtime (`src/engine`) + +Overview: [src/engine/README.md](../../../src/engine/README.md). Public server API: +`src/include/daos_srv/daos_engine.h`; IV API: `src/include/daos_srv/iv.h`. Internal: +`src/engine/srv_internal.h`. + +## Purpose + +`daos_engine` is the data-plane process (one per rank). `src/engine` is its runtime, not a +storage module: it parses arguments, initialises gurt/CaRT/Argobots/SPDK, loads the server +modules (`libpool.so`, `libobj.so`, ...), creates the xstreams and per-xstream CaRT contexts, +dispatches incoming RPCs to module handlers through a scheduler, hosts the dRPC server that +`daos_server` talks to, and provides the ULT, collective and IV helpers every module uses. + +## File map + +| File | Role | +|---|---| +| `src/engine/init.c` | `main`, argument parsing, `server_init` / `server_fini`, init-state machine | +| `src/engine/module.c` | Module loading (`dss_module_load`), `dss_module_init_all`, `dss_module_setup_all`, handler registration | +| `src/engine/srv.c` | Xstream creation (`dss_srv_init`, `dss_start_one_xstream`), per-xstream progress, `dss_rpc_hdlr`, `dss_sleep` | +| `src/engine/ult.c` | ULT helpers: `dss_ult_create`, `dss_ult_create_all`, `dss_ult_execute`, collectives, chores, `dss_offload_exec` | +| `src/engine/sched.c` | Per-xstream scheduler: request queues, throttling, space pressure, watchdog | +| `src/engine/server_iv.c` | IV framework on top of CaRT IV (`ds_iv_*`) | +| `src/engine/drpc_listener.c`, `src/engine/drpc_progress.c`, `src/engine/drpc_handler.c` | dRPC server: socket listener, poll loop, dispatch by dRPC module ID | +| `src/engine/drpc_client.c` | Engine -> `daos_server` dRPC calls (`dss_drpc_call`, `drpc_notify_ready`) | +| `src/engine/drpc_ras.c` | RAS events to the control plane (`ds_notify_ras_event`) | +| `src/engine/drpc_chk.c` | Checker upcalls to the control plane | +| `src/engine/srv_cli.c`, `src/engine/rpc.c`, `src/engine/util.c`, `src/engine/profile.c`, `src/engine/srv_metrics.c` | Server-side client helpers, RPC utils, misc, profiling, metrics | + +## Startup and shutdown + +1. `src/engine/init.c::main` -> `src/engine/init.c::server_init`: + TLS keys, logging, topology (`dss_topo_init()`), telemetry, `drpc_init()`, Argobots, + `crt_init_opt()`, placement init, `modules_load()` (dlopen each module), + `bio_nvme_init()`, `src/engine/module.c::dss_module_init_all` (each `sm_init`, register CaRT + protocols and dRPC handlers), `src/engine/srv.c::dss_srv_init` (xstreams + contexts). +2. `src/engine/drpc_client.c::drpc_notify_ready` tells `daos_server` the engine is up + (`DRPC_METHOD_SRV_NOTIFY_READY`), then `server_init` waits for `DSS_INIT_STATE_SET_UP`. +3. After the rank joins the system, `daos_server` sends `DRPC_METHOD_MGMT_SET_RANK` and + `DRPC_METHOD_MGMT_SET_UP`; `src/mgmt/srv_drpc.c::ds_mgmt_drpc_set_up` runs + `dss_module_setup_all()` (each module's `sm_setup`, e.g. start pool services) and sets the + init state. See [control.md](control.md) for the Go side. +4. Shutdown: `src/engine/init.c::server_fini` reverses the order (module cleanup, xstreams, + unload, CaRT, Argobots). + +## Xstreams and threading + +`src/include/daos_srv/daos_engine.h::dss_xs_type` selects the target of `dss_ult_create()`: + +| Xstream | Count | Runs | +|---|---|---| +| System xs 0 (`DSS_XS_SYS`) | 1 | Pool/container services, rdb/Raft, rebuild leader, dRPC handlers, mgmt | +| SWIM (`DSS_XS_SWIM`), dRPC listener (`DSS_XS_DRPC`) | 1 each | Failure detection, dRPC socket | +| Target main (`DSS_XS_VOS`) | `dss_tgt_nr` | Everything that touches the target's VOS: I/O handlers, aggregation, DTX, rebuild scan/migrate | +| Helper / offload (`DSS_XS_IOFW`, `DSS_XS_OFFLOAD`) | `dss_tgt_offload_xs_nr` | I/O forwarding (chores), EC/checksum/compression offload | + +- Xstream id math: `src/engine/srv_internal.h::DSS_MAIN_XS_ID`, + `src/engine/srv_internal.h::DSS_MAIN_XS_ID_WITH_HELPER_POOL` (helpers either per target or a + shared pool when there are fewer helpers than targets). +- Current context: `dss_get_module_info()` -> `dmi_xs_id`, `dmi_tgt_id`, `dmi_ctx`. +- Per-module per-xstream TLS: `sm_key` in `struct dss_module` + `dss_module_key_get()`. + +## Entry points + +- ULTs: `src/engine/ult.c::dss_ult_create` (xs type + target), `dss_ult_create_all()` (one per + target), `dss_ult_execute()` (run on another xstream and wait), `dss_ult_periodic()`. +- Collectives across targets: `src/engine/ult.c::dss_thread_collective`, + `src/engine/ult.c::dss_task_collective` and `*_reduce` variants; pool-aware wrapper + `ds_pool_thread_collective()`. +- Offload: `src/engine/ult.c::dss_chore_register` (queued on helper xstream), + `src/engine/ult.c::dss_offload_exec`. +- Scheduler: `src/engine/sched.c::sched_create_ult` (tracked background ULTs, e.g. aggregation, + GC), `sched_req_yield()`, `sched_req_sleep()`, `sched_req_space_check()`; request types + `SCHED_REQ_UPDATE`, `SCHED_REQ_FETCH`, `SCHED_REQ_GC`, `SCHED_REQ_SCRUB`, `SCHED_REQ_MIGRATE`. +- RPC dispatch: `src/engine/srv.c::dss_rpc_hdlr` (registered with CaRT for every context) asks + the module for request attributes and enqueues the handler with `sched_req_enqueue()`. + Handlers/protocols come from `struct dss_module` (`sm_handlers`, `sm_proto_fmt`), list in + [generated/modules.md](../generated/modules.md). +- dRPC server: `src/engine/drpc_handler.c::drpc_hdlr_process_msg` routes by dRPC module ID to + `sm_drpc_handlers`; method table in [generated/drpc.md](../generated/drpc.md). +- IV: `src/engine/server_iv.c::ds_iv_class_register`, `src/engine/server_iv.c::ds_iv_ns_create`, + `src/engine/server_iv.c::ds_iv_fetch`, `src/engine/server_iv.c::ds_iv_update`. + +## Key structures + +- `src/include/daos_srv/daos_engine.h::dss_module`: module descriptor (`sm_init`, `sm_fini`, + `sm_setup`, `sm_cleanup`, `sm_key`, `sm_handlers`, `sm_drpc_handlers`, `sm_metrics`). +- `src/include/daos_srv/daos_engine.h::dss_module_info`: per-xstream context. +- `src/engine/srv_internal.h::dss_xstream`: xstream state (ABT pool, scheduler, chore queue). +- `src/include/daos_srv/daos_engine.h::dss_chore`: offloaded unit of work. + +## Invariants and gotchas + +- Never block an xstream: no `sleep()`, blocking syscalls, pthread mutex/cond waits. Use + `dss_sleep()`, ABT primitives, or offload. +- Never yield inside a `umem`/VOS transaction. +- Deep call chains (VOS iteration, migration) need a big stack: pass `DSS_DEEP_STACK_SZ`. +- Work on a target's VOS must run on that target's main xstream; pool/rdb services run on xs 0. +- Modules are shared libraries loaded at runtime: `dss_module_load()` dlopens `lib.so` and + looks up the symbol `_module`. The default list is + `src/engine/init.c::MODULE_LIST` (`-m` overrides; aliases such as `vos` -> `vos_srv` are + mapped in `src/engine/init.c::modules_load`). Check mode (`-C`) uses `MODS_LIST_CHK`; + `pipeline` is appended only when `DAOS_PIPELINE` is set. + +## Tests + +- `src/engine/tests` (dRPC listener/handler/progress, Argobots stack/perf). +- Engine behaviour is mostly covered by module tests and `src/tests/ftest/server`. + +## Where to look for... + +| Question | Look at | +|---|---| +| Why an RPC runs on a given xstream | `src/engine/srv.c::dss_rpc_hdlr`, module `dms_get_req_attr` callbacks | +| Engine command-line options | `src/engine/init.c::parse` | +| How IO is throttled / space pressure | `src/engine/sched.c` | +| How the engine reports to `daos_server` | `src/engine/drpc_client.c`, `src/engine/drpc_ras.c` | diff --git a/docs/kb/modules/object.md b/docs/kb/modules/object.md new file mode 100644 index 00000000000..20d68cb45d1 --- /dev/null +++ b/docs/kb/modules/object.md @@ -0,0 +1,112 @@ +--- +id: object +kind: module +layer: client+server/io +depends_on: [placement, pool, container, dtx, vos, bio, rebuild, security, engine, cart] +used_by: [client, rebuild, pipeline, container] +related: [dtx, vos, placement, ec] +--- +# Object layer (`src/object`) + +Overview: [src/object/README.md](../../../src/object/README.md). Client-side header: +`src/include/daos/object.h`; server-side: `src/include/daos_srv/object.h`; RPC wire formats and +opcode list: `src/object/obj_rpc.h` (see [generated/rpcs.md](../generated/rpcs.md#object)). + +## Purpose + +Implements the object API (`daos_obj_*`) on both sides. The client computes the object layout +(placement), splits I/O per redundancy group / shard, handles EC encoding, checksums and retries, +and sends CaRT RPCs to targets. The server validates the request (handles, pool map version, +capabilities), wraps the modification in a DTX, forwards to the other shards, and executes against +VOS/bio. Also hosts EC aggregation, migration (used by rebuild), enumeration, collective punch/query +and distributed-transaction (CPD) RPCs. + +## File map + +Client (`libdaos`): + +| File | Role | +|---|---| +| `src/object/cli_obj.c` | Object open/close, layout, task entry points (`dc_obj_update_task`, `dc_obj_fetch_task`, punch, list, query), fan-out and retry | +| `src/object/cli_shard.c` | Per-shard RPC construction and completion (`dc_obj_shard_rw`) | +| `src/object/cli_ec.c`, `src/object/obj_ec.h` | EC request reassembly and parity encoding | +| `src/object/cli_csum.c` | Client checksums | +| `src/object/cli_coll.c` | Collective (object-wide) operations | +| `src/object/obj_tx.c` | Client distributed transactions (`daos_tx_*`), submitted as CPD RPCs | +| `src/object/cli_mod.c` | Client module init | + +Shared: + +| File | Role | +|---|---| +| `src/object/obj_class.c`, `src/object/obj_class_def.c` | Object classes (RP_*, EC_*, S*) | +| `src/object/obj_layout.c` | Layout helpers | +| `src/object/obj_rpc.c`, `src/object/obj_rpc.h` | RPC formats, opcode list `OBJ_PROTO_CLI_RPC_LIST` | +| `src/object/obj_enum.c`, `src/object/obj_utils.c`, `src/object/obj_verify.c` | Enumeration unpack, misc, replica verify | + +Server (`daos_engine`, module `obj`): + +| File | Role | +|---|---| +| `src/object/srv_mod.c` | Module registration (`obj_module`) and handler table | +| `src/object/srv_obj.c` | RPC handlers: `ds_obj_rw_handler`, `ds_obj_tgt_update_handler`, `ds_obj_enum_handler`, `ds_obj_punch_handler`, `ds_obj_cpd_handler`, `ds_obj_sync_handler`, ...; local I/O `obj_local_rw` | +| `src/object/srv_obj_remote.c` | Leader -> non-leader forwarding (`ds_obj_remote_update`, `ds_obj_remote_punch`) | +| `src/object/srv_ec.c`, `src/object/srv_ec_aggregate.c` | Server EC helpers and EC aggregation | +| `src/object/srv_obj_migrate.c` | Migration engine used by rebuild (`ds_obj_migrate_handler`) | +| `src/object/srv_enum.c`, `src/object/srv_csum.c`, `src/object/srv_coll.c`, `src/object/srv_io_map.c` | Enumeration, checksums, collective ops, I/O maps | +| `src/object/srv_cli.c` | Server acting as client of other targets (e.g. migration, EC) | + +## Entry points + +- Client API wrappers in `src/client/api/object.c` (`daos_obj_open()`, `daos_obj_update()`, + `daos_obj_fetch()`) create a task and schedule `dc_obj_*_task()` from `src/object/cli_obj.c`. +- Placement: `src/placement/pl_map.c::pl_obj_place` produces `daos_obj_layout`/shard targets. +- Server handlers: full list in [generated/rpcs.md](../generated/rpcs.md#object). + +## Key structures + +- Client: `src/object/obj_internal.h::dc_object`, `src/object/obj_internal.h::obj_auxi_args` + (per-task state, retry flags), `src/object/obj_ec.h::obj_reasb_req` (EC reassembly). +- Wire: `struct obj_rw_in` / `obj_rw_out`, generated by `CRT_RPC_DECLARE(obj_rw, ...)` from the + field list `src/object/obj_rpc.h::DAOS_ISEQ_OBJ_RW` (`orw_map_ver`, `orw_dti`, `orw_bulks`, ...). + All RPC structs follow this `CRT_RPC_DECLARE` pattern, so grep for the `DAOS_ISEQ_*` macro. +- Server: `src/object/obj_internal.h::obj_io_context` (pool/container child, map version, object class). +- Task args: `src/include/daos_task.h::daos_obj_rw_t`. + +## Threading + +- Client: everything is an asynchronous TSE task (`src/common/tse.c`); completion callbacks + (`src/object/cli_obj.c::obj_comp_cb`) decide retry vs. finish. +- Server: I/O handlers run as ULTs on the target's main (`DSS_XS_VOS`) xstream; leader forwarding + is offloaded with `dss_chore_register()` (via `dtx_leader_exec_ops()`); EC aggregation and + migration run as background ULTs. + +## Invariants and gotchas + +- Every I/O RPC carries the client's pool map version. The server (`obj_ioc_begin()`) rejects + stale requests with `-DER_STALE`; the client refreshes the pool map and retries + (`src/object/cli_obj.c::obj_retry_cb`). Don't swallow `-DER_STALE`. +- `-DER_INPROGRESS` / `-DER_TX_RESTART` / `-DER_AGAIN` are retryable conditions handled by the + client; see [modules/dtx.md](dtx.md). +- The dkey hash selects the redundancy group; all akeys under a dkey are co-located. +- EC: parity shards for a stripe live on dedicated shards; parity extents are marked with + `src/include/daos/object.h::DAOS_EC_PARITY_BIT`. Partial-stripe writes are replicated and later + folded into parity by EC aggregation. +- Protocol change -> bump `DAOS_OBJ_VERSION` and keep older formats (note the `ver == 9 ?` cases + in `OBJ_PROTO_CLI_RPC_LIST`). + +## Tests + +- `src/object/tests`, `src/tests/suite/daos_obj.c`, `src/tests/suite/daos_obj_ec.c`, + `src/tests/suite/daos_checksum.c`, `src/tests/suite/daos_degrade_ec.c`, + `src/tests/suite/daos_aggregate_ec.c`. + +## Where to look for... + +| Question | Look at | +|---|---| +| Which targets get an update | `src/object/cli_obj.c::obj_req_fanout`, `src/placement/pl_map.c::pl_obj_place` | +| Client retry decisions | `src/object/cli_obj.c::obj_retry_cb`, `src/object/cli_obj.c::obj_comp_cb` | +| Server I/O body | `src/object/srv_obj.c::obj_local_rw_internal` | +| How rebuild moves data | `src/object/srv_obj_migrate.c` | +| End-to-end write | [flows/write-path.md](../flows/write-path.md) | diff --git a/docs/kb/modules/pool.md b/docs/kb/modules/pool.md new file mode 100644 index 00000000000..f629ce3557a --- /dev/null +++ b/docs/kb/modules/pool.md @@ -0,0 +1,90 @@ +--- +id: pool +kind: module +layer: client+server/metadata +depends_on: [rsvc, rdb, rebuild, container, vos, bio, mgmt, object, security, engine, cart] +used_by: [object, container, dtx, rebuild, mgmt, chk, security] +related: [pool_map, iv, rsvc, rdb] +--- +# Pool (`src/pool`) + +Overview: [src/pool/README.md](../../../src/pool/README.md). Client header: +`src/include/daos/pool.h`; server header: `src/include/daos_srv/pool.h`; pool map: +`src/include/daos/pool_map.h` (implementation `src/common/pool_map.c`). Opcodes: +[generated/rpcs.md](../generated/rpcs.md#pool). + +## Purpose + +Three things live here: + +1. **Pool service** (`srv_pool.c`): replicated metadata service (rsvc over rdb/Raft) on a few + replica ranks. Stores pool map, properties, ACL, attributes, handles; serves connect / + disconnect / query / set-prop / exclude / drain / reintegrate / extend; schedules rebuild + when the map changes. The container service is hosted in the same rdb. +2. **Per-target pool state** (`srv_target.c`): `ds_pool` per engine and `ds_pool_child` per + target xstream (open VOS pool handle, local map version, GC/scrub/flush ULTs). +3. **Client** (`cli.c`): `dc_pool_*` connect/query/exclude, pool map cache, leader discovery. + +## File map + +| File | Role | +|---|---| +| `src/pool/cli.c` | Client API (`dc_pool_connect`, `dc_pool_disconnect`, `dc_pool_exclude`, ...), map cache `dc_pool_map_update` | +| `src/pool/srv_pool.c` | Pool service: create/load/start (`ds_pool_svc_dist_create`, `ds_pool_svc_start`), all client-facing handlers (`ds_pool_connect_handler`, `ds_pool_query_handler`, `ds_pool_update_handler`, ...), map updates `pool_svc_update_map` | +| `src/pool/srv_target.c` | `ds_pool` / `ds_pool_child` lookup and lifecycle, server-side pool handles | +| `src/pool/srv_iv.c` | IV keys for pool map, props, connection handles (`ds_pool_iv_map_update`, ...) | +| `src/pool/srv_pool_map.c` | Target state transitions in the map | +| `src/pool/srv_layout.c`, `src/pool/srv_layout.h` | rdb key layout of the pool service, global/layout versions | +| `src/pool/srv_pool_chkpt.c`, `src/pool/srv_pool_scrub_ult.c` | MD-on-SSD checkpointing, checksum scrubbing ULTs | +| `src/pool/srv_pool_check.c` | Checker (`dmg check`) integration | +| `src/pool/srv.c` | Module registration (`pool_module`) | +| `src/pool/rpc.h`, `src/pool/rpc.c` | RPC formats and opcode lists | + +## Key structures + +- `src/include/daos_srv/pool.h::ds_pool`: per-engine pool (map, map version, IV namespace). +- `src/include/daos_srv/pool.h::ds_pool_child`: per-target pool (VOS pool handle, map version, + background ULT requests). +- `src/include/daos_srv/pool.h::ds_pool_hdl`: server-side connection handle. +- `src/include/daos/pool.h::dc_pool`: client pool (map cache, rsvc client for leader hint). +- `src/include/daos/pool_map.h::pool_map`, `pool_domain`, `pool_target`, `pool_component`. +- `src/pool/srv_internal.h`, `src/pool/srv_pool.c` (`struct pool_svc`): service state. + +## Pool map and target states + +`PO_COMP_ST_NEW`, `UP` (being reintegrated/extended), `UPIN` (healthy), `DOWN` (excluded, rebuild +pending/running), `DOWNOUT` (excluded and rebuilt), `DRAIN` (being drained). Every change bumps +the map version; the new map is distributed with IV (`ds_pool_iv_map_update()`) and targets +refresh (`ds_pool_child_map_refresh_async()`). Rebuild is scheduled from the pool service +(`ds_rebuild_schedule()`). + +## Threading + +- Pool service (leader and replicas), rdb and IV handlers run on the system xstream (xs 0). +- `ds_pool_child` work (VOS, GC, scrubbing, checkpoint) runs on each target xstream. +- Clients find the leader via `rsvc_client` hints; non-leaders reply `-DER_NOTLEADER`. + +## Invariants and gotchas + +- Map versions are monotonic. RPCs carry the sender's map version; stale senders get + `-DER_STALE` and must refresh. +- All pool-service metadata changes go through an rdb transaction (`rdb_tx_begin()` -> + `rdb_tx_commit()`) on the current leader term; a term change aborts in-flight work. +- `DAOS_POOL_GLOBAL_VERSION` / layout versions gate upgrades (`dmg pool upgrade`); changing the + rdb layout requires a version bump in `src/pool/srv_layout.h`. +- Do not perform per-target VOS work on xs 0; use `ds_pool_task_collective()` / + `dss_thread_collective()` style helpers to fan out to targets. + +## Tests + +- `src/tests/suite/daos_pool.c`, `src/tests/suite/daos_mgmt.c`, `src/tests/suite/daos_rebuild.c`, + ftests under `src/tests/ftest/pool`. + +## Where to look for... + +| Question | Look at | +|---|---| +| Pool creation end-to-end | [flows/pool-create.md](../flows/pool-create.md) | +| What happens on `dmg pool exclude` | `src/pool/srv_pool.c::ds_pool_update_handler` -> `pool_svc_update_map` | +| How clients find the leader | `src/pool/cli.c::dc_pool_choose_svc_rank`, `src/common/rsvc.c` | +| Per-target pool startup | `src/pool/srv_target.c::ds_pool_child_start` | diff --git a/docs/kb/modules/rebuild.md b/docs/kb/modules/rebuild.md new file mode 100644 index 00000000000..26b0ff7e50f --- /dev/null +++ b/docs/kb/modules/rebuild.md @@ -0,0 +1,148 @@ +--- +id: rebuild +kind: module +layer: server/resilience +depends_on: [pool, object, dtx, container, placement, vos, engine, mgmt, security] +used_by: [pool, object] +related: [pool, object, dtx, placement, iv] +--- +# Rebuild (`src/rebuild` + `src/object/srv_obj_migrate.c`) + +Design: [src/rebuild/README.md](../../../src/rebuild/README.md), +[src/rebuild/rebuild_design.md](../../../src/rebuild/rebuild_design.md). Server API: +`src/include/daos_srv/rebuild.h`. + +## Purpose + +Restores redundancy and moves data whenever the pool map changes: a target is excluded (failure), +drained, reintegrated, or added (extend), and on layout upgrade. Rebuild is driven by the pool +service leader and executed by every engine of the pool in two phases: **scan** (each target +walks its local objects and works out which shards must exist elsewhere under the new map) and +**pull/migrate** (destination targets fetch the data from surviving replicas, or reconstruct it +from EC, and write it into their VOS). + +## Operations (`src/include/daos_srv/rebuild.h::daos_rebuild_opc_t`) + +| Op | Used for | +|---|---| +| `RB_OP_REBUILD` | All pool-map driven data movement: exclude, drain, reintegrate, extend. The target states in the map (DOWN, DRAIN, UP, NEW) decide what each scan looks for | +| `RB_OP_RECLAIM` | Scheduled after every successful `RB_OP_REBUILD`/`RB_OP_UPGRADE`: delete data that no longer belongs on a target under the new layout | +| `RB_OP_FAIL_RECLAIM` | After a failed rebuild: discard partially rebuilt data above the reclaim epoch | +| `RB_OP_UPGRADE` | Object layout upgrade after a pool upgrade | + +## File map + +| File | Role | +|---|---| +| `src/rebuild/srv.c` | Module (`rebuild_module`), task queue (`ds_rebuild_schedule`, `rebuild_ults`), leader (`rebuild_task_ult`, `rebuild_leader_start`, `rebuild_scan_broadcast`, `rebuild_leader_status_check`), target status ULT, admin start/stop, query | +| `src/rebuild/scan.c` | `REBUILD_OBJECTS_SCAN` handler (`rebuild_tgt_scan_handler`), per-target scanner, sending object lists to destinations | +| `src/rebuild/rebuild_iv.c` | Rebuild IV: targets report progress to the leader | +| `src/rebuild/rpc.c`, `src/rebuild/rpc.h` | RPC formats (`REBUILD_PROTO_SRV_RPC_LIST`) | +| `src/rebuild/rebuild_internal.h` | Trackers, task, global state | +| `src/rebuild/ras.c` | RAS events for rebuild | +| `src/object/srv_obj_migrate.c` | Migration engine: receive object lists, fetch + write data | + +## Flow + +End-to-end, including failure detection and the pool map update that triggers rebuild: +[flows/rebuild.md](../flows/rebuild.md). + +```mermaid +sequenceDiagram + participant PS as Pool service leader (xs 0) + participant T as Every pool engine (xs 0 + target xstreams) + participant D as Destination target + PS->>PS: pool_svc_update_map -> ds_rebuild_schedule (task queue) + PS->>PS: rebuild_ults -> rebuild_task_ult -> rebuild_leader_start + PS->>T: REBUILD_OBJECTS_SCAN (collective) + T->>T: rebuild_tgt_scan_handler -> rebuild_scan_leader -> rebuild_scanner per target + T->>D: DAOS_OBJ_RPC_MIGRATE (object lists) + D->>D: ds_obj_migrate_handler -> ds_migrate_object -> migrate_one_ult -> dsc_obj_fetch + vos_obj_update + T-->>PS: rebuild IV updates (scan_done / pull_done, counters) + PS->>PS: done -> ds_pool_tgt_finish_rebuild (map: UP->UPIN, DOWN->DOWNOUT) +``` + +1. Trigger: `src/pool/srv_pool.c::pool_svc_update_map` (exclude/drain/reint/extend) calls + `src/rebuild/srv.c::ds_rebuild_schedule` with `RB_OP_REBUILD` and a short delay to merge + multi-rank events; `DAOS_SELF_HEAL_DELAY_REBUILD` defers it, and reintegration mode + `DAOS_REINT_MODE_NO_DATA_SYNC` skips data movement. +2. Queue: tasks for the same pool/op are merged; `src/rebuild/srv.c::rebuild_ults` (one ULT, on the + pool-service xstream) starts `src/rebuild/srv.c::rebuild_task_ult` per task. +3. Leader: `src/rebuild/srv.c::rebuild_leader_start` creates the global tracker and + `src/rebuild/srv.c::rebuild_scan_broadcast` sends collective `REBUILD_OBJECTS_SCAN`. +4. Every engine: `src/rebuild/scan.c::rebuild_tgt_scan_handler` -> `src/rebuild/srv.c::rebuild_tgt_prepare` + (target tracker, IV) and starts `src/rebuild/scan.c::rebuild_scan_leader` and + `src/rebuild/srv.c::rebuild_tgt_status_check_ult`. +5. `rebuild_scan_leader` waits for the global DTX resync version, calls `dtx_cleanup_orphan()`, + then `ds_pool_thread_collective()` runs `src/rebuild/scan.c::rebuild_scanner` on each target + xstream: `vos_iterate()` over containers (`rebuild_container_scan_cb`) and objects + (`src/rebuild/scan.c::rebuild_obj_scan_cb`), `src/rebuild/scan.c::find_rebuild_shards` + (placement: `pl_obj_find_rebuild()`) and `rebuild_object_insert()`. +6. `src/rebuild/scan.c::rebuild_objects_send_ult` -> `src/object/srv_obj_migrate.c::ds_object_migrate_send` + sends `DAOS_OBJ_RPC_MIGRATE` to the destination. +7. Destination: `src/object/srv_obj_migrate.c::ds_obj_migrate_handler` -> + `src/object/srv_obj_migrate.c::ds_migrate_object` (per-target ULTs, `migrate_obj_iter_cb`) -> + `src/object/srv_obj_migrate.c::migrate_one_ult` -> `src/object/srv_obj_migrate.c::migrate_dkey`, + which reads from surviving shards with `dsc_obj_fetch()` (or rebuilds EC data/parity: + `migrate_fetch_update_parity()`) and writes locally (`vos_obj_update()`, + `src/object/srv_obj_migrate.c::migrate_one_insert_recx`). +8. Progress: target status ULT pushes counters and `scan_done`/`pull_done` through + `src/rebuild/rebuild_iv.c::rebuild_iv_update`; on the leader the IV callback calls + `src/rebuild/srv.c::rebuild_global_status_update`. +9. Completion: `rebuild_task_ult` calls `src/pool/srv_pool.c::ds_pool_tgt_finish_rebuild` to update + target states; `src/rebuild/srv.c::rebuild_task_complete_schedule` then queues `RB_OP_RECLAIM` + on success, or on failure uses `src/rebuild/srv.c::retry_rebuild_task` to decide on a retry + (possibly after `RB_OP_FAIL_RECLAIM`); `ds_pool_tgt_revert_rebuild()` reverts map changes. + +## Key structures (`src/rebuild/rebuild_internal.h`) + +- `src/rebuild/rebuild_internal.h::rebuild_task`: queued task (pool, map version, op, targets, + reclaim epoch, schedule time). +- `src/rebuild/rebuild_internal.h::rebuild_global_pool_tracker`: leader-side per-pool state + (per-rank status, DTX resync version, stable epoch, abort flags). +- `src/rebuild/rebuild_internal.h::rebuild_tgt_pool_tracker`: per-engine per-pool state (object + trees to send, counters, leader term). +- `src/object/srv_internal.h::migrate_pool_tls`: per-target migration state. +- `src/object/srv_obj_migrate.c::migrate_one`: one dkey/object migration unit. + +## Threading + +- Pool-service xstream (xs 0) of the leader: `rebuild_ults`, `rebuild_task_ult`, status checks. +- Every engine: the scan RPC handler starts `rebuild_scan_leader` and + `rebuild_tgt_status_check_ult` on its own xstream; tracker teardown is forced onto `DSS_XS_SYS`. +- Target main xstreams: `rebuild_scanner`, object send ULTs, migration ULTs. +- `REBUILD_OBJECTS_SCAN` and `DAOS_OBJ_RPC_MIGRATE` are classified as `SCHED_REQ_MIGRATE` + (`src/rebuild/srv.c::rebuild_get_req_attr`, `src/object/srv_mod.c`), so the scheduler can + throttle rebuild against client I/O. + +## Invariants and gotchas + +- One running rebuild per pool; newer map changes queue up (and may be merged). +- Rebuild only starts scanning after DTX resync, so in-doubt transactions are resolved first. +- Data is fetched at stable epochs; aggregation and reclaim use the reclaim epoch recorded in + the task to avoid discarding data still needed. +- Rebuild is driven by pool map state; to understand "why didn't X move", check target states + in the pool map and `src/placement/pl_map.c::pl_obj_find_rebuild` (the single placement entry + point for all rebuild ops; the jump/ring map implementations interpret target states). +- Admin control: `dmg pool rebuild stop|start` -> dRPC `DRPC_METHOD_MGMT_POOL_REBUILD_STOP|START` + -> `ds_pool_rebuild_stop_handler()` / `ds_pool_rebuild_start_handler()` -> + `src/rebuild/srv.c::ds_rebuild_admin_stop` / `src/rebuild/srv.c::ds_rebuild_admin_start`. +- Self-heal policy: pool property `DAOS_PROP_PO_SELF_HEAL` (`DAOS_SELF_HEAL_AUTO_EXCLUDE`, + `DAOS_SELF_HEAL_AUTO_REBUILD`, `DAOS_SELF_HEAL_DELAY_REBUILD` in `src/include/daos_prop.h`). + +## Tests + +- `src/tests/suite/daos_rebuild.c`, `src/tests/suite/daos_rebuild_simple.c`, + `src/tests/suite/daos_rebuild_ec.c`, `src/tests/suite/daos_drain_simple.c`, + `src/tests/suite/daos_extend_simple.c`, `src/tests/suite/daos_inc_reint.c`. +- Functional: `src/tests/ftest/rebuild`. + +## Where to look for... + +| Question | Look at | +|---|---| +| Rebuild didn't start | `src/pool/srv_pool.c::pool_svc_update_map` (self-heal, reint mode, delay), `src/rebuild/srv.c::ds_rebuild_schedule` | +| Rebuild hangs | `src/rebuild/srv.c::rebuild_leader_status_check`, `src/rebuild/srv.c::rebuild_tgt_status_check_ult` | +| Which shards are rebuilt | `src/rebuild/scan.c::find_rebuild_shards`, `src/placement/pl_map.c::pl_obj_find_rebuild` | +| Data copy / EC reconstruction | `src/object/srv_obj_migrate.c::migrate_dkey` | +| Rebuild status for `dmg pool query` | `src/rebuild/srv.c::ds_rebuild_query` | diff --git a/docs/kb/modules/vos.md b/docs/kb/modules/vos.md new file mode 100644 index 00000000000..1749e03f8fa --- /dev/null +++ b/docs/kb/modules/vos.md @@ -0,0 +1,108 @@ +--- +id: vos +kind: module +layer: server/storage +depends_on: [bio, vea, common, dtx] +used_by: [object, dtx, container, rebuild, rdb, mgmt, chk, pipeline] +related: [epoch, ilog, aggregation, dtx] +--- +# VOS: Versioning Object Store (`src/vos`) + +Design doc: [src/vos/README.md](../../../src/vos/README.md). Public API: +`src/include/daos_srv/vos.h`, types: `src/include/daos_srv/vos_types.h`. + +## Purpose + +Local, per-target persistent store for one pool shard. Stores every object/dkey/akey/value +version by epoch (MVCC), answers "what was visible at epoch E", records punches, persists local +DTX state, and reclaims old versions by aggregation. Metadata lives in SCM (or DRAM + WAL in +MD-on-SSD mode) through `umem`; bulk array data goes to NVMe via `bio` with space from `vea`. +VOS has no network code; callers (object, dtx, rebuild, rdb) run on the owning target xstream. + +## File map + +| File | Role | +|---|---| +| `src/vos/vos_pool.c` | Pool create/open/close/query, per-pool space and GC setup | +| `src/vos/vos_container.c` | Container create/open/close/destroy, per-container DTX tables | +| `src/vos/vos_io.c` | Fetch/update begin/end, SCM/NVMe space reservation and publish | +| `src/vos/vos_obj.c`, `src/vos/vos_obj_cache.c`, `src/vos/vos_obj_index.c` | Object punch/hold, DRAM object cache, object index (OI) tree | +| `src/vos/vos_tree.c` | dkey/akey btree classes and single-value tree | +| `src/vos/evtree.c`, `src/vos/evt_iter.c` | Extent-versioned tree for array values | +| `src/vos/ilog.c`, `src/vos/vos_ilog.c` | Incarnation log (create/punch history per entity) | +| `src/vos/vos_ts.c` | Timestamp cache for read/write conflict detection | +| `src/vos/vos_dtx.c`, `src/vos/vos_dtx_iter.c` | Local DTX table: attach, check, commit, abort, iteration | +| `src/vos/vos_aggregate.c` | Aggregation and discard | +| `src/vos/vos_iterator.c`, `src/vos/vos_query.c` | Generic nested iterator; key queries | +| `src/vos/vos_gc.c` | Background garbage collection of freed trees/extents | +| `src/vos/vos_layout.h` | Persistent (on-media) structure definitions | +| `src/vos/vos_internal.h`, `src/vos/vos_tls.h` | Internal types, per-xstream TLS | +| `src/vos/vos_common.c` | Module registration (`vos_srv_module`), init, standalone mode | + +## Entry points (all in `src/include/daos_srv/vos.h`) + +- Pool/container: `vos_pool_create()`, `vos_pool_open()`, `vos_cont_create()`, `vos_cont_open()`. +- Update: `src/vos/vos_io.c::vos_update_begin` -> (caller fills buffers via `bio_iod_prep()` / + bulk) -> `src/vos/vos_io.c::vos_update_end`. One-shot: `vos_obj_update()`. +- Fetch: `src/vos/vos_io.c::vos_fetch_begin` -> `vos_fetch_end()`; one-shot `vos_obj_fetch()`. +- Punch: `src/vos/vos_obj.c::vos_obj_punch`. +- Iterate: `src/vos/vos_iterator.c::vos_iterate` (or `vos_iter_prepare()`/`vos_iter_probe()`/ + `vos_iter_next()`/`vos_iter_fetch()`). +- Aggregate/discard: `src/vos/vos_aggregate.c::vos_aggregate`, `src/vos/vos_aggregate.c::vos_discard`. +- DTX: `src/vos/vos_dtx.c::vos_dtx_attach`, `src/vos/vos_dtx.c::vos_dtx_check`, + `src/vos/vos_dtx.c::vos_dtx_commit`, `src/vos/vos_dtx.c::vos_dtx_abort`. + +## Key structures + +- Persistent: `src/vos/vos_layout.h::vos_pool_df`, `src/vos/vos_layout.h::vos_cont_df`, + `src/vos/vos_layout.h::vos_obj_df`, `src/vos/vos_layout.h::vos_dtx_act_ent_df`, + `src/vos/vos_layout.h::vos_dtx_cmt_ent_df`, `src/vos/vos_layout.h::vos_dtx_blob_df`. +- DRAM: `src/vos/vos_internal.h::vos_pool`, `src/vos/vos_internal.h::vos_container`, + `src/vos/vos_io.c::vos_io_context`. + +## Data layout + +``` +vos pool -> container table -> container -> object index (OI) -> object + -> dkey btree -> akey btree -> single-value btree (SV) | evtree (EV, arrays) +``` + +- Each object, dkey, and akey has an ilog recording create/punch epochs; visibility at an epoch + is decided by the ilog first, then by the value trees. +- SV tree is keyed by epoch; reads return the newest version <= read epoch. +- evtree stores (extent, epoch) rectangles; reads compute visible extents at the read epoch. +- Values are stored inline in SCM when small; larger array data goes to NVMe (`bio` addresses, + space from `vea_reserve()` via `src/vos/vos_io.c::vos_reserve_blocks`). + +## Threading + +- VOS state is per target and only touched from the owning target xstream; no locks across xstreams. +- Aggregation is driven by the container module's per-container ULT + (`src/container/srv_target.c::cont_start_agg` -> `src/container/srv_target.c::cont_vos_aggregate_cb`) + and yields periodically (`src/vos/vos_aggregate.c::vos_aggregate_yield`). + +## Invariants and gotchas + +- Everything is epoch-versioned; never overwrite in place. Punch is a versioned tombstone. +- Do not yield (ABT yield, blocking I/O, RPC wait) inside a `umem` transaction. +- Conflicts surface as `-DER_INPROGRESS` (hit an uncommitted DTX; caller should refresh/retry) + and `-DER_TX_RESTART` (MVCC conflict; client restarts the transaction with a new epoch). +- `vos_update_begin()` reserves space; it is only published in `vos_update_end()`; errors must still + call `*_end()` so reservations are cancelled. +- Changing anything in `src/vos/vos_layout.h` is an on-media format change: requires a layout + version bump and upgrade handling. + +## Tests + +- Unit: `src/vos/tests` (`vos_tests` binary: `vts_io.c`, `vts_dtx.c`, `vts_aggregate.c`, + `vts_ilog.c`, `vts_mvcc.c`, ...). Run via `utils/run_utest.py`. +- Perf: `src/tests/vos_perf.c`. + +## Where to look for... + +| Question | Look at | +|---|---| +| Why a fetch returned `-DER_INPROGRESS` | `src/vos/vos_dtx.c` (`vos_dtx_check_availability()`) | +| How array extents become visible | `src/vos/evtree.c` | +| Space accounting / ENOSPACE | `src/vos/vos_space.c` | +| Persistent format | `src/vos/vos_layout.h` | diff --git a/utils/kb/benchmark/questions.md b/utils/kb/benchmark/questions.md new file mode 100644 index 00000000000..b9d3b1931c3 --- /dev/null +++ b/utils/kb/benchmark/questions.md @@ -0,0 +1,342 @@ +# DAOS KB benchmark + +Questions for measuring whether the knowledge base (`docs/kb`, `.github/copilot-instructions.md`, +`.github/instructions/`) makes an agent faster and more accurate. Every expected answer was +checked against the code; anchors are validated by `utils/kb/kb.py lint`. + +## Protocol + +1. Create two worktrees from the same commit: + - **baseline**: delete `docs/kb`, `.github/copilot-instructions.md`, `.github/instructions`. + - **kb**: keep them. + + In **both**, delete `utils/kb/benchmark` so the agent can't read the answers. +2. Ask each question in a **fresh chat** (same model, agent mode), verbatim, with no extra context. +3. Record per question: score, tool calls, wall time, and any hallucination (a function, file, + opcode or behaviour that does not exist). +4. Compare the totals per category. The KB is useful if the kb run scores higher with fewer tool + calls; any hallucination in the kb run that traces back to a card is a KB bug: fix the card. + +### Scoring + +| Score | Meaning | +|---|---| +| 2 | All key facts present and correct | +| 1 | Main fact correct, some key facts missing | +| 0 | Wrong, or main fact missing | +| H | Flag (in addition to the score): stated something that does not exist | + +Categories: **L** lookup, **F** flow, **I** invariant/gotcha, **C** change-guidance, +**T** trap (premise is false or commonly confused), **G** gap probe (not covered by cards; +checks the generated tables and that the KB doesn't hurt). + +### Results template + +| ID | Run | Score | H | Tool calls | Time | Notes | +|---|---|---|---|---|---|---| +| Q01 | baseline | | | | | | +| Q01 | kb | | | | | | + +--- + +## Questions + +### Q01 (L) Server handlers for an object update + +**Ask:** Which server functions handle a client object update RPC, and which handle the copy sent +to the other replicas? + +**Key facts:** +- `DAOS_OBJ_RPC_UPDATE` -> `src/object/srv_obj.c::ds_obj_rw_handler` on the leader shard. +- Forwarded as `DAOS_OBJ_RPC_TGT_UPDATE` -> `src/object/srv_obj.c::ds_obj_tgt_update_handler`. + +**KB source:** generated/rpcs.md, flows/write-path.md + +### Q02 (L) dRPC pool create + +**Ask:** Which engine function handles the dRPC pool-create request, and which Go function sends it? + +**Key facts:** +- Method `DRPC_METHOD_MGMT_POOL_CREATE` (207), dispatched in `src/mgmt/srv.c`. +- Engine: `src/mgmt/srv_drpc.c::ds_mgmt_drpc_pool_create`. +- Go: `src/control/server/mgmt_pool.go::mgmtSvc.poolCreate` via `harness.CallDrpc` with + `daos.MethodPoolCreate`. + +**KB source:** generated/drpc.md, flows/pool-create.md + +### Q03 (F) Replicated write fan-out + +**Ask:** For an object with 3-way replication, how many update RPCs does the client send per +redundancy group, and how do the other replicas get the data? + +**Key facts:** +- One RPC to the DTX leader shard (server-side dispatch, `ort_srv_disp`, set in + `src/object/cli_obj.c`). +- Leader runs `src/dtx/dtx_common.c::dtx_leader_exec_ops` with + `src/object/srv_obj.c::obj_tgt_update`; remote shards via + `src/object/srv_obj_remote.c::ds_obj_remote_update`, local via `obj_local_rw()`. + +**KB source:** flows/write-path.md + +### Q04 (I) When is a DTX committed? + +**Ask:** When a client gets a successful reply to a replicated update, is the distributed +transaction already committed on all replicas? + +**Key facts:** +- Not necessarily. `src/dtx/dtx_common.c::dtx_leader_end` marks it committable and usually adds it + to the CoS cache (`dtx_cos_add()`); it is committed later in batches by + `src/dtx/dtx_common.c::dtx_batched_commit` or piggybacked on later RPCs. +- Synchronous commit only when `dth_sync` is set. + +**KB source:** modules/dtx.md + +### Q05 (I) -DER_INPROGRESS + +**Ask:** A fetch returns `-DER_INPROGRESS`. What does it mean and how is it handled? + +**Key facts:** +- The read hit a prepared but not yet committed/aborted DTX. +- The server first tries `dtx_refresh()` to ask the leader; otherwise the client retries. + +**KB source:** modules/dtx.md, modules/vos.md + +### Q06 (L) Xstream roles + +**Ask:** Which engine xstream runs the pool service and rdb, and which runs VOS I/O for a target? + +**Key facts:** +- System xstream 0 (`DSS_XS_SYS`): pool/container services, rdb, dRPC handlers. +- One main xstream per target (`DSS_XS_VOS`) for everything touching that target's VOS. +- Defined in `src/include/daos_srv/daos_engine.h::dss_xs_type`. + +**KB source:** modules/engine.md, architecture.md + +### Q07 (F) Engine startup handshake + +**Ask:** After `daos_engine` starts, how does it get its rank and when do modules' `sm_setup` +callbacks run? + +**Key facts:** +- Engine calls `src/engine/drpc_client.c::drpc_notify_ready`, then `server_init` waits for + `DSS_INIT_STATE_SET_UP`. +- `daos_server` (`src/control/server/mgmt_drpc.go::srvModule.handleNotifyReady`) joins the + system (`mgmtSvc.Join`), then `src/control/server/instance.go::EngineInstance.SetupRank` sends + `SET_RANK` and `SET_UP`. +- `src/mgmt/srv_drpc.c::ds_mgmt_drpc_set_up` calls `dss_module_setup_all()`. + +**KB source:** modules/engine.md, modules/control.md + +### Q08 (F) NVMe space during an update + +**Ask:** Where is NVMe space allocated for an array update, and when does the allocation become +permanent? + +**Key facts:** +- Reserved in `src/vos/vos_io.c::vos_update_begin` -> `src/vos/vos_io.c::vos_reserve_blocks` -> + `src/vea/vea_api.c::vea_reserve`. +- Published in `src/vos/vos_io.c::vos_update_end` inside the VOS transaction; an error passed to + `vos_update_end()` cancels the reservation. + +**KB source:** flows/write-path.md, modules/vos.md + +### Q09 (T) Rebuild op codes + +**Ask:** What does the `RB_OP_DRAIN` rebuild operation do differently from `RB_OP_REINT`? + +**Key facts:** +- Neither exists. `src/include/daos_srv/rebuild.h::daos_rebuild_opc_t` has `RB_OP_REBUILD`, + `RB_OP_RECLAIM`, `RB_OP_FAIL_RECLAIM`, `RB_OP_UPGRADE`. +- Drain, reintegrate, exclude and extend all use `RB_OP_REBUILD`; target states in the pool map + (`DRAIN`, `UP`, `DOWN`, `NEW`) drive the difference. + +**KB source:** modules/rebuild.md + +### Q10 (F) Engine death to exclusion + +**Ask:** When an engine dies, which code in the pool service turns that into excluding its +targets? + +**Key facts:** +- `src/pool/srv_pool.c::ds_pool_crt_event_cb` -> `queue_event()` -> + `src/pool/srv_pool.c::events_handler` -> `src/pool/srv_pool.c::handle_event` -> + `src/pool/srv_pool.c::pool_svc_exclude_ranks` -> `pool_svc_update_map()` with `MUS_SWIM`. +- Only if pool self-heal has `DAOS_SELF_HEAL_AUTO_EXCLUDE` (and the system self-heal allows it). + +**KB source:** flows/rebuild.md + +### Q11 (I) Rebuild not starting + +**Ask:** A target was excluded but no rebuild started. List the reasons this can happen. + +**Key facts (any 3 for score 2):** +- Environment `DAOS_REBUILD=no`. +- System self-heal property disables pool rebuild for automatic events. +- `src/include/daos_srv/pool.h::is_pool_rebuild_allowed` false: rebuild stopped by admin, or + automatic event without `AUTO_REBUILD`/`DELAY_REBUILD` in pool self-heal. +- `DAOS_SELF_HEAL_DELAY_REBUILD`: scheduled but deferred until an admin starts it. +- Reintegration mode `DAOS_REINT_MODE_NO_DATA_SYNC`. + +**KB source:** flows/rebuild.md + +### Q12 (F) After rebuild + +**Ask:** What happens to the pool map and the data once a rebuild after an exclude succeeds? + +**Key facts:** +- `src/pool/srv_pool.c::ds_pool_tgt_finish_rebuild`: `DOWN` -> `DOWNOUT`. +- `src/rebuild/srv.c::rebuild_task_complete_schedule` queues `RB_OP_RECLAIM` to remove data that no + longer belongs on each target. + +**KB source:** flows/rebuild.md, modules/rebuild.md + +### Q13 (F) Stale pool map on the client + +**Ask:** How does a client that holds an old pool map find out during I/O and recover? + +**Key facts:** +- Server `obj_ioc_begin()` compares the RPC's map version and returns `-DER_STALE`. +- Client completion/retry (`src/object/cli_obj.c::obj_retry_cb`) refreshes the pool map and + resends. + +**KB source:** modules/object.md, flows/write-path.md + +### Q14 (F) Initial pool metadata + +**Ask:** During pool create, which function writes the initial pool map and properties, and where +are they stored? + +**Key facts:** +- `src/pool/srv_pool.c::ds_pool_create_handler` on the new pool-service leader, inside an rdb + transaction: `src/pool/srv_pool.c::init_pool_metadata` and `ds_cont_init_metadata()`, then + `rdb_tx_commit()`. +- Stored in the pool service's rdb (Raft-replicated, backed by VOS). + +**KB source:** flows/pool-create.md + +### Q15 (T) Starting pool service replicas + +**Ask:** Which dRPC call does the engine use to start the pool service replicas on other ranks? + +**Key facts:** +- None: it is a CaRT collective RPC, not dRPC. `src/rsvc/srv.c::ds_rsvc_dist_start` broadcasts + `RSVC_START`; each replica runs `src/rsvc/srv.c::ds_rsvc_start_handler` -> + `src/rsvc/srv.c::ds_rsvc_start` -> `src/rdb/rdb.c::rdb_create`. + +**KB source:** flows/pool-create.md + +### Q16 (C) Engine module loading + +**Ask:** How does `daos_engine` load a server module, and what must a new module provide? + +**Key facts:** +- `src/engine/module.c::dss_module_load` dlopens `lib.so` and looks up `_module` + (`struct dss_module`). +- Default list `src/engine/init.c::MODULE_LIST`; aliases in `src/engine/init.c::modules_load`; + check mode uses `MODS_LIST_CHK`. + +**KB source:** modules/engine.md + +### Q17 (C) Adding a dRPC method + +**Ask:** I need a new dRPC call from `daos_server` to the engine. Which files must change? + +**Key facts:** +- Method ID in `src/include/daos/drpc_modules.h`. +- Go constant in `src/control/lib/daos/drpc.go`. +- Protobuf in `src/proto` (regenerate Go and C code). +- Engine dispatch in `src/mgmt/srv.c` and handler in `src/mgmt/srv_drpc.c`. +- Go caller (e.g. in `src/control/server/mgmt_*.go`). + +**KB source:** modules/control.md, .github/instructions/control.instructions.md + +### Q18 (C) Changing a VOS persistent struct + +**Ask:** I want to add a field to `struct vos_obj_df`. What else must I do? + +**Key facts:** +- It is on-media format (`src/vos/vos_layout.h`): bump the layout/pool version and handle + upgrade of existing pools. + +**KB source:** modules/vos.md + +### Q19 (I) VOS transactions + +**Ask:** Can I call a function that may yield the ULT while inside a VOS/umem transaction? + +**Key facts:** +- No. Never yield inside a `umem` transaction (`vos_tx_begin()` ... `vos_tx_end()`). + +**KB source:** modules/vos.md, .github/copilot-instructions.md + +### Q20 (L) MS leadership and state + +**Ask:** In `daos_server`, how does a management-service handler make sure it runs on the +leader, and how is MS state persisted? + +**Key facts:** +- `src/control/server/mgmt_svc.go::mgmtSvc.checkLeaderRequest` -> + `src/control/system/raft/database.go::Database.CheckLeader`. +- State changes are Raft log entries: `src/control/system/raft/raft.go::Database.submitRaftUpdate`, + applied by `src/control/system/raft/raft.go::fsm.Apply` (bbolt-backed). + +**KB source:** modules/control.md + +### Q21 (L) EC parity marker + +**Ask:** How are EC parity extents distinguished from data extents? + +**Key facts:** +- High bit of the offset: `src/include/daos/object.h::DAOS_EC_PARITY_BIT` (`1ULL << 63`). + +**KB source:** modules/object.md + +### Q22 (I) Rebuild throttling + +**Ask:** How does the engine stop rebuild traffic from starving client I/O? + +**Key facts:** +- Rebuild scan and migrate RPCs are classified as `SCHED_REQ_MIGRATE` + (`src/rebuild/srv.c::rebuild_get_req_attr`, `src/object/srv_mod.c`), and the per-xstream + scheduler (`src/engine/sched.c`) throttles by request type. + +**KB source:** modules/rebuild.md, modules/engine.md + +### Q23 (T) DFS write path + +**Ask:** Which DAOS KV API call does `dfs_write()` use to store file data? + +**Key facts:** +- None: `src/client/dfs/io.c::dfs_write` calls `daos_array_write()` (array API), which becomes + `DAOS_OPC_OBJ_UPDATE` sub-tasks (`src/client/array/dc_array.c::dc_array_io`). + +**KB source:** flows/write-path.md + +### Q24 (G) Container open handler + +**Ask:** Which server function handles `CONT_OPEN`? + +**Key facts:** +- `src/container/srv_container.c::ds_cont_op_handler` (shared by most container ops). + +**KB source:** generated/rpcs.md only (no container card) + +### Q25 (G) Raft vote handling in rdb + +**Ask:** Which function handles Raft vote requests between pool service replicas? + +**Key facts:** +- `RDB_REQUESTVOTE` -> `src/rdb/rdb_raft.c::rdb_requestvote_handler`. + +**KB source:** generated/rpcs.md only (no rdb card) + +### Q26 (G) Aggregation ULTs + +**Ask:** Which functions start and run VOS and EC aggregation for a container on a target? + +**Key facts:** +- `src/container/srv_target.c::cont_start_agg` creates `src/container/srv_target.c::cont_agg_ult` + and `src/container/srv_target.c::cont_ec_agg_ult` with `sched_create_ult()`. +- VOS aggregation calls `src/vos/vos_aggregate.c::vos_aggregate`. + +**KB source:** modules/vos.md (partial; no container card) diff --git a/utils/kb/kb.py b/utils/kb/kb.py new file mode 100755 index 00000000000..02b9b752383 --- /dev/null +++ b/utils/kb/kb.py @@ -0,0 +1,457 @@ +#!/usr/bin/env python3 +""" + Copyright 2026 Hewlett Packard Enterprise Development LP + All rights reserved. + + SPDX-License-Identifier: BSD-2-Clause-Patent + + Build and validate the DAOS knowledge base under docs/kb. + + extract regenerate docs/kb/generated/* from the source tree + lint verify that paths and symbols referenced by the KB still exist + check fail if generated files are stale, then run lint (for CI) +""" +import argparse +import json +import os +import re +import subprocess # nosec +import sys +from collections import defaultdict +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +SRC = ROOT / "src" +KB = ROOT / "docs" / "kb" +GEN = KB / "generated" +GEN_HEADER = "\n\n" + +PRUNE_DIRS = {"vendor", "tests", "raft", ".git", "__pycache__"} +NON_C_MODULES = {"include", "tests", "proto", "control"} + +# Owner module of each src/include/daos_srv header. +DAOS_SRV_OWNER = { + "ad_mem": "common", "bio": "bio", "checker": "chk", "container": "container", + "daos_chk": "chk", "daos_engine": "engine", "daos_mgmt_srv": "mgmt", "dtx_srv": "dtx", + "evtree": "vos", "iv": "engine", "mgmt_tgt_common": "mgmt", "object": "object", + "pool": "pool", "ras": "engine", "rdb": "rdb", "rebuild": "rebuild", "rsvc": "rsvc", + "security": "security", "smd": "bio", "srv_csum": "object", "srv_obj_ec": "object", + "vea": "vea", "vos": "vos", "vos_types": "vos", +} + +C_KEYWORDS = {"if", "for", "while", "switch", "return", "sizeof", "do", "else", "case"} +FUNC_DEF_RE = re.compile(r"^(?:[A-Za-z_][\w \t*]*?[\s*])?([A-Za-z_]\w*)\s*\([^;]*$") +RPC_LIST_DEF_RE = re.compile(r"^#define\s+(\w+_PROTO_(CLI|SRV)_RPC_LIST)\b") +DSS_MODULE_RE = re.compile(r"^struct dss_module (\w+)\s*=\s*\{(.*?)\};", re.M | re.S) +DSS_FIELD_RE = re.compile(r"\.sm_(\w+)\s*=\s*([^,\n]+)") +INC_SRV_RE = re.compile(r'^#include\s+[<"]daos_srv/(\w+)\.h[>"]', re.M) +INC_REL_RE = re.compile(r'^#include\s+"((?:\.\./)+[\w./]+)"', re.M) +DRPC_ENUM_RE = re.compile(r"\b(DRPC_METHOD_\w+)\s*=\s*(\d+)") +DRPC_GO_RE = re.compile(r"\b(Method\w+)\s+\w+\s*=\s*C\.(DRPC_METHOD_\w+)") +DRPC_CASE_RE = re.compile(r"case\s+(DRPC_METHOD_\w+)\s*:\s*\n\s*(?:rc\s*=\s*)?(\w+)\(") + + +def rel(path): + """Return a repo-relative posix path.""" + return Path(path).relative_to(ROOT).as_posix() + + +def walk(base, exts, prune=PRUNE_DIRS): + """Yield files under base with one of the given extensions, pruning noisy dirs.""" + for dirpath, dirnames, filenames in os.walk(base): + dirnames[:] = sorted(d for d in dirnames if d not in prune) + for name in sorted(filenames): + if name.endswith(exts): + yield Path(dirpath) / name + + +def read(path): + """Read a source file, tolerating odd encodings.""" + return Path(path).read_text(encoding="utf-8", errors="replace") + + +def module_of(path): + """Map a source path to its DAOS module (first dir under src/).""" + parts = Path(path).relative_to(SRC).parts + return parts[0] if len(parts) > 1 else None + + +def function_index(): + """Map C function name -> sorted list of files that define it.""" + index = defaultdict(set) + for path in walk(SRC, (".c",), PRUNE_DIRS | {"control"}): + for line in read(path).splitlines(): + if not line or line[0] in " \t#/{}*": + continue + match = FUNC_DEF_RE.match(line) + if match and match.group(1) not in C_KEYWORDS: + index[match.group(1)].add(rel(path)) + return {name: sorted(files) for name, files in index.items()} + + +def macro_body(lines, start): + """Join a backslash-continued #define starting at lines[start].""" + body = [] + i = start + while True: + line = lines[i] + body.append(line.rstrip("\\ \t")) + if not line.rstrip().endswith("\\"): + return " ".join(body) + i += 1 + + +def split_x_entries(body): + """Yield the top-level argument lists of every X(...) in a macro body.""" + pos = 0 + while True: + start = body.find("X(", pos) + if start < 0: + return + depth, args, cur = 0, [], [] + i = start + 1 + for i in range(start + 1, len(body)): + char = body[i] + if char == "(": + depth += 1 + if depth == 1: + continue + elif char == ")": + depth -= 1 + if depth == 0: + break + if char == "," and depth == 1: + args.append("".join(cur).strip()) + cur = [] + else: + cur.append(char) + args.append("".join(cur).strip()) + pos = i + 1 + yield args + + +def extract_rpcs(funcs): + """Parse every *_PROTO_{CLI,SRV}_RPC_LIST macro into RPC records.""" + rpcs = [] + for path in walk(SRC, (".h",), PRUNE_DIRS | {"control"}): + lines = read(path).splitlines() + for i, line in enumerate(lines): + match = RPC_LIST_DEF_RE.match(line) + if not match: + continue + for args in split_x_entries(macro_body(lines, i)): + if len(args) < 4: + continue + handler = args[3] + rpcs.append({ + "opcode": args[0], + "module": module_of(path), + "direction": "client->server" if match.group(2) == "CLI" else "server->server", + "handler": handler, + "handler_files": funcs.get(handler, []), + "collective": len(args) > 4 and args[4] != "NULL", + "defined_in": rel(path), + }) + return rpcs + + +def extract_modules(): + """Parse every 'struct dss_module X = {...}' engine module registration.""" + mods = [] + for path in walk(SRC, (".c",), PRUNE_DIRS | {"control"}): + for match in DSS_MODULE_RE.finditer(read(path)): + fields = {k: v.strip() for k, v in DSS_FIELD_RE.findall(match.group(2))} + mods.append({ + "symbol": match.group(1), + "name": fields.get("name", "").strip('"'), + "mod_id": fields.get("mod_id", ""), + "init": fields.get("init", ""), + "setup": fields.get("setup", ""), + "file": rel(path), + "module": module_of(path), + }) + return sorted(mods, key=lambda m: m["name"]) + + +def extract_deps(): + """Infer module -> module dependencies from #include directives.""" + deps = defaultdict(lambda: defaultdict(int)) + for path in walk(SRC, (".c", ".h"), PRUNE_DIRS | NON_C_MODULES): + src_mod = module_of(path) + text = read(path) + targets = {DAOS_SRV_OWNER.get(h) for h in INC_SRV_RE.findall(text)} + for inc in INC_REL_RE.findall(text): + resolved = (path.parent / inc).resolve() + if SRC in resolved.parents: + targets.add(module_of(resolved)) + for dst in targets: + if dst and dst != src_mod and dst not in NON_C_MODULES: + deps[src_mod][dst] += 1 + return {src: dict(sorted(dsts.items())) for src, dsts in sorted(deps.items())} + + +def extract_drpc(funcs): + """Join dRPC method IDs with their Go constants, Go callers and engine handlers.""" + ids = dict(DRPC_ENUM_RE.findall(read(SRC / "include/daos/drpc_modules.h"))) + go_const = {c: g for g, c in DRPC_GO_RE.findall(read(SRC / "control/lib/daos/drpc.go"))} + callers = defaultdict(set) + for path in walk(SRC / "control", (".go",)): + if path.name.endswith("_test.go"): + continue + for name in set(re.findall(r"\bdaos\.(Method\w+)\b", read(path))): + callers[name].add(rel(path)) + handlers = {} + for path in walk(SRC, (".c",), PRUNE_DIRS | {"control"}): + for method, func in DRPC_CASE_RE.findall(read(path)): + handlers[method] = {"func": func, "files": funcs.get(func, [rel(path)])} + out = [] + for method, num in sorted(ids.items(), key=lambda kv: int(kv[1])): + gname = go_const.get(method, "") + out.append({ + "method": method, "id": int(num), "go_const": gname, + "go_callers": sorted(callers.get(gname, [])), + "engine_handler": handlers.get(method, {}).get("func", ""), + "engine_files": handlers.get(method, {}).get("files", []), + }) + return out + + +FRONT_RE = re.compile(r"\A---\n(.*?)\n---\n", re.S) + + +def card_frontmatter(): + """Parse the simple 'key: value' / 'key: [a, b]' frontmatter of hand-written KB cards.""" + cards = [] + for path in sorted(KB.rglob("*.md")): + if GEN in path.parents: + continue + match = FRONT_RE.match(read(path)) + if not match: + continue + meta = {"file": rel(path)} + for line in match.group(1).splitlines(): + if ":" not in line: + continue + key, val = (s.strip() for s in line.split(":", 1)) + if val.startswith("["): + meta[key] = [v.strip() for v in val.strip("[]").split(",") if v.strip()] + else: + meta[key] = val + if "id" in meta: + cards.append(meta) + return cards + + +def code(text): + """Wrap text in backticks for markdown tables.""" + return f"`{text}`" if text else "" + + +def render_rpcs(rpcs): + """Render the RPC table grouped by module.""" + out = [GEN_HEADER, "# CaRT RPC opcodes by module\n\n", + "Source: `*_PROTO_{CLI,SRV}_RPC_LIST` X-macros. client->server RPCs are sent by " + "libdaos; server->server RPCs are engine-internal (collective = has corpc ops).\n"] + by_mod = defaultdict(list) + for rpc in rpcs: + by_mod[rpc["module"]].append(rpc) + for mod in sorted(by_mod): + out.append(f"\n## {mod}\n\nDefined in {code(by_mod[mod][0]['defined_in'])}\n\n") + out.append("| Opcode | Direction | Handler | Handler file | Collective |\n") + out.append("|---|---|---|---|---|\n") + for rpc in by_mod[mod]: + files = ", ".join(code(f) for f in rpc["handler_files"]) + out.append(f"| {code(rpc['opcode'])} | {rpc['direction']} | {code(rpc['handler'])} " + f"| {files} | {'yes' if rpc['collective'] else ''} |\n") + return "".join(out) + + +def render_modules(mods, deps): + """Render engine module registrations and the include-derived dependency table.""" + out = [GEN_HEADER, "# Engine modules and dependencies\n\n", + "## Engine modules (`struct dss_module`)\n\n", + "Loaded by `daos_engine` (see `src/engine/module.c`).\n\n", + "| Name | Module ID | Symbol | Init | File |\n|---|---|---|---|---|\n"] + for mod in mods: + out.append(f"| {mod['name']} | {code(mod['mod_id'])} | {code(mod['symbol'])} " + f"| {code(mod['init'])} | {code(mod['file'])} |\n") + out.append("\n## Module dependencies (from `#include`)\n\n" + "An edge A -> B means files in `src/A` include a `daos_srv/` header owned by B or a " + "file under `src/B`. The number is how many files do so.\n\n" + "| Module | Depends on |\n|---|---|\n") + for src, dsts in deps.items(): + listed = ", ".join(f"{d} ({n})" for d, n in dsts.items()) + out.append(f"| {src} | {listed} |\n") + users = defaultdict(list) + for src, dsts in deps.items(): + for dst in dsts: + users[dst].append(src) + out.append("\n## Reverse dependencies (used by)\n\n| Module | Used by |\n|---|---|\n") + for dst in sorted(users): + out.append(f"| {dst} | {', '.join(sorted(users[dst]))} |\n") + return "".join(out) + + +def render_drpc(drpc): + """Render the control-plane <-> engine dRPC method table.""" + out = [GEN_HEADER, "# dRPC methods (daos_server <-> daos_engine)\n\n", + "IDs from `src/include/daos/drpc_modules.h`; Go constants from " + "`src/control/lib/daos/drpc.go`. 2xx are handled by the engine (`src/mgmt/srv.c`); " + "3xx are engine -> daos_server notifications handled in Go.\n\n", + "| ID | Method | Go const | Go callers | Engine handler | Handler file |\n", + "|---|---|---|---|---|---|\n"] + for item in drpc: + callers = ", ".join(code(c) for c in item["go_callers"]) + files = ", ".join(code(f) for f in item["engine_files"]) + out.append(f"| {item['id']} | {code(item['method'])} | {code(item['go_const'])} " + f"| {callers} | {code(item['engine_handler'])} | {files} |\n") + return "".join(out) + + +def build_graph(rpcs, mods, deps, drpc, cards): + """Assemble a node/edge graph combining extracted facts and card frontmatter.""" + nodes, edges = {}, [] + + def node(nid, kind, **attrs): + nodes.setdefault(nid, {"id": nid, "kind": kind}).update(attrs) + + for src, dsts in deps.items(): + node(src, "module") + for dst, weight in dsts.items(): + node(dst, "module") + edges.append({"src": src, "dst": dst, "kind": "includes", "weight": weight}) + for mod in mods: + node(mod["module"], "module", engine_module=mod["name"]) + for rpc in rpcs: + rid = f"rpc:{rpc['opcode']}" + node(rid, "rpc", direction=rpc["direction"], handler=rpc["handler"], + handler_files=rpc["handler_files"]) + edges.append({"src": rpc["module"], "dst": rid, "kind": "handles_rpc"}) + for item in drpc: + did = f"drpc:{item['method']}" + node(did, "drpc", method_id=item["id"], handler=item["engine_handler"]) + if item["engine_handler"]: + edges.append({"src": "mgmt", "dst": did, "kind": "handles_drpc"}) + for card in cards: + node(card["id"], card.get("kind", "module"), card=card["file"]) + nodes[card["id"]]["kind"] = card.get("kind", "module") + for card in cards: + for key in ("depends_on", "used_by", "related"): + for dst in card.get(key, []): + node(dst, "module" if key != "related" else "concept") + edges.append({"src": card["id"], "dst": dst, "kind": key, "from": "card"}) + edges.sort(key=lambda e: (e["src"], e["kind"], e["dst"])) + return {"nodes": sorted(nodes.values(), key=lambda n: n["id"]), "edges": edges} + + +def generate(): + """Return {path: content} for every generated KB file.""" + funcs = function_index() + rpcs = extract_rpcs(funcs) + mods = extract_modules() + deps = extract_deps() + drpc = extract_drpc(funcs) + graph = build_graph(rpcs, mods, deps, drpc, card_frontmatter()) + return { + GEN / "rpcs.md": render_rpcs(rpcs), + GEN / "modules.md": render_modules(mods, deps), + GEN / "drpc.md": render_drpc(drpc), + GEN / "graph.json": json.dumps(graph, indent=1, sort_keys=True) + "\n", + } + + +def cmd_extract(_args): + """Write generated files.""" + GEN.mkdir(parents=True, exist_ok=True) + for path, content in generate().items(): + path.write_text(content, encoding="utf-8") + print(f"wrote {rel(path)}") + return 0 + + +LINK_RE = re.compile(r"\[[^\]]*\]\(([^)\s]+)\)") +SCOPED_SYM_RE = re.compile(r"`([\w./-]+\.\w+)::([\w.]+)(?:\(\))?`") +BARE_SYM_RE = re.compile(r"`([A-Za-z_]\w*)\(\)`") +PATH_RE = re.compile(r"`((?:src|docs|utils|site_scons|ci|\.github)/[\w./-]*[\w/])`") + + +def kb_docs(): + """Hand-written files that the linter validates.""" + files = [p for p in KB.rglob("*.md") if GEN not in p.parents] + files += sorted((ROOT / ".github" / "instructions").glob("*.md")) + files += [p for p in [ROOT / ".github" / "copilot-instructions.md"] if p.exists()] + files += sorted((ROOT / ".github" / "skills").rglob("*.md")) + files += sorted((ROOT / "utils" / "kb" / "benchmark").glob("*.md")) + return sorted(files) + + +def existing_words(words): + """Return the subset of words that occur as whole words somewhere under src/.""" + if not words: + return set() + cmd = ["git", "-C", str(ROOT), "grep", "-h", "-o", "-w", "-F"] + for word in sorted(words): + cmd += ["-e", word] + cmd += ["--", "src", ":!src/control/vendor", ":!src/rdb/raft"] + result = subprocess.run(cmd, capture_output=True, text=True, check=False) # nosec + return set(result.stdout.split()) + + +def cmd_lint(_args): + """Check that every path / symbol referenced in the KB exists.""" + errors, bare = [], defaultdict(list) + for doc in kb_docs(): + text = read(doc) + where = rel(doc) + for target in LINK_RE.findall(text): + if re.match(r"^[a-z]+:", target) or target.startswith("#"): + continue + path = target.split("#", 1)[0] + base = ROOT if path.startswith("/") else doc.parent + if path and not (base / path.lstrip("/")).exists(): + errors.append(f"{where}: broken link {target}") + for path in PATH_RE.findall(text): + if not (ROOT / path).exists(): + errors.append(f"{where}: missing path {path}") + for path, sym in SCOPED_SYM_RE.findall(text): + name = sym.rsplit(".", 1)[-1] + target = ROOT / path + if not target.is_file(): + errors.append(f"{where}: missing file {path} (for {sym})") + elif not re.search(rf"\b{re.escape(name)}\b", read(target)): + errors.append(f"{where}: {name} not found in {path}") + for sym in BARE_SYM_RE.findall(text): + bare[sym].append(where) + found = existing_words(set(bare)) + for sym, wheres in sorted(bare.items()): + if sym not in found: + errors.append(f"{wheres[0]}: symbol {sym}() not found under src/") + for err in errors: + print(err) + print(f"kb lint: {len(kb_docs())} files, {len(errors)} problem(s)") + return 1 if errors else 0 + + +def cmd_check(args): + """Fail if generated files are out of date, then lint.""" + stale = [rel(p) for p, c in generate().items() + if not p.exists() or read(p) != c] + for path in stale: + print(f"stale: {path} (run utils/kb/kb.py extract)") + return max(1 if stale else 0, cmd_lint(args)) + + +def main(): + """Entry point.""" + parser = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + sub = parser.add_subparsers(dest="cmd", required=True) + sub.add_parser("extract").set_defaults(func=cmd_extract) + sub.add_parser("lint").set_defaults(func=cmd_lint) + sub.add_parser("check").set_defaults(func=cmd_check) + args = parser.parse_args() + return args.func(args) + + +if __name__ == "__main__": + sys.exit(main())