DAOS-19887 doc: add DAOS knowledge base

.github/copilot-instructions.md

@@ -0,0 +1,48 @@
+# DAOS: guidance for Copilot
+
+DAOS is a distributed object store: Go control plane (`src/control`), C data plane
+(`daos_engine`, `src/engine` + server modules) and C client library (`libdaos`, `src/client` +
+`cli*.c` in pool/container/object).
+
+## Use the knowledge base first
+
+Before searching `src/` broadly, read the relevant files in [docs/kb](../docs/kb/README.md):
+
+- [docs/kb/architecture.md](../docs/kb/architecture.md): processes, channels (gRPC / dRPC / CaRT),
+  engine xstreams, module index.
+- [docs/kb/concepts.md](../docs/kb/concepts.md): pool, container, object, epoch, DTX, target, ... mapped to code.
+- Module cards in [docs/kb/modules](../docs/kb/modules): vos, dtx, object, pool, rebuild, engine,
+  control (Go control plane).
+- Flow traces in [docs/kb/flows](../docs/kb/flows): write path, pool create, rebuild.
+- Generated lookups:
+  - "who handles RPC X?": [docs/kb/generated/rpcs.md](../docs/kb/generated/rpcs.md)
+  - "who handles dRPC method X / who calls it from Go?": [docs/kb/generated/drpc.md](../docs/kb/generated/drpc.md)
+  - "what does module X depend on / who uses it?": [docs/kb/generated/modules.md](../docs/kb/generated/modules.md)
+
+KB references use `path::symbol`; jump to the file and search for the symbol (no line numbers).
+If a KB statement contradicts the code, trust the code and fix the KB.
+
+## Code conventions
+
+- C style: `.clang-format`; function return type on its own line, name at column 0 (so
+  `^name(` finds definitions).
+- Errors are negative `-DER_*` codes (`src/include/daos_errno.h`); log with `D_ERROR`/`DL_ERROR`
+  etc. from `src/include/gurt/debug.h`; goto-based cleanup (`D_GOTO(out, rc = ...)`).
+- Memory: `D_ALLOC*`/`D_FREE` (never raw malloc/free in DAOS code).
+- Engine code runs in Argobots ULTs: never block an xstream (no sleeping syscalls, no pthread
+  waits); yield or use ABT primitives. Never yield inside a `umem`/VOS transaction.
+- RPCs are declared with X-macro lists (`*_PROTO_*_RPC_LIST`) and `CRT_RPC_DECLARE`; changing a
+  wire format requires a protocol version bump.
+- Go code is under `src/control`; ignore `src/control/vendor`.
+
+## Build and test
+
+- Build: `scons --jobs=$(nproc) install` (see [docs/dev/development.md](../docs/dev/development.md)).
+- C unit tests: `utils/run_utest.py` (config `utils/utest.yaml`); Go: `src/control/run_go_tests.sh`.
+- Functional tests: `src/tests/ftest` (Avocado, `launch.py`).
+
+## Keeping the KB correct
+
+After renaming/moving code referenced by the KB, run `utils/kb/kb.py check` (regenerates nothing,
+fails on stale generated files or broken references); fix with `utils/kb/kb.py extract` and by
+editing the cards.

.github/instructions/control.instructions.md

@@ -0,0 +1,20 @@
+---
+applyTo: "src/control/**,src/proto/**"
+description: "Go control plane (daos_server, daos_agent, dmg) context and rules"
+---
+# Working in the control plane
+
+Read [docs/kb/modules/control.md](../../docs/kb/modules/control.md) first; dRPC method mapping is
+in [docs/kb/generated/drpc.md](../../docs/kb/generated/drpc.md) and an end-to-end example in
+[docs/kb/flows/pool-create.md](../../docs/kb/flows/pool-create.md).
+
+- Never edit `src/control/vendor` or generated protobuf code (`*.pb.go`, `*.pb-c.[ch]`);
+  change `src/proto` and regenerate.
+- MS state changes go through Raft (`system/raft.Database`); leader-only handlers call
+  `checkLeaderRequest`.
+- A new dRPC method needs matching changes in `src/include/daos/drpc_modules.h`,
+  `src/control/lib/daos/drpc.go`, the Go caller and the engine handler (`src/mgmt/srv.c`),
+  then `utils/kb/kb.py extract`.
+- Follow existing patterns: `dmg` command -> `lib/control` function -> `server/mgmt_*.go`
+  handler; add `_test.go` coverage with the existing mocks.
+- Tests: `src/control/run_go_tests.sh`.

.github/instructions/dtx.instructions.md

@@ -0,0 +1,19 @@
+---
+applyTo: "src/dtx/**,src/include/daos_srv/dtx_srv.h,src/include/daos/dtx.h"
+description: "DTX (distributed transaction) context and rules"
+---
+# Working in DTX
+
+Read [docs/kb/modules/dtx.md](../../docs/kb/modules/dtx.md) first; the leader/non-leader sequence
+is in [docs/kb/flows/write-path.md](../../docs/kb/flows/write-path.md).
+
+- Leader: `dtx_leader_begin()` -> `dtx_leader_exec_ops()` -> `dtx_leader_end()`.
+  Non-leader: `dtx_begin()` -> local work -> `dtx_end()`.
+- Commit is normally asynchronous (CoS cache + `dtx_batched_commit` ULT); only `dth_sync` commits
+  inline. Do not assume a DTX is committed when the client gets its reply.
+- `-DER_INPROGRESS` means a reader hit a prepared DTX: refresh (`dtx_refresh()`) or retry; never
+  treat it as a hard failure.
+- Resends must be detected (`dtx_handle_resend()`) so updates are never applied twice.
+- Persistent DTX state lives in VOS (`src/vos/vos_dtx.c`); keep the two in sync when changing
+  states (`src/include/daos/dtx.h`).
+- Unit tests: `src/dtx/tests`; integration: `src/tests/suite/daos_dist_tx.c`.

.github/instructions/engine.instructions.md

@@ -0,0 +1,17 @@
+---
+applyTo: "src/engine/**,src/include/daos_srv/daos_engine.h,src/include/daos_srv/iv.h"
+description: "daos_engine runtime (xstreams, ULTs, scheduler, module loading, dRPC server) context and rules"
+---
+# Working in the engine runtime
+
+Read [docs/kb/modules/engine.md](../../docs/kb/modules/engine.md) first; module/RPC tables are in
+[docs/kb/generated/modules.md](../../docs/kb/generated/modules.md) and
+[docs/kb/generated/drpc.md](../../docs/kb/generated/drpc.md).
+
+- Everything runs in Argobots ULTs: never block an xstream (no `sleep()`, blocking syscalls,
+  pthread waits); use `dss_sleep()`, ABT primitives, `dss_ult_execute()` or `dss_chore_register()`.
+- Respect xstream roles: VOS work on the target main xstream, pool/rdb/dRPC work on xs 0.
+- Startup ordering matters: modules are initialised in `dss_module_init_all()` before xstreams
+  exist; `sm_setup` runs later from `dss_module_setup_all()` after the rank is set up.
+- Changes to `struct dss_module` or scheduler request types affect every server module.
+- Tests: `src/engine/tests`.

.github/instructions/kb.instructions.md

@@ -0,0 +1,17 @@
+---
+applyTo: "docs/kb/**,utils/kb/**"
+description: "How to maintain the DAOS knowledge base"
+---
+# Maintaining the DAOS knowledge base
+
+- Never hand-edit `docs/kb/generated/*`; run `utils/kb/kb.py extract`.
+- Every claim in a card must be checkable in code. Reference code as `<path>::<symbol>` or
+  `<symbol>()`; no line numbers. `utils/kb/kb.py lint` verifies them.
+- Cards start with frontmatter (`id`, `kind`, `depends_on`, `used_by`, `related`; lists in
+  `[a, b]` form) that `extract` merges into `generated/graph.json`.
+- Keep cards under ~200 lines with sections: Purpose, File map, Entry points, Key structures,
+  Threading, Invariants and gotchas, Tests, Where to look for. Flows are ordered hop lists plus a
+  Mermaid sequence diagram.
+- When adding a card, also add a scoped `.github/instructions/<module>.instructions.md` that
+  points at it, and list it in `docs/kb/README.md`.
+- Run `utils/kb/kb.py check` before committing.

.github/instructions/object.instructions.md

@@ -0,0 +1,19 @@
+---
+applyTo: "src/object/**,src/include/daos_srv/object.h,src/include/daos/object.h"
+description: "Object layer (client and server I/O) context and rules"
+---
+# Working in the object layer
+
+Read [docs/kb/modules/object.md](../../docs/kb/modules/object.md) first; end-to-end I/O is in
+[docs/kb/flows/write-path.md](../../docs/kb/flows/write-path.md); opcode -> handler table in
+[docs/kb/generated/rpcs.md](../../docs/kb/generated/rpcs.md).
+
+- `cli_*.c` run in libdaos (client tasks on the TSE scheduler); `srv_*.c` run in `daos_engine`
+  on target xstreams. Don't mix assumptions between the two.
+- Server handlers must validate the request with `obj_ioc_begin()` (handles, capabilities, pool
+  map version) and return `-DER_STALE` for old maps; the client refreshes and retries.
+- Modifications go through DTX (`dtx_leader_begin()` on the leader shard,
+  `ds_obj_tgt_update_handler()` on others). Don't write to VOS outside a DTX handle.
+- RPC formats: `src/object/obj_rpc.h` (`DAOS_ISEQ_*`/`DAOS_OSEQ_*` + `CRT_RPC_DECLARE`). Any wire
+  change requires bumping `DAOS_OBJ_VERSION` and keeping the previous format working.
+- Tests: `src/object/tests`, `src/tests/suite/daos_obj.c`, `src/tests/suite/daos_obj_ec.c`.

.github/instructions/pool.instructions.md

@@ -0,0 +1,18 @@
+---
+applyTo: "src/pool/**,src/include/daos_srv/pool.h,src/include/daos/pool.h,src/include/daos/pool_map.h,src/common/pool_map.c"
+description: "Pool, pool service and pool map context and rules"
+---
+# Working in the pool module
+
+Read [docs/kb/modules/pool.md](../../docs/kb/modules/pool.md) first; creation end-to-end is in
+[docs/kb/flows/pool-create.md](../../docs/kb/flows/pool-create.md); opcode -> handler table in
+[docs/kb/generated/rpcs.md](../../docs/kb/generated/rpcs.md).
+
+- Pool-service code (`src/pool/srv_pool.c`) runs on the system xstream of replica ranks, only does
+  work as leader, and changes metadata only inside an rdb transaction on the current term.
+- Per-target state is `ds_pool_child`; target work must run on the target xstream (use collective
+  helpers), never inline on xs 0.
+- Pool map versions are monotonic; every map change must be distributed (IV) and may need a
+  rebuild (`ds_rebuild_schedule()`).
+- rdb layout changes need a version bump in `src/pool/srv_layout.h` and upgrade handling.
+- Tests: `src/tests/suite/daos_pool.c`, `src/tests/suite/daos_rebuild.c`, `src/tests/ftest/pool`.

.github/instructions/rebuild.instructions.md

@@ -0,0 +1,17 @@
+---
+applyTo: "src/rebuild/**,src/include/daos_srv/rebuild.h,src/object/srv_obj_migrate.c"
+description: "Rebuild and object migration context and rules"
+---
+# Working in rebuild / migration
+
+Read [docs/kb/modules/rebuild.md](../../docs/kb/modules/rebuild.md) first; the end-to-end trace
+(failure -> exclude -> rebuild -> reclaim) is [docs/kb/flows/rebuild.md](../../docs/kb/flows/rebuild.md);
+placement and target states are covered in [docs/kb/modules/pool.md](../../docs/kb/modules/pool.md).
+
+- Rebuild is leader-driven (pool service leader, xs 0) and executed on every engine; scan and
+  migration run on target xstreams. Keep that split.
+- All map-driven movement uses `RB_OP_REBUILD`; behaviour differs by target state in the pool
+  map, not by op code. Successful rebuilds are followed by `RB_OP_RECLAIM`.
+- Scanning must wait for DTX resync; migration must fetch at stable epochs.
+- Progress is reported through rebuild IV; never block the leader waiting on targets.
+- Tests: `src/tests/suite/daos_rebuild.c` and friends, `src/tests/ftest/rebuild`.

.github/instructions/vos.instructions.md

@@ -0,0 +1,20 @@
+---
+applyTo: "src/vos/**,src/include/daos_srv/vos.h,src/include/daos_srv/vos_types.h"
+description: "VOS (Versioning Object Store) context and rules"
+---
+# Working in VOS
+
+Read [docs/kb/modules/vos.md](../../docs/kb/modules/vos.md) first; for how VOS is reached from the
+network see [docs/kb/flows/write-path.md](../../docs/kb/flows/write-path.md).
+
+- VOS is per target and single-threaded per target xstream; it has no locking across xstreams and
+  no network code. Never call it from the system xstream for a target's pool.
+- Never yield (ABT yield, blocking I/O, RPC wait) inside a `umem` transaction
+  (`vos_tx_begin()` ... `vos_tx_end()`).
+- Everything is epoch-versioned (MVCC); punch is a tombstone recorded in the ilog. Do not update
+  in place.
+- `*_begin()` reserves resources; always pair with `*_end()` passing the error so reservations are
+  cancelled.
+- Any change to `src/vos/vos_layout.h` is an on-media format change: bump the layout version and
+  handle upgrade of existing pools.
+- Unit tests: `src/vos/tests` (`vos_tests`); add a test with each behaviour change.

docs/kb/README.md

@@ -0,0 +1,52 @@
+# DAOS Knowledge Base
+
+A compact, link-checked map of the DAOS code base, written for LLM agents (and humans) who need to
+find their way around `src/` quickly. Prefer these files over broad searches; follow the paths and
+`file::symbol` references they contain to jump straight to code.
+
+## How to use
+
+1. Start with [architecture.md](architecture.md) (processes, layers, channels, module index) and
+   [concepts.md](concepts.md) (vocabulary mapped to code).
+2. For a module, open its card in [modules/](modules/). Cards list purpose, file map, entry points,
+   key structs, threading, invariants, and where to look for common tasks.
+3. For cross-module behaviour, open a flow trace in [flows/](flows/). Each is an ordered call chain
+   with `file::function` at every hop.
+4. For exhaustive facts, use the generated tables (regenerated from source, never hand-edited):
+   - [generated/rpcs.md](generated/rpcs.md): every CaRT RPC opcode -> handler -> file.
+   - [generated/drpc.md](generated/drpc.md): every dRPC method -> Go caller -> engine handler.
+   - [generated/modules.md](generated/modules.md): engine modules and include-derived dependencies.
+   - [generated/graph.json](generated/graph.json): the same facts plus card metadata as a node/edge
+     graph.
+
+## Contents
+
+| Kind | File | Covers |
+|---|---|---|
+| Overview | [architecture.md](architecture.md) | Processes, layering, communication, module index |
+| Overview | [concepts.md](concepts.md) | Pool, container, object, epoch, DTX, target, xstream, ... |
+| Module | [modules/vos.md](modules/vos.md) | Versioning Object Store (per-target persistent store) |
+| Module | [modules/dtx.md](modules/dtx.md) | Distributed transactions |
+| Module | [modules/object.md](modules/object.md) | Object client/server I/O, EC, migration |
+| Module | [modules/pool.md](modules/pool.md) | Pool client, pool service, pool map |
+| Module | [modules/rebuild.md](modules/rebuild.md) | Rebuild, reclaim and object migration |
+| Module | [modules/engine.md](modules/engine.md) | `daos_engine` runtime: startup, xstreams, ULTs, scheduler, dRPC server, IV |
+| Module | [modules/control.md](modules/control.md) | Go control plane: `daos_server`, MS, `dmg`, `daos_agent` |
+| Flow | [flows/write-path.md](flows/write-path.md) | `dfs_write()` down to SCM/NVMe |
+| Flow | [flows/pool-create.md](flows/pool-create.md) | `dmg pool create` through control plane, engine, rsvc, rdb |
+| Flow | [flows/rebuild.md](flows/rebuild.md) | Rank failure / `dmg pool exclude` -> pool map update -> rebuild -> reclaim |
+| Flow | [flows/drive-replace.md](flows/drive-replace.md) | NVMe replacement and engine recovery -> device and pool verification |
+
+## Maintenance
+
+- `utils/kb/kb.py extract` regenerates `generated/`.
+- `utils/kb/kb.py check` fails if `generated/` is stale or if any path, link, or symbol referenced
+  by the KB or by `.github/` instructions no longer exists. Run it after renaming code.
+- `utils/kb/benchmark/questions.md` holds benchmark questions with verified answers for
+  measuring the KB (baseline vs. KB runs); its anchors are linted too.
+- Conventions for hand-written files:
+  - Reference code as `<path>::<symbol>` (checked: the file must contain the symbol) or
+    `<symbol>()` (checked to exist somewhere under `src/`). Never cite line numbers.
+  - Keep each card under ~200 lines; link instead of copying.
+  - Cards start with frontmatter (`id`, `kind`, `depends_on`, `used_by`, `related`) which
+    `kb.py extract` merges into `generated/graph.json`.