Skip to content

tests/e2e: generate against the shipped base - #2707

Open
ideaship wants to merge 1 commit into
mainfrom
sonic-e2e-base-config
Open

ideaship wants to merge 1 commit into
mainfrom
sonic-e2e-base-config

Conversation

@ideaship

Copy link
Copy Markdown
Contributor

Exports SONIC_BASE_CONFIG_PATH in tests/e2e/sonic_golden_test.sh, pointing it
at the in-repo files/sonic/config_db.json, and regenerates all nine goldens
against it.

Why

The harness set no base config path, so the generator fell back to
/etc/sonic/config_db.json — a path the Containerfile creates inside the
conductor image, and which does not exist on the host the harness runs on. Every
device started from {} and logged the base config as not found, so the goldens
pinned output for a configuration that never occurs in production.

The path stopped being hardcoded when SONIC_BASE_CONFIG_PATH was added; the
harness simply never set it. This is the one-line follow-up that takes advantage
of it, and it closes the base-config gap described under "Scope of the coverage
claim" in the series context issue.

What the diff contains

24 tables appear in every golden that appeared in none before — those populated
in the shipped base and never emitted by the generator:

CLASSIFIER_TABLE, COREDUMP, ECMP_LOADSHARE_TABLE_IPV4,
ECMP_LOADSHARE_TABLE_IPV6, FEATURE, FLEX_COUNTER_TABLE, HARDWARE,
HOST_FEATURE, KDUMP, MGMT_PORT, MGMT_VRF_CONFIG, NAT_GLOBAL, NEIGH_GLOBAL,
NTP, POLICY_BINDING_TABLE, POLICY_SECTIONS_TABLE, POLICY_TABLE,
SNMP_SERVER_GROUP, SNMP_SERVER_GROUP_ACCESS, SNMP_SERVER_VIEW, SSHD_COMMON,
SWITCH, TELEMETRY, ZTP

They reach real switches by passing through untouched, so nothing covered them
before. The union of populated tables across the golden set goes from 38 to 62,
and no golden loses a table — which is what the regeneration coverage guard
checks.

DEVICE_METADATA.localhost is the one table that changes content rather than
gaining presence: it inherits default_config_profile,
frr_mgmt_framework_config, intf_naming_mode and type. That is
INHERITED_TABLE_KEYS working as documented — purely additive, no generated
field overwritten.

One thing worth a second look

The inherited type is LeafRouter on all nine devices, spine and OOB
included
, because the shipped base hardcodes it and the generator never sets
that field. So every switch deployed from this base is labelled LeafRouter
regardless of role. That is pre-existing production behaviour, not something this
change introduces — but it was invisible while the goldens had no base, and it is
worth deciding whether it matters before it is pinned as expected output.

Reviewing this

The diff is +3681 / -17, but only one line of it is logic. The nine goldens are
near-identical +409 blocks of the same base content, so: read the harness line,
read one golden in full, spot-check the rest.

Verified

Regenerated against a fresh NetBox stack, not a warm one, so the result is
reproducible in CI:

  • 3483 unit tests pass
  • test_validator_artifacts accepts all nine enriched goldens — the merged base
    content validates, which is the question the regeneration existed to answer
  • the golden coverage gate still passes
  • the python-osism-sonic-e2e job will re-derive these goldens on this PR; its
    file matcher covers both files/sonic/ and tests/e2e/

🤖 Generated with Claude Code

Export SONIC_BASE_CONFIG_PATH in sonic_golden_test.sh, pointing it at
the in-repo files/sonic/config_db.json next to the existing
SONIC_PORT_CONFIG_PATH, and regenerate all nine goldens against it.

The harness set no base config path before this, so the generator fell
back to /etc/sonic/config_db.json -- a path the Containerfile creates
inside the conductor image, and which does not exist on the host the
harness runs on. Every device started from {}, and the goldens pinned
output for a configuration that never occurs in production. The path
itself stopped being hardcoded when SONIC_BASE_CONFIG_PATH was added;
the harness simply never set it.

Regenerating adds 24 tables to every golden -- those populated in the
shipped base and never emitted by the generator, among them FEATURE,
TELEMETRY, SWITCH, ZTP and the POLICY_* and ECMP_LOADSHARE_* groups.
They reach real switches by passing through untouched, so nothing
covered them before. The union of populated tables across the set goes
from 38 to 62 and no golden loses one.

DEVICE_METADATA.localhost changes content rather than gaining presence,
inheriting default_config_profile, frr_mgmt_framework_config,
intf_naming_mode and type. That is INHERITED_TABLE_KEYS working as
documented: purely additive, with no generated field overwritten.

The inherited type is LeafRouter on all nine devices, spine and OOB
included, because the shipped base hardcodes it and the generator never
sets that field. That is what production gets; the empty base was hiding
it.

Verified by regenerating against a fresh stack: 3483 unit tests pass,
the committed-artifact validator accepts all nine enriched goldens, and
the golden coverage gate still passes.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="tests/e2e/sonic_golden_test.sh" line_range="207" />
<code_context>
 export SONIC_EXPORT_DIR="${EXPORT_DIR}"
 export SONIC_EXPORT_IDENTIFIER="hostname"
 export SONIC_PORT_CONFIG_PATH="${REPO_ROOT}/files/sonic/port_config"
+export SONIC_BASE_CONFIG_PATH="${REPO_ROOT}/files/sonic/config_db.json"

 echo ">>> Generating SONiC configurations (tests/e2e/generate.py)"
</code_context>
<issue_to_address>
**issue (bug_risk):** All nine generated configurations now inherit `DEVICE_METADATA.localhost.type` as `LeafRouter` from the shared base, including the spine and OOB devices. The committed goldens therefore pin role metadata that is incorrect for those devices, and deploying those generated configs labels non-leaf switches as leaf routers.

**Triggers:** When the generated SONiC configuration is consumed for the spine or OOB fixtures.

**Suggested fix:** Make the base metadata role-neutral or set `DEVICE_METADATA.localhost.type` from the NetBox device role during generation, then regenerate the goldens.
</issue_to_address>

Sourcery assessment

Approval pending. 1 finding to address first.

Blocking findings: tests/e2e/sonic_golden_test.sh:207


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

export SONIC_EXPORT_DIR="${EXPORT_DIR}"
export SONIC_EXPORT_IDENTIFIER="hostname"
export SONIC_PORT_CONFIG_PATH="${REPO_ROOT}/files/sonic/port_config"
export SONIC_BASE_CONFIG_PATH="${REPO_ROOT}/files/sonic/config_db.json"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): All nine generated configurations now inherit DEVICE_METADATA.localhost.type as LeafRouter from the shared base, including the spine and OOB devices. The committed goldens therefore pin role metadata that is incorrect for those devices, and deploying those generated configs labels non-leaf switches as leaf routers.

Triggers: When the generated SONiC configuration is consumed for the spine or OOB fixtures.

Suggested fix: Make the base metadata role-neutral or set DEVICE_METADATA.localhost.type from the NetBox device role during generation, then regenerate the goldens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New

Development

Successfully merging this pull request may close these issues.

2 participants