Validate model configs passed to get_dsp() as JSON - #339
Open
guicybercode wants to merge 1 commit into
Open
guicybercode wants to merge 1 commit into
guicybercode wants to merge 1 commit into
Conversation
get_dsp(const std::filesystem::path&) validates its input through
validate_nam_file() before loading, but get_dsp(const nlohmann::json&) handed
its config straight to populate_dsp_data(), which indexes "version",
"architecture" and "config" with operator[]. On a const nlohmann::json a
missing key asserts rather than throws, so well-formed JSON that simply is not
a model aborted the process:
$ ./repro
about to call get_dsp()
Assertion failed: (it != m_data.m_value.object->end()), function operator[],
file json.hpp, line 22188.
[exit 134]
Consumers that load whatever a user hands them could not defend against this;
the process died regardless of any try/catch. "weights" was already handled
correctly a few lines up, via GetWeights() throwing std::runtime_error.
Rather than add a second set of checks, split the in-memory half of
validate_nam_file() into validate_nam_json() and call it from both entry
points. The error strings for the file path case are unchanged byte for byte;
the "source" parameter is what varies. Also check that "version" and
"architecture" are strings, so a numeric version reports the same
NamFileValidationError as everything else on this path instead of leaking an
nlohmann::json::type_error.
NamFileValidationError derives from std::runtime_error, so consumers already
catching std::runtime_error are unaffected. The .wav branch of get_dsp() builds
its dspData directly and does not go through populate_dsp_data().
Validated on macOS/AppleClang: the new tests abort with exit 134 against the
unmodified sources and pass with the fix; full run_tests in both the default
and -DNAM_USE_INLINE_GEMM configurations; benchmodel, render, and loadmodel
against all nine files in example_models/.
Fixes sdatkinson#328
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #328.
The problem
get_dsp(const std::filesystem::path&)validates throughvalidate_nam_file()before loading.get_dsp(const nlohmann::json&)does not — it hands its config straight topopulate_dsp_data(), which indexes"version","architecture"and"config"withoperator[]. On aconst nlohmann::jsona missing key asserts instead of throwing, so well-formed JSON that simply is not a model takes the process down:The
try/catchin the reporter's repro cannot help — it is anabort(), not an exception."weights"was already handled correctly a few lines above viaGetWeights(), so it was only these three keys.The fix
Rather than add a second set of
contains()checks next to the existing ones, I split the in-memory half ofvalidate_nam_file()into avalidate_nam_json(config, source)and called it from both entry points.validate_nam_file()keeps the exists/read/parse handling and delegates the rest, so there is one definition of what a minimally valid config is.sourceis the text spliced in after"Invalid ". For the file path it is.nam file [/path/to/model.nam], which keeps those messages byte-identical to before (verified directly, not just via the existing assertions); for the JSON overloads it ismodel config, so the message readsInvalid model config: missing required key "version".I also added a check that
"version"and"architecture"are strings, as you suggested in the issue. Without it{"version": 7, ...}reaches.get<std::string>()and throwsnlohmann::json::type_error, which is a different type from theNamFileValidationErrorevery other failure on this path produces.I deliberately stopped there rather than also type-checking
"config"and"weights". Those already throw rather than abort, so it is a separate consistency question and not part of this bug — happy to add them if you want them.NamFileValidationErrorderives fromstd::runtime_error, so consumers catchingstd::runtime_errorare unaffected. The.wavbranch ofget_dsp()builds itsdspDatadirectly and never reachespopulate_dsp_data(). The file path now validates twice, which is a handful of hash lookups once at load time.Validation
On macOS / AppleClang:
threw, as one would hope: Invalid model config: missing required key "version".and exits 0.run_testsin the default build and with-DNAM_USE_INLINE_GEMM, matching the two Ubuntu configurations in CI.benchmodelonwavenet.namandlstm.nam,renderonexample_audio/input.wav, andloadmodelagainst all nine files inexample_models/— including the Sequential, Container and Slimmable envelopes, since those take a different route through config parsing.