diff --git a/.claude/commands/mendix/diff-script.md b/.claude/commands/mendix/diff-script.md index 7c9d183166..558bf0e696 100644 --- a/.claude/commands/mendix/diff-script.md +++ b/.claude/commands/mendix/diff-script.md @@ -72,25 +72,31 @@ Entity: MyModule.Order ## What Gets Compared -- **Entities**: Attributes, constraints, indexes, documentation -- **Enumerations**: Values and captions -- **Associations**: Type, owner, delete behavior -- **Microflows**: Parameters, return type, body statements +`mxcli diff` runs the script with exec's own code on a scratch copy of the +project, then compares the copy with the project unit by unit. It reports +exactly the units exec would write — every document kind (domain model per +entity and association, pages, snippets, layouts, flows, security, navigation, +settings, folders) and every statement kind — each shown as its `describe` +before and after. The project itself is not changed. + +A unit exec would rewrite although its description does not change is listed +on one line with the properties that change (`Modified: Page X: changed: +CanvasHeight`), or the folder it moves to. ## Summary Output Every diff ends with a summary: ``` -Summary: 2 new, 3 modified, 5 unchanged +Summary: 2 new, 3 modified, 0 removed — exec would write 4 unit(s) ``` -When a `create or modify` of an existing microflow or nanoflow would be refused -by `exec` (under `mdl 1;`, a change the splice cannot make), diff lists it as -`Refused: …` with the reason and adds `, N refused` to the summary. -One `exec` would write although its MDL renders as stored (a `folder` clause -naming another folder, applied as a move) is counted as modified and listed as -`Modified: … exec would write it: …`. +A script exec has already applied reports `exec would write nothing`. + +A script exec would refuse (its pre-flight checks) or a statement exec would +stop at (a plain `create` of an existing document; a flow change the splice +cannot make under `mdl 1;`) is reported as `Refused: …` with exec's message. +`--no-check` and `--continue-on-error` behave as they do for exec. ## Use Cases @@ -104,4 +110,6 @@ naming another folder, applied as a move) is counted as modified and listed as - Use `--color` for terminal output to easily spot changes - Use `--format struct` for a quick overview of what changes - Use `--format side` when comparing large objects -- Combine with `mxcli check` to validate syntax first +- Run it before `exec` on a Studio Pro-authored project: a write you did not + intend (a rewrite of a document you only meant to read back) shows up as a + modified unit diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli.jsonl b/.claude/skills/fix-issue/findings/cmd-mxcli.jsonl index 17a066416f..5033f17a65 100644 --- a/.claude/skills/fix-issue/findings/cmd-mxcli.jsonl +++ b/.claude/skills/fix-issue/findings/cmd-mxcli.jsonl @@ -135,3 +135,7 @@ {"date": "2026-10-01", "area": "cmd/mxcli", "symptom": "The LSP's CREATE MICROFLOW / CREATE NANOFLOW / CREATE ENUMERATION snippet completions inserted MDL that does not parse under any language version (`missing '(' at 'BEGIN'`, a quoted enumeration value name), the CONSTANT snippet the deprecated clause form (MDL-DEPR136), and PAGE/SNIPPET a statement mdl 1 refuses for its missing `;`; completion also offered alias-only keywords (SHOW_PAGE, DELETE_BEHAVIOR, DEFINE) and the `show entities` listings after SHOW.", "cause": "The snippets and keyword lists were hand-written once and never parsed; the grammar moved on (R2 property lists, R8 words, `list` for `show`) and nothing tied the completion text to it. The keyword list is generated from the lexer alone, which cannot tell an alias token from a canonical one: that is said by the `/* @alias MDL-DEPRnnn */` markers in the parser grammar.", "fix": "Snippets rewritten to canonical mdl 1. cmd/gen-completions reads the parser grammar too and drops a token every parser-rule use of which carries an @alias marker (the `keyword` rule, which lists tokens usable as names, does not count). `list` gets the listings; `show` offers page/message/home page.", "insight": "A completion text is MDL that ships in the binary, so it is held to what docs are held to: TestCompletionSnippetsAreMdl1 parses every snippet, expanded with its defaults, under `mdl 1;` and requires no deprecation. Deciding alias-only from the markers is the registry's own data; a token-swap rewrite word (`snippet`, `column`, `comment`) is NOT alias-only \u2014 those words stay canonical elsewhere \u2014 and neither is SHOW (show page, show message).", "issue": "ako/mxcli#714 (decision 5)", "file": "cmd/mxcli/lsp_completion.go, cmd/gen-completions/aliases.go", "test": "cmd/mxcli/lsp_mdl1_test.go TestCompletionSnippetsAreMdl1, TestCompletionOffersNoDeprecatedSpelling, TestCompletionListAndShowContinueIntoMdl1; cmd/gen-completions/aliases_test.go"} {"date": "2026-10-01", "area": "cmd/mxcli", "symptom": "A stub-then-real script set (two files each with `create or modify microflow X`) upgraded with `fmt --upgrade -p` one file at a time ended up split: the stub's file declined the header (MDL-V1-REBUILD) and the real flow's file took it. Run in order, the mdl 0 stub rebuilt the stored real flow and the mdl 1 real statement was refused, every run; mx check 0 errors, the app running the placeholder.", "cause": "Each command judged one file against the model as stored; check/fmt took exactly one file, so nothing could see a flow declared twice across the set, and a per-file header decision is wrong for a pair whose files must agree.", "fix": "check and fmt accept several files as one script set (cmd/mxcli/script_set.go): findFlowRedeclarations finds a flow declared by two create-or-modify statements; check warns MDL-STUB01 naming both; fmt --upgrade groups files sharing such a flow and declines the header for all of them when any one cannot take it (canTakeHeader = the per-file verdict). One file alone is unchanged.", "insight": "A per-file verdict is only right when the files are independent; the run is the unit when two files write the same document. The symptom hides because each half is individually correct (decline is right for the stub, header is right for the real file) - only the combination is wrong, so test the set, with the single-file decision as the control.", "issue": "ako/mxcli#905", "file": "cmd/mxcli/script_set.go", "test": "cmd/mxcli/script_set_test.go TestFmtUpgrade_ScriptSetDecidesStubThenRealHeaderTogether, TestCheck_ScriptSetWarnsOnStubThenReal"} {"date": "2026-10-01", "area": "cmd/mxcli", "symptom": "fmt --upgrade over a stub-then-real set still split the pair when the stub's file carried a written `mdl 0;`: the real file took `mdl 1;`. With the real file already `mdl 1;`, fmt printed 'no language header added' about it while the pair stayed split.", "cause": "canTakeHeader returned true for ANY written header (langver.ScanWrittenHeader's bool), reading a pinned `mdl 0;` as 'already has the header'; and the group decline was applied to a file that already carries the header, which fmt never removes.", "fix": "canTakeHeader: a written header can take it only when it is langver.Latest. decideSetHeaders: a file already under the header is not declined; fmt says the pair stays under different headers and to drop the stub or take the header off.", "insight": "ScanWrittenHeader's bool means 'a header is written', not 'the header is mdl 1'; a written mdl 0 pin is the strongest 'cannot take it' there is. Test the group decision with every header state of each file, not only headerless ones.", "issue": "ako/mxcli#905", "file": "cmd/mxcli/cmd_fmt.go", "test": "cmd/mxcli/script_set_test.go TestFmtUpgrade_ScriptSetPinnedMdl0StubHoldsTheRealFileBack, TestFmtUpgrade_ScriptSetAlreadySplitPairIsReportedAsSplit"} +{"date": "2026-10-01", "area": "cmd/mxcli", "symptom": "`mxcli version` fails with `unknown command \"version\" for \"mxcli\"` while `mxcli --version` works", "cause": "cobra's `Version:` field registers only the --version flag; no subcommand existed, though shouldSuppressWarning already listed a `version` argument", "fix": "Add a `version` subcommand (cmd/mxcli/cmd_version.go) printing exactly cobra's --version text (name, version, build time)", "insight": "Code that special-cases an argument (warning suppression) is evidence of an intended command; a test pinning the subcommand output to the flag output keeps the two from drifting", "issue": "ako/mxcli#534", "file": "cmd/mxcli/cmd_version.go", "test": "cmd/mxcli/cmd_version_test.go"} +{"date": "2026-10-01", "area": "cmd/mxcli", "symptom": "`mxcli check`/`exec`/`fmt`/`diff` fail on a script saved by Windows PowerShell 5.1: a UTF-8 BOM gives `line 1:0 token recognition error at: '\\ufeff'` (an invisible character), UTF-16LE gives a token error on almost every character; the same through stdin and in .test.mdl files", "cause": "Every script reader passed the raw file bytes to the lexer, which reads UTF-8 without a BOM; there was no shared reader (fmt, diff, the multi-file check pass, the test runner and EXECUTE SCRIPT each called os.ReadFile on their own)", "fix": "New mdl/srctext.Decode (strip a leading UTF-8 BOM, decode UTF-16LE/BE by BOM); readMDLSource calls it and fmt, diff and parseScriptSet now read through readMDLSource; testrunner.ParseTestFile and EXECUTE SCRIPT call it directly", "insight": "A BOM also hides a `mdl 1;` header from langver.ScanWrittenHeader, so stripping it in the parser alone would have left the language version wrong: decode where the bytes are read, before anything inspects the text. Enumerate the readers (grep os.ReadFile / io.ReadAll(os.Stdin)), not just the one the report names", "issue": "mendixlabs/mxcli#1253", "file": "mdl/srctext/srctext.go; cmd/mxcli/mdlsource.go", "test": "mdl/srctext/srctext_test.go; cmd/mxcli/mdlsource_encoding_test.go"} +{"date": "2026-10-01", "area": "cmd/mxcli", "symptom": "`mxcli -p App.mpr -c \"\"` opens the interactive REPL (a generator spawning mxcli with an open stdin hangs at `mdl>`); `-c \"describe entity System.User; describe entity String; describe entity System.FileDocument\"` stops at statement 2 with `module name is required: objects must be created within a module` and the third statement is silently never run", "cause": "Root Run tested `commands != \"\"` to choose -c over the REPL, so an empty flag value was indistinguishable from no flag; the -c path used ExecuteProgram, which returns the first error without its position, and describe entity/association reached findModule(\"\"), whose message is written for the create path", "fix": "`cmd.Flags().Changed(\"command\")` selects the one-liner path; runCommandLine (cmd/mxcli/oneliner.go) refuses empty input, reports `statement N of M` and how many later statements were not run (via new Executor.ExecuteProgramReportingStop), and takes --continue-on-error like exec; execDescribe names an unqualified entity/association name", "insight": "A flag's zero value is not its absence: use Changed() whenever an empty value must mean something other than not given. Decided semantics: -c is fail-fast like exec (a later statement may depend on an earlier one), but a stop is never silent", "issue": "mendixlabs/mxcli#1218", "file": "cmd/mxcli/oneliner.go; cmd/mxcli/main.go; mdl/executor/executor.go; mdl/executor/executor_query.go", "test": "cmd/mxcli/oneliner_test.go; mdl/executor/describe_unqualified_name_test.go"} +{"date": "2026-10-01", "area": "cmd/mxcli", "symptom": "`mxcli lsp --stdio` exits rc=2 with `panic: only file URIs are supported, got mendix-mdl` on textDocument/didOpen of a `mendix-mdl:` virtual document (the VS Code extension's describe previews); VS Code restarts it, it crashes again, and after 5 crashes it stops restarting the server", "cause": "checkableDocument (diagnostics) and CodeAction called go.lsp.dev/uri URI.Filename(), which panics on any scheme but file, to decide whether the document is a .test.mdl", "fix": "documentPath(uri) returns Filename() only for file: URIs and the URI's path component otherwise; runSemanticCheck (which shells out `mxcli check `) skips non-file documents; virtual documents are still diagnosed in memory", "insight": "Third-party helpers that panic on unexpected input are a crash path in a long-running server; every URI an LSP client sends is untrusted shape. A test with a non-file URI plus a file-URI control (same diagnostics) proves the virtual case is handled, not skipped", "issue": "mendixlabs/mxcli#1245", "file": "cmd/mxcli/lsp_helpers.go (documentPath, isFileURI); cmd/mxcli/lsp_diagnostics.go; cmd/mxcli/lsp_language.go", "test": "cmd/mxcli/lsp_virtual_uri_test.go"} diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index b42032884a..17f1c217bc 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -149,3 +149,10 @@ {"area": "mdl/backend", "date": "2026-09-29", "symptom": "describe -> exec, CREATE OR MODIFY or an ALTER that rebuilds the document turns an API-exported document Hidden: enumerations, pages, layouts, rules, view-entity OQL source documents, import/export mappings, JSON structures, published and consumed REST services, scheduled events, workflows, database connections, business event services, data transformers, queues, regular expressions and agent-editor documents. The run reports success, mx check is clean; the module's public surface silently shrinks. A workflow's own `export level API` clause was a no-op on create and on rewrite.", "cause": "Each rewrite converter builds a fresh document and writes ExportLevel as a constant (\"Hidden\"), or passes the semantic model's value where the executor itself filled in \"Hidden\" (mappings, database connection, business events), and the unit is replaced wholesale. MDL has no export-level spelling for most of these kinds, so describe cannot print it and the executed script cannot restore it. workflowToGen ignored wf.ExportLevel entirely. The round-trip harness could not see it: every document in TestApp and PedApp is Hidden, the constant itself.", "file": "mdl/backend/modelsdk/export_level_carry.go", "fix": "One byte-level carry, keepStoredExportLevel(unitID, contents): replaces only the top-level ExportLevel element of the freshly encoded rewrite with the stored value, copying every other element verbatim, and never adds the key. Wired into every Update path that writes ExportLevel (UpdateEnumeration/Rule/Layout/ImportMapping/ExportMapping/JsonStructure/PublishedRestService/ConsumedRestService/DataTransformer/DatabaseConnection/BusinessEventService, writeCustomBlob update, WriteViewEntitySourceDocument update; page via carryStoredPageHeader). Kinds with an MDL spelling (workflow, scheduled event, queue, regular expression) use keepStoredExportLevelUnlessSet: an authored level wins. workflowToGen now writes orDefault(wf.ExportLevel, \"Hidden\").", "insight": "A fixture-driven round trip is blind to any constant that happens to equal every fixture value: 775 TestApp documents round-tripped while 10 kinds hid API documents. Set the subject to the non-default value first (here: patch ExportLevel to API on the working copy) and run both the plain describe output and an edited one, because an elided unchanged write passes a converter that still writes the constant. Carrying at the encoded-bytes level covers gen-typed, newElem-built and hand-serialized writers with one helper, where a gen setter per converter would have needed three mechanisms.", "refs": ["ako/mxcli#816", "ako/mxcli#801", "ako/mxcli#812"], "test": "mdl/backend/modelsdk/issue816_export_level_test.go (TestUpdatePaths_KeepStoredExportLevel, 18 kinds); mdl/roundtrip/export_level_test.go (TestTestAppExportLevelSurvivesRoundTrip, -tags integration)"} {"date": "2026-09-30", "area": "mdl/backend", "symptom": "ako/mxcli#859 review of PR #864: after the built comparison landed, changing or adding `show page M.P with title = 'X'` in a `create or modify microflow` reported \"Unchanged microflow\" and wrote nothing, under mdl 0 and mdl 1 (main spliced it). Nothing warned.", "cause": "builtAsStored compares the declared flow and the stored flow both READ BACK through the codec, so any property the reader drops compares equal whatever either side holds. The ShowFormAction reader never read FormSettings.TitleOverride. Probing encode(built) against encode(readback(built)) over mdl-examples found the reader also dropped ExclusiveSplit/LoopedActivity ErrorHandlingType and a REST call's bound output variable (ResultHandling.ResultVariableName -> RestCallAction.OutputVariable), plus CallWebServiceAction (#861). Before the built comparison such a loss was a visible phantom re-splice; after it, a silently dropped edit.", "fix": "ReadBackMicroflow/ReadBackNanoflow re-encode what they read back and refuse (error -> statement diff, the pre-#859 path) when it is not the document first written, $IDs aside (sameWritten). The reader now reads TitleOverride, the split's and loop's ErrorHandlingType, and a bound REST call's OutputVariable, so those flows keep matching.", "insight": "A comparison made on both sides through the same lossy reader cannot see what the reader loses; the lost property becomes a change that is never written. When equality is decided after a decode, prove the decode lossless for the value at hand (write it again and compare bytes) and fall back when it is not. The probe that found the fields: diff encode(x) with encode(decode(encode(x))) over every mdl-examples flow.", "issue": "ako/mxcli#859", "file": "mdl/backend/modelsdk/microflow_readback.go, mdl/backend/modelsdk/microflow_read_actions.go, mdl/backend/modelsdk/microflow.go, mdl/roundtrip/flow_idempotent_shapes_test.go"} {"date": "2026-09-30", "area": "mdl/backend", "symptom": "ako/mxcli#843 (rehearsal M2): under mdl 1, `create or modify nanoflow … returns Boolean as $Done` over a nanoflow stored without a return variable refuses \"the stored document has no ReturnVariableName property … set it in Studio Pro\"; the same statement on a microflow reports \"set: ReturnVariableName\".", "cause": "mfmutator.SetHeader refuses any stated header key the stored document lacks (a key the project version does not declare makes the document unopenable). mxcli's nanoflow writer omits ReturnVariableName when the statement has no `as $Var`, while the microflow writer always writes it on 10+, so only nanoflows hit the refusal.", "fix": "Optional mfmutator.PropertyDeclarer on Deps; the codec deps answer from the metamodel version data (type, then Microflows$MicroflowBase; ReturnVariableName is 10.12+) against the project version, and SetHeader inserts the key after its predecessor in the encoder's order. No answer (MCP, unknown version) keeps the refusal.", "insight": "A refusal keyed on 'the stored document lacks the key' conflates 'this version has no such property' with 'the writer left it out'; the metamodel version data separates the two. Studio Pro 11 stores ReturnVariableName on every nanoflow (PedApp: 13 of 13), so adding it matches what Studio Pro writes.", "issue": "ako/mxcli#843", "file": "mdl/backend/mfmutator/header.go"} +{"date": "2026-10-01", "area": "mdl/backend", "symptom": "mendixlabs/mxcli#1214: `alter page … { set Editable = true on lvRows }` printed \"Altered page\" and the list view stayed Editable = false, so its inputs rendered read-only.", "cause": "pagemutator's `editable` case wrote the value only when it was a string (the input-widget Always/Never enum). A list view's and data view's Editable is a BOOLEAN, so `true` fell through and returned nil — success with nothing stored. A lower-case string ('never') was stored verbatim.", "fix": "setWidgetEditableMut: the stored value's type decides the vocabulary (bool → true/false; string → canonical Always/Never, Conditional only via [expr]); anything else, or a widget with no Editable, is refused. EditableIf now also sets the Conditional enum, as CREATE does.", "insight": "A setter switch that type-asserts and returns nil on a miss is a silent no-op generator; read the STORED field's type first, it says which vocabulary the property has on this widget.", "issue": "mendixlabs/mxcli#1214", "file": "mdl/backend/pagemutator/mutator.go"} +{"date": "2026-10-01", "area": "mdl/backend", "symptom": "ako/mxcli#528 / mendixlabs/mxcli#293: a data view footer could not be edited — `replace footer1 with {…}` (describe's name), the name the script wrote, and `dvMain.footer` all reported not found.", "cause": "A data view footer is a REGION: its widgets live in the data view's FooterWidgets and the footer has no stored Name. The builder hoisted the children and discarded the name; describe invented `footer1`.", "fix": "`.footer` resolves as a region in ResolveAlterTarget/ResolvesTarget; INSERT INTO appends (turning ShowFooter on for an empty footer), REPLACE swaps the content (a describe-style `footer { }` block is unwrapped), DROP empties it; a not-found names the footer addresses. A name on a data view footer is MDL-DEPR005; describe prints `footer { }`. REPLACE may also reuse the names of the widgets it removes (ContainedWidgetNames).", "insight": "Same shape as scroll-container regions: a slot without a name is addressed positionally by its owner. Check the grammar's own doc comment too — it advertised `replace footer1`, a form that never resolved.", "issue": "ako/mxcli#528", "file": "mdl/backend/pagemutator/footerregion.go"} +{"date": "2026-10-01", "area": "mdl/backend", "symptom": "mendixlabs/mxcli#1247: `alter page … replace comboBox12 with { combobox … }` to add a sort reset the combo box's translated placeholder and readOnlyStyle; on TestApp's Studio Pro-authored Rules.BusinessRule_NewEdit it also turned the stored Editable \"Never\" into \"Always\" and an expression property's PrimitiveValue \"\" into \"false\". exec reported success, mx check stayed green.", "cause": "A pluggable widget is rebuilt from its template plus the MDL mappings, so every property MDL cannot state comes back at the template value. #830's passthrough is all-or-nothing and only for unchanged widgets.", "fix": "For one pluggable widget replaced by one of the same package, the executor also builds the STORED widget as describe prints it (baseline); pagemutator.mergeUnstatedPluggable keeps the stored Type and every stored property/field on which replacement and baseline agree, and grafts the differing values with their TypePointers re-aimed at the stored Type by key path. Anything that cannot be lined up falls back to the plain replace.", "insight": "'What the statement states' is measured by building the description of the stored widget beside the statement, not by listing mapped keys: a property that comes out equal from both builds is the template's, not the author's.", "issue": "mendixlabs/mxcli#1247", "file": "mdl/backend/pagemutator/pluggable_replace.go"} +{"date": "2026-10-01", "area": "mdl/backend", "symptom": "mendixlabs/mxcli#1247 on a DataGrid 2: restating TestApp's Studio Pro-authored dataGrid2_1 with one column caption changed silently turned itemSelectionMethod \"rowClick\" into \"checkbox\", reset onClickTrigger and rebuilt every column's and filter's unmapped properties — the combo-box fix's merge did not apply and REPLACE fell back to the template rebuild.", "cause": "remapTypePointers re-aimed every TypePointer in the grafted columns value, including those of the nested filter widgets, which point into their OWN Type; no counterpart -> merge refused -> plain replace. And even when it applied, the columns list was grafted whole, so one changed caption reset every column's unstated properties.", "fix": "remapTypePointers leaves a nested CustomWidgets$CustomWidget as built; an object-list property whose objects line up one for one with the baseline's is merged object by object (properties indexed by their property type's path), so only the stated property of the stated column changes.", "insight": "A fallback that restores the old behaviour is the old bug: test the merge on the most nested pluggable widget available (DataGrid 2 with filters), not only on a flat one, and count the fallbacks.", "issue": "mendixlabs/mxcli#1247", "file": "mdl/backend/pagemutator/pluggable_replace.go"} +{"date": "2026-10-01", "area": "mdl/backend", "symptom": "mendixlabs/mxcli#698: over --mcp, `call microflow … on error rollback|continue` was created with PED's default error handling; a custom handler's error edge was written as an ordinary sequence flow. No error either way.", "cause": "mapMicroflowAction built every action map without errorHandlingType (only notify carried it), and buildFlowDocContent wrote every flow alike — PED's SequenceFlow constructor has no property for an error-handler flow (ped_get_schema, Studio Pro 11.14).", "file": "mdl/backend/mcp/microflow.go (mapObjectTree, buildFlowDocContent)", "fix": "carryErrorHandlingType writes the action's ErrorHandlingType (read by reflection); Custom/CustomWithoutRollBack and IsErrorHandler flows are refused with a message to run without --mcp.", "test": "TestMapObjectTree_CarriesErrorHandlingType, TestMapObjectTree_RefusesACustomErrorHandler", "insight": "ped_get_schema answers 'can PED express this' in one call; check it before mapping a property, and refuse what it cannot hold."} +{"date": "2026-10-01", "area": "mdl/backend", "symptom": "ako/mxcli#628: `move entity A.Parent to B; move entity A.Child to B;` both report success, then Studio Pro cannot open the project: System.AggregateException: The given key '' was not present in the dictionary.", "cause": "MoveEntity scanned only sourceDM.AssociationsItems(); an existing cross-association (created by the first move) was invisible, so moving its FROM entity left its ParentPointer naming an element absent from the unit, and moving its TO entity left Child naming the old module. Cross-unit damage, so the #1119 write guard cannot see it.", "fix": "association_move_cross.go: own cross-associations (FROM = moved entity) travel to the target, or become a plain DomainModels$Association there when the TO entity is already in it (raw transform: $Type, Child -> ChildPointer binary id, connection points added, GUID passed through); a cross-association in the target naming the entity becomes plain; any other module's cross-association naming it is re-pointed. MovedAssociation.SameModule lets the executor report the plain conversions apart.", "insight": "Assert on the stored documents (every ParentPointer/ChildPointer resolves in its own unit, every cross Child resolves to an entity in the named module); mx check is a weak signal for this class. The four sequences (to-then-from, from-then-to, travel, re-point) are distinct shapes, not one.", "issue": "ako/mxcli#628", "file": "mdl/backend/modelsdk/association_move_cross.go"} +{"date": "2026-10-01", "area": "mdl/backend", "symptom": "ako/mxcli#627: the fluent API's AttributeModifier.Apply() (Backend.UpdateAttribute) on a Studio Pro-authored attribute is refused with \"refusing to write unit …: 1 element(s) kept their $ID but would be written with a different GUID (DomainModels$Attribute)\"; without the #1119 guard it would re-mint the GUID and drop the column on the next deploy.", "cause": "UpdateAttribute rebuilds the attribute with attributeToGen (raw == nil) and carried only the $ID, so the codec's EmitGUID default wrote GUID = $ID. No MDL statement reaches it — every ALTER ENTITY form goes through UpdateEntity, which has the carry — so it was found by enumerating the converter's call sites, not by a repro.", "fix": "UpdateAttribute carries the stored attribute's raw bytes and export level onto the rebuild via carryStoredAttribute, the helper now shared with carryAttributeIdentity. Attribute -> Attribute keeps $Type, so SetRaw suffices.", "insight": "Test on PedApp (GUID != $ID); an mxcli-created attribute re-mints the same value and cannot fail. The unfixed code fails the test via the write guard's refusal, which is the observable symptom on main.", "issue": "ako/mxcli#627", "file": "mdl/backend/modelsdk/domainmodel_alter.go"} diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index db4ac13631..192cd2a632 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -791,3 +791,11 @@ {"date": "2026-10-01", "area": "mdl/executor", "symptom": "ako/mxcli#890 (rehearsal 2, R-rep): re-running a settled script wrote nothing (units_written=0) but still printed \"Granted access on ...\", \"Set project security level to ...\", \"Added module roles ... to user role ...\", \"Updated ... settings\" / \"Updated configuration ...\", and `move ... to folder` printed \"Moved ... to new location\"; the output could not serve as the #859 'second run reports Unchanged' gate. Also: a second `move` of the same document in one session failed \"microflow not found\".", "cause": "Those handlers printed their sentence with fmt.Fprintf after the backend call instead of going through ReportMutation's write-elision evidence (WriteStats offered vs written). Grants/revokes inside a program run are deferred (#872 accessRuleRun), so even ReportMutation could not see their write at statement time. The typed movers changed a container without invalidating the cached hierarchy.", "file": "mdl/executor/report_mutation.go, mdl/executor/access_rule_run.go, mdl/executor/cmd_security_write.go, mdl/executor/cmd_settings.go, mdl/executor/cmd_move.go", "fix": "ExecContext.reportWrite(unchanged, sentence...) prints the sentence or `Unchanged ` (through the run tally) on the ReportMutation evidence rule; used for project security level/demo users/strict mode/guest access, alter user role module roles, settings section/configuration/constant updates. Access-rule reports go through reportAccessRule: held on the open accessRuleRun and printed at its flush, Unchanged when the flush offered and elided (notices like 'No access rules found' print regardless). execMove short-circuits a document already in the target container (alreadyPlaced -> Unchanged) and invalidates the hierarchy after every move.", "test": "mdl/executor/noop_reporting_pedapp_test.go TestNoopRerun_ReportsUnchanged (PedApp, per statement: run 1 reports its write = control; run 2 writes no file and reports Unchanged, for grant, security level, demo users, strict mode, user role module roles, settings runtime, configuration (alter and create or modify), move) and TestNoopRerun_ProgramReportsUnchanged (program run incl. a revoke+grant reset; run 2 writes nothing and reports no write verb; run 1's net-nothing reset reports no write). Revert check: every case fails with the write sentence on run 2; the move case with 'microflow not found'.", "insight": "A report printed after a backend call is a claim about storage the handler cannot make on its own; route every write report through the write-stats evidence, and where writes are deferred, defer the report with them. The output only becomes an idempotency gate when no statement prints a write verb by construction."} {"date": "2026-10-01", "area": "mdl/executor", "symptom": "ako/mxcli#840 (mxcli-ledger, finding 162): `mxcli describe` (no header, no option) wrote mdl 0 spellings that mdl 1 refuses, and its microflow output held `$N = count($Hits)` / `$x = find($L, …)` — call forms registered as deprecated (MDL-DEPR003/004), so subcommand output was neither mdl 1-runnable nor mdl 0-clean. At the freeze a second instance surfaced on TestApp: describe of a chart series' text template wrote `staticTooltipHoverTextParams: [{1} = X]`, the bracketed form MDL-DEPR124 deprecates, under both versions.", "cause": "formatListOperation / the AggregateListAction case gated the statement form on describeLanguage >= V1 and fell back to the call form for mdl 0, although the statement form parses with the same meaning and no warning under mdl 0. The object-list describer (cmd_pages_describe_objectlist.go) wrote a TextTemplate's Params as \"[\" + … + \"]\". The roundtrip test that should have caught both (describeUsesCanonicalSpellings) filtered deprecations to an R8 allowlist ('describe keeps them under mdl 0'), so any code outside the list was invisible.", "file": "mdl/executor/cmd_microflows_format_action.go, mdl/executor/cmd_pages_describe_objectlist.go, mdl/roundtrip/describe_canonical_spelling_test.go", "fix": "Describe writes the List operation / Aggregate list statement in every language; only an activity the statement cannot express falls back to the call. Object-list template parameters are written in ( ). The canonical-spelling roundtrip test now checks EVERY registered deprecation under both describe languages (describeAs V1 and V0) on PedApp and TestApp, with no code filter. Freeze: langver.Frozen = V1, every describe output starts with `mdl 1;`, `--mdl 0|1` on describe/context/diff-local.", "test": "mdl/executor/cmd_microflows_format_list_activity_test.go TestDescribeListActivityUnderMdl1 (both versions, plus the call-form control warning under mdl 0); mdl/roundtrip/describe_canonical_spelling_test.go TestTestAppDescribeUsesCanonicalSpellings (failed on Snip_TaskDashboard_Numbers & 2 more with MDL-DEPR124 before the objectlist fix).", "insight": "A test that filters warnings to a list of 'codes this test owns' hides every code added later; a 'never emits X' property must check the whole registry, and in every output language the command can be asked for."} {"date": "2026-10-01", "area": "mdl/executor", "symptom": "ako/mxcli#905 part 1 (#897 item 4, rehearsal 3 G2): under `mdl 1`, `create or modify microflow` that grows a stored flow by an activity whose custom error handler ends in its own `return` (`$Ok = call microflow … on error begin log …; return; end error;`) was refused on every run, as an insert or a replace: \"an error handler in the fragment ends at an end event of its own … a return inside an error handler is not spliced yet\". Created fresh the statement worked; mdl 0 rebuilt (MDL-V1-REBUILD). On CapTrack the stub-then-real pair (13-actions stub, 30-export real ACT_Export_Excel) left the placeholder stored on every run with mx check clean. Once spliced, the second run of the real CapTrack script was refused: \"replace $Written: cannot replace the ActionActivity …: it has an error handler\".", "cause": "addErrorHandlerFlow (cmd_microflows_builder_flows.go) builds a handler body with a child flowBuilder and merged its objects and flows into the parent, but not its returnEndIDs, so cutFragment saw the handler's return end event as one the builder added and refused it. Second defect, exposed by the grow: a log/show message/validation feedback message written as an expression (`'failed for ' + $User/Name`) is stored as template '{1}' with the expression as parameter, and describe prints that form; declaredMatches compared the two spellings as different statements. Where builtAsStored is false (a spliced or Studio Pro-drawn flow), the statement diff then replaced the activity: absorbed by write elision on the main path, but refused when the message sits in a stored activity's error handler.", "file": "mdl/executor/cmd_microflows_builder_flows.go, mdl/executor/flow_declared_match.go", "fix": "addErrorHandlerFlow copies errBuilder.returnEndIDs into the parent's (lastReturnEndID untouched), so a handler's return is a new end event of the flow like a guard's (#888); placement/room checks (checkRoom/checkBranches) apply unchanged and refuse where the handler's return branch would cross a stored flow. matchValue normalises LogStmt/ShowMessageStmt/ValidationFeedbackStmt (messageAsTemplate) to the builder's stored form: a non-literal message becomes '{1}' with the expression as first parameter (a log stating its own `with (...)` keeps its message, as the builder does).", "test": "mdl/executor/cmd_alter_flow_handler_return_test.go TestCutFragment_HandlerReturnIsANewEndEvent; mdl/executor/flow_message_respelling_test.go (with controls); mdl/roundtrip/flow_splice_handler_return_test.go TestSpliceRerun_GrowByHandlerReturn (replace/insert under mdl 1, insert under mdl 0, verdict agreement check/diff/exec, twice-exec, a changed-flow control), TestSpliceRerun_GrowStudioProFlowByHandlerReturn (PedApp ShowPasswordForm, all stored IDs kept, description fixed point), TestSpliceRerun_HandlerReturnWithNoRoomIsRefused. Revert checks: returnEndIDs not copied -> every grow refused 'not spliced yet' (check predicts it); messageAsTemplate off -> second run refused 'replace $Ok … it has an error handler'. CapTrack copy: fmt --upgrade -p 13+30, exec 13, 30, 30 -> real flow stored, third run 0 units; mx check identical to baseline (0 errors). PedApp repros mx check identical to baseline.", "insight": "Child builders (error handler, loop) each keep builder state the parent's consumers rely on; when a new piece of builder state is added for the splice (returnEndIDs, #888), enumerate the child builders and decide per child whether it propagates. A grow test that only re-runs on an mxcli-authored flow can pass because builtAsStored short-circuits the statement diff; the respelling only surfaced on the real project and the Studio Pro-drawn flow, where the statement diff decides."} +{"date": "2026-10-01", "area": "mdl/executor", "symptom": "`describe enumeration` writes a caption with an apostrophe as `V1 'It's a test'`, which does not re-parse (`the statement ending at '' has no terminating ;`); a folder path with an apostrophe the same", "cause": "describeEnumeration hand-rolled `'%s'` for the caption and the folder clause instead of mdlQuote; cmd_enumerations.go was not in the list scanned by TestDescribers_HaveNoHandRolledStringLiterals because the file also holds many prose error messages containing `'%s'`", "fix": "Caption and folder go through mdlQuote(ctx, …), so the escape follows the describe language (mdl 1 doubles, mdl 0 also escapes backslash/line break)", "insight": "The #1006 source-scan guard only covers the files it lists; a describer living in a file dominated by validation prose escaped it. A re-parse test on the emitted statement, under both describe languages, is the guard that does not depend on a file list. Twice-exec checked on a PedApp copy: describe -> exec reports Unchanged and touches no file", "issue": "mendixlabs/mxcli#394", "file": "mdl/executor/cmd_enumerations.go (describeEnumeration)", "test": "mdl/executor/cmd_enumerations_caption_quote_test.go"} +{"date": "2026-10-01", "area": "mdl/executor", "symptom": "check -p passed and exec then refused (part-way, earlier statements written) or both accepted a broken write, in seven places: existing demo user, existing jar dependency, translations into the source language (ako/mxcli#906); a task queue created earlier in the script 'not found' for in queue (mendixlabs/mxcli#1211 = ako/mxcli#558); a page button calling a microflow created later (mendixlabs/mxcli#1212); a variable passed to a Microflow-typed Java action parameter, written as the reference text so the project no longer loaded (mendixlabs/mxcli#1210); referenceselector passing check, refused by the page builder with a misleading 'widget init' hint (ako/mxcli#563); retrieve constraints startsWith()/endsWith(), 'Assoc != empty', unknown bare members and CreatedDate for createdDate, CE0161 at build (mendixlabs/mxcli#1213).", "cause": "Each exec refusal was decided inside its handler (execCreateDemoUser's loop, applyJarDepAction, execCreateTranslations, buildQueueSettings, the page builder's resolveMicroflow / widget switch fall-through, addCallJavaActionAction) and check had no copy or a different copy: stmtCreateKind lacked the demo user, the scriptContext had no queues, eagerDefRefs had no widget flow refs, the whole-script reference pass counts names defined anywhere in the script (so a later create-or-modify looked resolved), MDL047's regex matched '=' only, and the retrieve walk never resolved members (the widget walk did, #1049).", "file": "mdl/executor/validate_exec_refusals.go, validate_duplicates.go, validate.go, validate_definition_order.go, javaaction_flow_param.go, validate_widget_unbuildable.go, validate_retrieve_members.go, validate_microflow.go", "fix": "Route check through exec's decision: demo user into the create registry (stmtCreateKind/setFor/stmtDropInfo); CheckExecRefusals replays alter module jar actions through applyJarDepAction on cloned settings and calls translationsRefusal (shared with exec); sc.queues; widgetFlowDefRefs + queueDefRef in eagerDefRefs (MDL-ORDER01) and validateForwardDefRefs for a later create-or-modify against the project; microflowParamArgRefusal used by the builder and the reference pass; formsWidgetsWithoutWriter used by MDL-WIDGET38 and the builder fall-through; MDL047 matches != too, MDL091 startsWith/endsWith (exec-enforced), validateRetrieveMembers reuses unresolvableXPathSteps on the stored constraint with a script-entity resolver.", "test": "mdl/executor/check_exec_agreement_pedapp_test.go (TestCheckExecAgree_*: each refusal predicted by check AND made by exec, each with a control both accept, on PedApp), validate_widget_unbuildable_test.go (structural: set vs builder switch cases). Each fix reverted: its test fails.", "insight": "A check/exec disagreement is a decision made twice; the cheap durable fix is to make check CALL exec's decision (replay applyJarDepAction on a copy, share translationsRefusal / microflowParamArgRefusal) rather than restate it. The whole-script name set is order-blind: any reference exec resolves eagerly needs an ordered pass, and a later create-or-modify can only be judged against the project. XPath shapes: measure each against a control on mxbuild before making a rule exec-enforced (startsWith, endsWith, != empty retrieve and list view, CreatedDate all CE0161 on 11.13.0; starts-with, Assoc/Target, createdDate clean). Bare createdDate is not an attribute in the model, so a member walk must exempt the implicit XPath members or it reports valid constraints."} +{"date": "2026-10-01", "area": "mdl/executor", "symptom": "mendixlabs/mxcli#1235: `checkbox cb (Attribute: $ShowAll)` with $ShowAll a page variable was refused as MDL-WIDGET34, and describe printed a Studio Pro input bound to a page variable with no binding, so describe → exec silently unbound it.", "cause": "Only AttributeRef (or a data-view SourceVariable.Widget) was read or written; Studio Pro stores a page-variable binding as SourceVariable{LocalVariable} with no AttributeRef (measured over MCP on TestApp, Pages.Probe1235_pagevar).", "fix": "Builders map a bare `$name` that is a declared page variable to SourceVariable{Kind: local}; describe prints `$name` for SourceVariable.LocalVariable with no AttributeRef; check validates `$name` against the document's Variables; ALTER sees the stored variables. Text box / text area get MaxLengthCode 0 (unlimited) there — -1 is mx check CE6553.", "insight": "Writing the new shape surfaced a second rule the shape implies (CE6553: no attribute length for a variable); run mx check on the written page, not just a describe round trip.", "issue": "mendixlabs/mxcli#1235", "file": "mdl/executor/cmd_pages_input_binding_context.go"} +{"date": "2026-10-01", "area": "mdl/executor", "symptom": "mendixlabs/mxcli#591: a nanoflow with `$L = create list of …`, `add $x to $L`, `remove …`, `$n = count $l`, `$h = head $l`, filter/sort or a cast passed check and exec, then mx check (11.14.0) reported CE6035 \"Error handling type is not supported\" at Create list / Change list / Aggregate list / List operation activity — one per activity. `call nanoflow … on error continue` in a nanoflow, and create/commit/call microflow with continue, rollback or a custom handler WITH rollback, failed the same way. `mxcli check` reported none of it: the nanoflow rules ran only inside exec's build.", "cause": "The list activities and cast had no ErrorHandlingType on their sdk structs, so the builder could not supply the flow flavour's default and both writers stamped a literal \"Rollback\" — CE6035 in a nanoflow, where Studio Pro stores \"Abort\" on every action (130 actions over ako/TestApp's 11 nanoflows) and \"Rollback\" on the same actions in a microflow. The per-clause rule for nanoflows covered six statements; create/commit/call nanoflow/call microflow accept ONLY a handler without rollback there (measured, one nanoflow per cell). ValidateNanoflow ran only MDL044, so check passed every nanoflow-body refusal exec's build makes.", "file": "sdk/microflows/microflows_actions.go; mdl/executor/cmd_microflows_builder_actions.go; mdl/backend/modelsdk/microflow_write.go + microflow_read_actions.go; mdl/executor/nanoflow_validation.go; mdl/executor/validate_nanoflow.go", "fix": "ErrorHandlingType on CreateList/ChangeList/ListOperation/Aggregate/Cast, set to fb.ehType(nil) (Abort in a nanoflow, Rollback in a microflow), written with orDefault(…,\"Rollback\") and read back so built-vs-stored compares it. nanoflowWithoutRollbackOnly refuses every other clause on create/commit/call nanoflow/call microflow in a nanoflow. ValidateNanoflow reports validateNanoflowBody's messages as MDL091 and runs the annotation rules.", "test": "TestListActions_ErrorHandlingFollowsTheFlowFlavour, TestCastAction_ErrorHandlingFollowsTheFlowFlavour, TestListActions_ErrorHandlingTypeRoundTrips, TestNanoflow_RefusesAllButWithoutRollbackOnCreateCommitAndCalls, TestValidateNanoflow_ReportsWhatExecWouldRefuse", "insight": "A hardcoded writer literal is a default for ONE flow flavour. The measurement that settled it was a one-nanoflow-per-activity script built with mx check, and the Studio Pro survey (Abort everywhere in nanoflows) told which value is right. Survey stored values per flavour BEFORE trusting a literal; and a rule that exec's build enforces must have a check-side twin, or check passes what exec refuses."} +{"date": "2026-10-01", "area": "mdl/executor", "symptom": "mendixlabs/mxcli#991: inside `on error … begin … end error`, @anchor and @curve parsed, passed check and exec, and changed nothing (the edges kept the default sides and 0;0 control vectors); DESCRIBE printed no @position/@anchor/@curve/@caption/@color/@excluded for a handler-body statement, so describe → exec put a laid-out handler back on auto-placement and reported success.", "cause": "addErrorHandlerFlow created the handler's edges with newErrorHandlerFlow/newHorizontalFlow and never applied the statements' anchors; a @curve was recorded on the child builder's curveByOrigin, which nothing applied. The describer's handler traversal (collectErrorHandlerStatementSpans) emitted only the notes. And the anchor emitter judged `to:` against the main path's default (left), while the error edge enters the TOP by default, so `to: left` on a handler's first statement read as default and was omitted.", "file": "mdl/executor/cmd_microflows_builder_flows.go (addErrorHandlerFlow); mdl/executor/cmd_microflows_show_helpers.go (collectErrorHandlerStatementSpans, emitAnchorAnnotationWithActivityMap)", "fix": "applyUserAnchors on the error edge (destination = first statement's to:) and between handler statements; the tail edge takes only the last statement's from: (originOnly); the child builder's curves merge into the parent's. The handler traversal calls emitObjectAnnotations; an activity whose only incoming flow is an error edge is judged against top.", "test": "TestErrorHandlerBody_HonoursAnchorAndCurve, TestErrorHandlerBody_TailEdgeTakesTheLastStatementsFromAndCurve, TestErrorHandlerBody_DescribeKeepsItsLayout, TestValidateMicroflow_ChecksAnnotationsInsideAHandler", "insight": "The error-handler body is a second, smaller builder AND a second, smaller describer; every annotation feature added to the main path has to be checked against both. A describe fixed-point test on a hand-placed handler is what exposes the default-side mismatch."} +{"date": "2026-10-01", "area": "mdl/executor", "symptom": "mendixlabs/mxcli#870: `lock workflow all;` / `unlock workflow all;` passed check and exec and failed mx check with CE1825 \"The 'Workflow' property is required\". Worse, DESCRIBE printed the Studio Pro activity WorkflowCommons.ACT_WorkflowDefinition_Lock (PauseAllWorkflows true WITH a $WorkflowDefinition selection) as `lock workflow all;` — describe → exec dropped the workflow and wrote the unbuildable form. `lock workflow Mod.Wf;`, which describe printed for a name selection, did not parse.", "cause": "MDL read PauseAllWorkflows as \"all workflows\"; it is Studio Pro's \"Pause instances\" checkbox ON the selected definition (Unlock: \"Unpause instances\", ResumeAllPausedWorkflows), on by default — the metamodel has no all-definitions selection. The writer omitted the selection whenever the flag was set.", "file": "mdl/grammar/domains/MDLMicroflow.g4 (lock/unlockWorkflowStatement); mdl/visitor/visitor_microflow_workflow.go; mdl/executor/cmd_microflows_format_action.go; mdl/backend/modelsdk/microflow_workflow_write.go; mdl/executor/validate_microflow.go", "fix": "`lock workflow $WfDef|Mod.Wf [pause all]` / `unlock workflow … [unpause all]`; the writer always writes a named selection; describe prints the selection plus the flag; a bare `all` is refused at check and exec (MDL-WF17).", "test": "TestLockWorkflow_PauseInstancesIsAFlagOnASelection, TestLockWorkflow_DescribeKeepsTheSelection, TestLockWorkflow_AllWithoutAWorkflowIsRefused, TestLockWorkflow_FlagKeepsTheSelection", "insight": "The Studio Pro data point the issue asked for was already in ako/TestApp (WorkflowCommons stores both branches of the checkbox). Look for a Studio Pro-authored instance in the fixtures before designing syntax from the metamodel's property name."} +{"date": "2026-10-01", "area": "mdl/executor", "symptom": "mendixlabs/mxcli#175: `call workflow … on error continue` passed check and exec, then mx check: CE6035 at Call workflow activity. On 11.14.0 only continue fails: no clause, rollback and both custom handlers build.", "cause": "continueUnsupportedOn did not list call workflow, and adapters.StatementErrorHandling — a hand-kept list of statements carrying a clause — did not list CallWorkflowStmt (nor the REST/other workflow statements), so MDL076 could not even see the clause. MDL076 was also check-only: exec without the pre-check (-c, REPL, --no-check) wrote it.", "file": "mdl/executor/validate_microflow_error_handling.go; mdl/exprcheck/adapters/adapter_scope.go; mdl/executor/validate.go", "fix": "call workflow in continueUnsupportedOn; StatementErrorHandling falls back to reading any statement's ErrorHandling field by reflection (getErrorHandling delegates to it); MDL076 is exec-enforced.", "test": "TestMDL076_ReportsContinueOnCallWorkflow, TestMDL076_CallWorkflowAcceptsTheOtherClauses, TestMDL076_IsExecEnforced", "insight": "Three lists of 'statements with an ON ERROR clause' existed; each had drifted. A rule keyed on such a list silently does nothing for a statement the list forgot."} +{"date": "2026-10-01", "area": "mdl/executor", "symptom": "ako/mxcli#571 / mendixlabs/mxcli#1206: `create published rest service` wrote only the path's {name} placeholders as operation parameters, each a String: every query and body microflow parameter failed mx check with CE0350, an Integer {id} with CE6539. `import mapping` / `export mapping` / `commit` on an operation parsed and were thrown away (CE0350 on the body, CE0354 on an object-returning microflow). Executing describe of a Studio Pro service (TestApp Services.OrdersRestApi) reported 'Modified' and broke a 0-error app with 5 errors: mappings cleared, Integer path params retyped String, body param dropped, Commit No->Yes, Basic+Session authentication turned off", "cause": "publishedRestOperationToGen built parameters from the path alone and wrote ExportMapping/ImportMapping \"\" and Commit \"Yes\" as constants; the reader never read parameters, mappings or commit, so describe could not print them and ALTER (which rewrites every operation) lost them too; the service writer also emitted constants for AuthenticationTypes / AuthenticationMicroflow / CorsConfiguration / Documentation / PublicDocumentation with no carry; Resources and operation Parameters were registered with list marker 2 where Studio Pro writes 3", "file": "mdl/executor/cmd_published_rest.go, mdl/backend/modelsdk/published_rest_write.go, mdl/backend/modelsdk/integration_read.go, mdl/backend/modelsdk/export_level_carry.go, mdl/visitor/visitor_rest.go, model/types.go", "fix": "the executor derives operation parameters from the microflow as Studio Pro does (path name -> Path, object/list -> Body, System.HttpRequest/HttpResponse -> none, else Query; the microflow parameter's type), merged over the stored parameters per bound microflow parameter so a header/renamed/described parameter survives; mappings and commit flow AST -> model -> BSON and back, describe prints them (commit when not Yes) and notes parameters MDL cannot state; an unknown commit value is refused at exec and by check (MDL-REST03); create or modify carries summary/documentation/object handling of the restated operation; UpdatePublishedRestService carries the stored service-level keys MDL cannot state (keepStoredTopLevel); list markers measured from TestApp", "test": "mdl/executor/cmd_published_rest_params_test.go; mdl/backend/modelsdk/published_rest_write_test.go TestCreatePublishedRestService_WritesParametersAndBindings, TestWithStoredTopLevel; mdl/roundtrip TestTestAppRoundTrip/published_rest_service_Services.OrdersRestApi (allowlist entry struck); mdl-examples/bug-tests/571-published-rest-parameters-and-mappings.mdl (TestApp copy: old binary 16 mx check errors, fixed 0, describe->exec Unchanged twice)", "insight": "A clause that parses and is then ignored is worse than a parse error: the grammar advertised import/export mapping for months while the writer hard-coded them empty. The fastest witness was the round-trip harness's own allowlist entry for the one Studio Pro published REST service in TestApp: removing it printed the whole loss set (bindings, parameter types, markers, authentication) in one diff. Studio Pro's metamodel (ped_get_schema over the MCP tunnel) gave the enum values and defaults: Commit defaults to No there, while mxcli keeps writing Yes when the clause is absent so existing scripts do not churn, and describe prints commit whenever it is not Yes."} diff --git a/.claude/skills/fix-issue/findings/mdl-grammar.jsonl b/.claude/skills/fix-issue/findings/mdl-grammar.jsonl index 883563ccef..d8f8d9fcfd 100644 --- a/.claude/skills/fix-issue/findings/mdl-grammar.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-grammar.jsonl @@ -67,3 +67,4 @@ {"area": "mdl/grammar", "date": "2026-09-28", "symptom": "After `ai` and `sample` became lexer keywords (R10), `alter enumeration M.E drop value Sample;` (and add/rename/modify value) failed with `mismatched input 'Sample' expecting IDENTIFIER`, though `create enumeration M.E ( Sample 'Sample' )` still parsed.", "cause": "alterEnumerationAction named values with a bare IDENTIFIER while create used enumValueName (IDENTIFIER | QUOTED_IDENTIFIER | keyword); every new lexer keyword silently shrinks the set of value names alter can reach. Model and older keywords were already unreachable.", "file": "mdl/grammar/domains/MDLDomainModel.g4, mdl/visitor/visitor_enumeration.go", "insight": "alterEnumerationAction now takes enumValueName and the visitor unquotes it. When a PR adds a lexer token, grep the grammar for bare IDENTIFIER in name positions (not identifierOrKeyword / a *Name rule) and try the new word there; the round-trip apps only catch it if they happen to use the word. Test TestAlterEnumerationValueNamedLikeAKeyword (fails on the IDENTIFIER-only rule). ako/mxcli#755 review.", "refs": ["#755", "#785"]} {"area": "mdl/grammar", "date": "2026-09-30", "symptom": "`create or modify constant M.DbPassword type string default '' PRIVATE;` -> `line 22:80 no viable alternative at input 'PRIVATE'` after upgrading past v0.24.0; `fmt --upgrade` failed with the same error because it must parse first. 9 credential constants in mxcli-formula1; the database-connections skill had taught the word.", "cause": "No grammar rule ever had PRIVATE. In v0.24.0 (and nightly 8c227f46) `helpStatement: IDENTIFIER (helpTopicWord …)*` was a catch-all and `;` was optional, so `… default '' private;` parsed as TWO statements: the constant, then a help statement `private` that the visitor built nothing for (`check` said `Syntax OK (2 statements)`). R7's IsHelpWord predicate (#755, bf0d1d38; the gated b518ae72 was reverted by dd977eea) closed the catch-all, turning the silent no-op into a parse error with no registry entry.", "file": "`mdl/grammar/domains/MDLDomainModel.g4` (constantOption `{IsPrivateWord(...)}? IDENTIFIER /* @alias MDL-DEPR138 */`), `mdl/grammar/MDLParser.g4` (IsPrivateWord), `mdl/deprecation/deprecation.go` (ConstantPrivate, RemovedIn 1), `mdl/visitor/visitor_enumeration.go` (record + delete fix), `mdl/visitor/visitor_deprecations.go` (recordDeprecation refuses at RemovedIn), tests `mdl/visitor/visitor_constant_private_test.go`, `mdl/upgrade/constant_private_test.go`, example `mdl-examples/deprecated-aliases/bug-tests--865-constant-private.mdl`", "insight": "To find what an old catch-all silently accepted, do not diff grammars — the word was in neither. Dump the OLD parse tree (`ToStringTree`) for the failing line; it showed `(helpStatement private)`. Then audit the class by walking every error-free old parse of a corpus for helpStatement nodes whose word is not help/exit/quit: over the repo's skills/docs/examples at 8c227f46 and all of mxcli-formula1, the only such word in a clean parse was PRIVATE (after a constant), so a targeted alias is the whole fix, not re-opening the catch-all. Match the word by a semantic predicate on IDENTIFIER, not a new keyword, so `private` stays usable as a name. Divergence to know: the old catch-all also swallowed words AFTER private (`private exposed to client` dropped the exposure); the alias does not.", "refs": ["ako/mxcli#865", "ako/mxcli#755", "ako/mxcli#714"]} {"area": "mdl/grammar", "date": "2026-10-01", "symptom": "`fmt --upgrade --header` refused `$Ordered = sort($Rows, Position);` with `MDL-V1-LIST: the operand is not a variable (a nested call or an expression)` and left the whole file at mdl 0 (2 mxcli-formula1 files); any keyword attribute (Position, Status, Type, Date, Value, Title, Caption, Content, Index) did it, and `sort($L, Status desc)` did not parse at all.", "cause": "The call form's `sortSpec` took only `IDENTIFIER | QUOTED_IDENTIFIER`, while the statement form's `listSortItem` took `identifierOrKeyword`. With a keyword attribute the listOperationStatement alternative failed and the line fell through to setStatement as a generic function call; mdl 0 still built the sort from it (buildListOrAggregateStatement), but the upgrade fix (setCallFix -> singleListCall) found no ListOperationContext and reported a nested call.", "file": "`mdl/grammar/domains/MDLMicroflow.g4` (sortSpec: identifierOrKeyword (ASC|DESC)?), `mdl/visitor/visitor_microflow_actions.go` (buildSortSpecList), `mdl/visitor/visitor_microflow_expression.go` (sort spec args); tests `mdl/upgrade/gated_test.go` TestUpgrade_KeywordAttributeNames, `mdl/visitor/visitor_microflow_sort_quoted_test.go` TestUnquotedKeywordSortAttribute", "insight": "A misleading upgrade reason ('not a variable') was a grammar asymmetry between a call form and its statement form: when one rule falls through to the generic expression path, the upgrader sees a function call and loses the structure. Compare the two forms' operand rules side by side. Proven with a control binary: `check` over 741 example/doc/skill scripts identical, and `fmt --upgrade --header` over 172 rehearsal scripts differs in exactly the two formula1 files. find/filter by member and sum/min/max over `$L.Keyword` were already fine.", "refs": ["ako/mxcli#889", "ako/mxcli#714"]} +{"date": "2026-10-01", "area": "mdl/grammar", "symptom": "mendixlabs/mxcli#992: `@anchor(true: (to: top))` — the per-case form DESCRIBE emits, with one side — parsed, passed check and exec, and left the true edge on the default sides. `@anchor(true: (from: right, to: top))` worked. `@curve(true: …)` on a NANOFLOW passed check and exec and was dropped (on a microflow MDL060 refused it).", "cause": "annotationParam listed `annotationValue | annotationParenValue`; `(to: top)` also matches annotationValue's expression alternative, as `to : top` — `:` is Mendix's division operator — and ANTLR took the first alternative, so the nested-anchor reader found no paren value. The visitor then skipped anything it could not use. Nanoflows never ran the annotation rules (ValidateNanoflow ran only MDL044).", "file": "mdl/grammar/domains/MDLSettings.g4 (annotationParam); mdl/visitor/visitor_microflow_statements.go (parseAnchorAnnotation)", "fix": "annotationParenValue before annotationValue; parseAnchorAnnotation records every parameter it cannot use in InvalidAnchors, refused as MDL092 at check and exec; ValidateNanoflow runs checkUnknownAnnotations over the whole body.", "test": "TestAnchorAnnotation_SplitBranchWithOneSide, TestAnchorAnnotation_RecordsWhatItCannotUse, TestSplitBranch_OneSidedAnchorReachesTheEdge, TestAnchorParameterItCannotUseIsRefused", "insight": "A two-key test case hid the one-key failure: `(from: x, to: y)` cannot be an expression (comma), `(to: y)` can. Dump the parse tree (ToStringTree) for the exact failing text before reading the visitor."} diff --git a/.claude/skills/fix-issue/findings/mdl-other.jsonl b/.claude/skills/fix-issue/findings/mdl-other.jsonl index 2414fe3408..9927f38452 100644 --- a/.claude/skills/fix-issue/findings/mdl-other.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-other.jsonl @@ -76,3 +76,5 @@ {"area": "mdl/catalog", "date": "2026-09-26", "symptom": "`impact Module.Entity.Attr` answered \"(no impact - element is not referenced)\" for an attribute a change activity sets and a page displays (Evora Factory Management: DigitalTwin.Machine.NumberOfIncidents); `impact` on an enumeration said the same; `callers` of a workflow started by a microflow said \"(no callers found)\". An agent auditing the app read these as safe-to-delete.", "cause": "The refs graph stopped at documents: no ATTRIBUTE / ENUMERATION / ENUMERATION_VALUE targets at all, and no edge for WorkflowCallAction, for a page navigating an association, or for an import/export mapping mapping an entity. Page/snippet XPath constraints also had no TargetEntity, because resolveEntityRefFromBSON read EntityRef.QualifiedName, a key no stored DirectEntityRef carries (it is `Entity`; an IndirectEntityRef ends on its last step's DestinationEntity).", "file": "mdl/catalog/builder_member_refs.go (memberRefsInUnit, scanPaths, extractXPathRefs, extractEnumerationTypeRefs), builder_references.go (microflowActionRef WorkflowCallAction), builder_xpath.go (resolveEntityRefFromBSON), catalogdb.go (CatalogTx.Query)", "insight": "Member references are found by a RAW-document walk matching every string value against the names the model declares (whole-string = structured ref: MemberChange.Attribute, AttributeRef.Attribute, EntityRefStep.Association, EnumerationType.Enumeration, ObjectMappingElement.Entity; path tokens inside expressions = association paths and qualified enum values). A typed walk would reach only the sites someone wrote a case for; the raw walk reached 4043 attribute bindings on Evora with no per-type code, and exact-set matching means prose cannot produce an edge (Documentation is skipped anyway; the test's control is an unused attribute that must stay unreferenced). XPath is resolved separately because its context entity IS known: bare names resolve against the target entity and its generalizations, predicates after a path switch context, and an enum attribute compared to a literal names the value. What stays invisible is a bare member through a variable in an expression ($Order/Total) \u2014 so the executor must not say 'not referenced' (see the executor finding). New member kinds deliberately stay OUT of graphRefKinds and off graph_god_nodes' asset side: attributes are members, not assets, and reusing change/create/retrieve for them would have pulled every attribute into communities/centrality. Bump CatalogSchemaVersion for any new edge (refs are only written by REFRESH CATALOG FULL). Verified the lint output on Evora is byte-identical in counts before/after, so no rule changed verdicts silently."} {"area": "cmd/mxcli/syntax", "date": "2026-09-28", "symptom": "`mxcli syntax database-connection` documented `DROP DATABASE CONNECTION Module.Name;`, which was a parse error; `SHOW REFERENCES OF` in docs-site was never valid (the grammar is `REFERENCES TO`).", "cause": "The syntax reference and docs-site are hand-written and nothing parses their statements against the grammar; the drop was documented ahead of an implementation that never came.", "file": "cmd/mxcli/syntax/features_integration.go, mdl/grammar/MDLParser.g4, docs-site/src/tools/references-impact.md", "insight": "Implemented `drop database connection [if exists]` (and `drop validation rule`, `drop external entity`) and corrected the docs. `make check-skill-mdl` only checks fenced blocks it judges runnable; a documented statement inside prose or a Syntax template is not checked, so grep the syntax entries for each documented statement keyword when adding or removing a form. ako/mxcli#755.", "refs": ["#755"]} {"area": "mdl/upgrade", "date": "2026-09-30", "symptom": "Re-running an upgraded script against a flow an older mxcli stored: `create or modify microflow Ledger.ACT_ApplyRules: this change cannot be spliced into the stored flow: the Loop at (2960, 200) changes inside its body` on every run (6 flows in mxcli-ledger), and outside a loop the re-run silently turned event handlers on (`ACT_MoveRuleUp/Down` spliced).", "cause": "mendixlabs/mxcli#895 changed what a bare `commit $X;` means (without events -> with events, Studio Pro's default) in every version, not behind the header. The stored flows hold `WithEvents=false`; the unchanged script now declares true, so diff-then-patch sees a change inside the loop body. fmt --upgrade had nothing to rewrite: the script alone cannot know which meaning is stored.", "file": "`mdl/visitor/visitor_flow_commits.go` (ast.Program.FlowCommits, with the insertion Fix), `mdl/upgrade/commit_events.go` (pinCommitEvents, matched by variable), `mdl/executor/flow_commit_events.go` (StoredCommitEvents, loop bodies included), `cmd/mxcli/cmd_fmt.go` (-p opens the project with or without --header), tests `mdl/upgrade/commit_events_test.go`, `mdl/roundtrip/upgrade_commit_events_test.go`", "insight": "A default change that is NOT version-gated still breaks the migration: every stored flow built before it disagrees with its own script. The upgrade is the one place that can reconcile it, and only with the project, so `-p` must be read even when no header is added. Match script commits to stored Commit activities by variable, not position: explicit flags consume their stored twin, the bare ones are pinned only when every remaining stored commit of the variable is without events and none are left over; anything else is a note, never a guess. Control in the test: the unpinned script is refused (mdl 1) / rewrites the flow (mdl 0) against the same setup.", "refs": ["ako/mxcli#873", "ako/mxcli#714", "mendixlabs/mxcli#895"]} +{"date": "2026-10-01", "area": "mdl/scriptdiff", "symptom": "`mxcli diff` disagrees with exec: after exec has applied a script (a second exec writes nothing), diff still reports the entity modified — `Success: Boolean` against the stored `Success: Boolean default false`, `Body: String` against `String(unlimited)`, an `@Position` line, a re-laid-out flow as 'N statements added'; and it is silent where exec writes (pages, translations, a layout repointed — 'Not compared'), reports a plain `create` of an existing element as unchanged although exec stops on 'already exists' (#807), and gives 'Could not diff' for a flow that calls one the script creates first (#856). Rehearsal 3: phantom changes on 66 scripts, silent on 37 that wrote", "cause": "diff answered 'what would exec write?' a second way: it rendered each statement as MDL with its own per-statement converters and compared the text with the stored document's describe output, without running any statement. Each fix (#997 one flow renderer, #794 module name, #839 the splice verdict, the storage carry) synchronised one more spot of the second answer, and every spelling exec normalises but the converter did not (implicit defaults, positions) stayed a phantom", "file": "mdl/scriptdiff/scriptdiff.go, cmd/mxcli/cmd_diff.go, cmd/mxcli/exec_preflight.go", "fix": "Remove the second answer: diff copies the project to a scratch folder, runs the script there with exec's own code (the same pre-flight, ExecuteProgram / ContinueOnError, the script's header), and compares the copy with the project unit by unit (bytes plus the Unit-table container, so a move is seen); the changed units are rendered by DESCRIBE on both sides (a domain model per entity/association, module and project security per role/demo user). The statement-based differ (cmd_diff_mdl.go, cmd_diff_render.go, spliceVerdict) is deleted. The scratch run always uses the file engine, also under --mcp", "test": "mdl/scriptdiff/scriptdiff_test.go (PedApp: applied script diffs empty with a control that the first run adds the entity; plain create refused; earlier statements run; layout change shown; folder move reported); property_test.go -tags integration: over mdl-examples doctype scripts, diff's written-unit set equals exec's, twice; MXCLI_DIFF_LEGS runs the rehearsal legs", "insight": "A preview that predicts a side-effecting command should BE that command run somewhere harmless, not a model of it. Every normalisation exec applies (canonical defaults, builtAsStored, write elision, the splice verdict) has to be re-derived by a renderer-based differ, and the drift only shows on real projects. Copying a 56 MB project and executing costs well under a second; seven rounds of synchronising the renderer did not converge. Also: a write that changes no unit bytes (a move) lives in the .mpr Unit table, so the snapshot must include each unit's container", "refs": ["#907", "#807", "#856"]} +{"date": "2026-10-01", "area": "mdl/scriptdiff", "symptom": "`mxcli diff` of a headerless (mdl 0) script that starts with `connect local ''` wrote the script's changes into the real project and reported `exec would write nothing`; a connect to another project (or one inside an `execute script`) wrote that project; `sql ` and `import from` would run against real databases. Also: with TMPDIR inside the project folder the scratch copy copied itself until 'file name too long'", "cause": "diff executes the script for real on a scratch copy, but exec follows a CONNECT (allowed under mdl 0 with a MDL-V1-SESSION warning, and inside nested scripts under any header) and SQL/IMPORT act on databases, so 'run it on a copy' only holds for statements whose effects stay in the connected project. The snapshot compared was the copy's, which saw no write, so the report said nothing changed", "file": "mdl/scriptdiff/scriptdiff.go (outsideGuard), mdl/executor/executor.go (SetStatementGuard), mdl/scriptdiff/copy.go", "fix": "Executor.SetStatementGuard: a hook every statement passes through in Execute, nested EXECUTE SCRIPT included. diff's guard redirects a connect to the -p project onto the copy, and refuses a connect elsewhere, a SQL query and an IMPORT with an error (not as exec's verdict). copyProject skips the folder the copy is created in", "test": "mdl/scriptdiff/outside_test.go: connect to the project diffs onto the copy and agrees with exec, project untouched; connect to another project and a nested one refused, other project untouched; SQL query / import refused; scratch folder inside the project", "insight": "Running the real command on a copy is only a dry run for effects that stay inside the copy. Enumerate every statement that reaches outside the connected project (connect, SQL, import, nested scripts) and guard them at the dispatch point, not per top-level statement, or a nested script walks around it", "refs": ["#907", "#913"]} diff --git a/.claude/skills/fix-issue/findings/mdl-visitor.jsonl b/.claude/skills/fix-issue/findings/mdl-visitor.jsonl index 859d976098..cf8d6f572a 100644 --- a/.claude/skills/fix-issue/findings/mdl-visitor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-visitor.jsonl @@ -51,3 +51,5 @@ {"date": "2026-09-30", "area": "mdl/visitor", "symptom": "`fmt --upgrade --header` refused the header over `$Hit = find($Regions, $currentObject = $X)` when `$Regions` was assigned by `call microflow` (MDL-V1-LIST \"depends on a type the script does not state\"), although the called flow's return type settles it — in the same script (CapTrackV6 11-viewstate) or in the project (26-admin-goals)", "cause": "operandKind counted every `$x = call …` as an unknown definition: the visitor has no project, and never looked at the flows the script itself creates. The mdl 0 flow builder decides at exec time from `declaredVars[$x] == \"String\"`, which registerResultVariableType sets from the called flow's return type, looked up in the model as the earlier statements left it", "file": "mdl/visitor/visitor_upgrade_fixes.go (collectOperandDefs, operandReading, scriptFlowReturn), mdl/visitor/visitor_list_activities.go (both call sites: ExitListOperationStatement, setCallFix), mdl/ast/ast.go (LanguageNote.Operand / OperandChoice), mdl/upgrade/upgrade.go (Options.Flows, resolveOperand), mdl/executor/flow_return_types.go (FlowReturnTypes), cmd/mxcli/cmd_fmt.go (-p)", "fix": "Resolve in exec's order: a flow an earlier statement of the script creates (b.statements, the statements built so far) answers statically; otherwise the note carries both fixes as an OperandChoice and `fmt --upgrade -p` resolves it through executor.FlowReturnTypes, which calls the builder's own lookupMicroflowReturnType + registerResultVariableType. A drop/rename/move/`if not exists` of the callee earlier in the script, a callee the project lacks, and definitions that disagree stay refused", "insight": "Answer the upgrade's type question with the builder's resolver, not a copy (duplicate-resolver-drift). The execute-both control only discriminates in one direction: the builder turns a List operation over a declared String into the string function under EVERY version (addListOperationAction), so misreading a String as a list writes the same model; the dangerous misreading is a list read as a String (`set $x = find(…)` is always the string function under mdl 1). Choose the control from the direction that can hurt, or a green execute-both proves nothing. Verified on the CapTrackV6 rehearsal: 11-viewstate upgrades with no project, 26-admin-goals refuses without -p and upgrades with it, and its upgraded SUB_Admin_GoalScope execs as Unchanged on the project, same as the original", "issue": "ako/mxcli#860"} {"area": "mdl/visitor", "date": "2026-09-30", "symptom": "`LastImport: date,` in a headerless script (mxcli-ledger 01-domain-model.mdl) -> `type `date` is not supported \u2014 Mendix has no date-only type; mxcli silently stored it as DateTime` from fmt, check and exec alike; `fmt --upgrade` could not parse the file, so the migration needed a hand edit although `date` -> `DateTime` is exactly what every earlier build stored.", "cause": "#706 refused `date` together with `float`/`currency` in every version. The two cases differ: float/currency stored a String (a wrong model, refusal right), `date` stored the DateTime the author could only have meant \u2014 a respelling, which ADR-0011 makes a deprecated alias, not an error.", "file": "`mdl/visitor/visitor_silent_drops.go` (recordDateType), `mdl/visitor/visitor_helpers.go` + `visitor_microflow.go` (DATE_TYPE builds TypeDateTime), `mdl/grammar/domains/MDLDomainModel.g4` (`@alias MDL-DEPR160` on dataType and nonListDataType), `mdl/deprecation/deprecation.go` (DateType, RemovedIn 1), tests `mdl/visitor/visitor_date_alias_test.go`, `mdl/upgrade/date_alias_test.go`", "insight": "Sort a #706-style 'silently stored as X' refusal by what X was: if X is what the author meant (date -> DateTime), the form is an alias and belongs in the registry (warn, fmt rewrite, refuse under mdl 1); only when X is wrong (float -> String, dropped association options) is a refusal in every version right. For the alias to be provable, the builder must produce the canonical AST (TypeDateTime, not TypeDate), or the registry's Example/CanonicalExample same-AST test fails. Upgrading the ledger's pre-migration 01 now reproduces its hand-finished committed file byte for byte.", "refs": ["ako/mxcli#714", "ako/mxcli#706"]} {"date": "2026-10-01", "area": "mdl/visitor", "symptom": "ako/mxcli#877: mxcli-rest wrote `drop microflow if exists X;` between each flow's `/** … */` doc comment and its `create`; six flows lost their documentation. `mx check` passed, `exec` was clean, describe round-tripped, `mxcli check` said nothing; only lint QUAL002 (19 -> 22) noticed.", "cause": "The grammar's `statement: docComment? (…)` lets every statement start with a doc comment, but only the create exits read one (findDocComment); every other statement drops it silently. Nothing recorded that a comment was parsed and not stored.", "file": "mdl/visitor/visitor_doc_comment_placement.go, mdl/visitor/visitor_r2_children.go, mdl/visitor/visitor_strict_terminators.go, mdl/ast/ast.go, mdl/executor/validate_detached_doc_comments.go, mdl/executor/validate_program.go", "fix": "Enter/ExitStatement record, per top-level statement, a doc comment on a statement that does not store one (not a createStatement, or a create whose AST has no Documentation/OuterDocumentation/DocumentationSet field: create module, module/user role, demo user, …) as ast.Program.DetachedDocComments, with the next storing statement's head and line filled in when it is reached. ValidateProgram warns MDL089 naming that statement (check and exec, every version). The write-microflows pitfalls reference and check-syntax skill show drop-then-create with the comment between the drop and the create.", "test": "mdl/visitor TestDetachedDocComments (drop between comment and create, grant/revoke, create module; controls: comment above its create, entity and constant); mdl/executor TestValidateProgram_DetachedDocComment (message names the create and its line; control: kept comment not reported). Revert check: without the ExitStatement hook, 3 visitor subtests fail and the executor test reports 0 MDL089 warnings.", "insight": "A grammar that accepts a construct everywhere but a handful of exits read it is a silent-drop generator; record what was parsed and not consumed at the one rule that admits it, rather than auditing every reader. Surveying mdl-examples found the same loss in the repo's own scripts (section-header doc comments above grant/revoke/create module role)."} +{"date": "2026-10-01", "area": "mdl/visitor", "symptom": "`show features for version 10.24` (and `list features for version 10.24`) prints the session `show version` output (Mendix Version: 11.x ...) instead of the feature table; `show features` and `show features added since 10.24` work", "cause": "ExitShowStatement is one long else-if chain over token presence; the bare `ctx.VERSION() != nil` branch for `show version` sat ~380 lines above the FEATURES branch, and `for version` also carries the VERSION token, so the earlier, broader test won", "fix": "Guard the show-version branch with `ctx.FEATURES() == nil`", "insight": "In a token-presence else-if chain, a branch keyed on ONE token claims every alternative containing it; order the chain most-specific first or make each branch exclude the head keyword of the others. TestEveryShowAlternativeProducesAStatement cannot see this: it only checks that SOME statement is produced, not the right one", "issue": "ako/mxcli#910", "file": "mdl/visitor/visitor_query.go (ExitShowStatement)", "test": "mdl/visitor/show_features_for_version_test.go"} +{"date": "2026-10-01", "area": "mdl/visitor", "symptom": "(a) `grant … where '[\"Status\" = ''Accepted'']'` (and the mdl 1 bracketed form) stores `\"Status\"` verbatim: a string literal to Mendix, the rule is always false and grants no rows; check, exec, lint and mx check all pass, found only by an as-user runtime test. (b) `retrieve … where [Name = \"ServiceManager\"]` passes check and stores `Name = ServiceManager`, a bare member token instead of the string", "cause": "(a) The retrieve/datasource XPath sinks route their source through stripExpressionIdentifierQuotes (2026-07-08 finding); the grant (both spellings) and workflow targeting (both spellings, plus alter workflow) did not, and only a constraint the multi-line canonicaliser rewrote lost its quotes by accident. (b) stripExpressionIdentifierQuotes strips EVERY double-quoted token outside single quotes, so a value written with the wrong quotes was turned into a name", "fix": "bracketedXPathText and the quoted grant/targeting branches strip identifier quotes; a new listener check (ExitXpathConstraint for every bracketed XPath, plus the string-literal forms of retrieve, grant and targeting) refuses a double-quoted token that is the right-hand operand of a comparison, naming the single-quoted spelling", "insight": "Quote-stripping is right for names and wrong for values, and position tells them apart: Mendix XPath never compares two members, so a quoted token after =, !=, <, <=, >, >= is always a mis-quoted string. Refusing it fixes a silent wrong write, so it applies under mdl 0 and mdl 1 (ADR-0011). When a normalisation lives in N sinks, grep its callers (bracketedXPathText, unquoteStringLit on an XPath) — the reported statement was 1 of 6", "issue": "mendixlabs/mxcli#1243; ako/mxcli#566", "file": "mdl/visitor/visitor_xpath_quotes.go; mdl/visitor/visitor_xpath_brackets.go; mdl/visitor/visitor_security.go; mdl/visitor/visitor_workflow.go; mdl/visitor/visitor_alter_workflow.go; mdl/visitor/visitor_retrieve_range.go", "test": "mdl/visitor/xpath_double_quotes_test.go"} diff --git a/.claude/skills/fix-issue/findings/modelsdk.jsonl b/.claude/skills/fix-issue/findings/modelsdk.jsonl index e55dc06ee8..3cdc21513f 100644 --- a/.claude/skills/fix-issue/findings/modelsdk.jsonl +++ b/.claude/skills/fix-issue/findings/modelsdk.jsonl @@ -24,3 +24,4 @@ {"area":"modelsdk/mpr","date":"2026-09-25","symptom":"`alter page FeedbackModule.ShareFeedback_Logo { insert after textBox1 { image zzImg (ImageType: imageUrl, ImageUrl: '{1}', ImageUrlParams: [{1} = ImageB64]) } }` (data view over a nanoflow the project lacks, 11.13.0) reported \"Altered page\"; `mxcli docker check` then could not LOAD the project: ArgumentNullException setting 'Attribute' of an Attribute in a Page","cause":"The bare-AttributeRef refusal (#678) lived in encodePage/encodeSnippet only. ALTER PAGE patches the stored BSON in pagemutator and saves via UpdateRawUnit, never passing the encoder; with no entity in scope the pluggable-widget template-parameter builder (widgetobj) writes the name as given, so DomainModels$AttributeRef{Attribute:\"ImageB64\"} reached disk","file":"modelsdk/canon/attributeref.go; modelsdk/mpr/writer_core.go (updateUnit, insertUnit)","insight":"A guard placed in one encoder covers one write path; the page family has at least four (encodePage/Snippet, pagemutator Save, widget sync apply, layout/template raw writes). Put an unloadable-shape refusal at the writer beside DuplicateElementIDError, as that one already argued. Measured before refusing stored refs too: 73 of 73 AttributeRefs across all 374 units of a stock 11.13 project are qualified (71 page, 1 snippet, 1 page template, none elsewhere) — a stored bare one cannot have come from Studio Pro, so refusing ALL bare refs (not only new ones) blocks nothing legitimate. The textbox path does NOT reproduce it: attributeRefToGen nulls a bare name (a silent binding drop instead); the pluggable/column template builders are the ones that write it verbatim. The test goes through the real mutator + writer on the expr-checker fixture (InsertColumns with a bare CaptionParams ref).","refs":["#678"]} {"area": "modelsdk/widgets", "date": "2026-09-26", "symptom": "CE0463 \"The definition of this widget has changed\" on every page carrying a pluggable widget built from its .mpk whose action properties declare `` (Signature 2.1.0, Calendar 2.6.0 on 11.12.2) — even with no action configured. `mx update-widgets` clears it", "cause": "The .mpk parser had no field for `` (modelsdk/widgets/mpk/mpk.go xmlProperty/PropertyDef), and createDefaultValueType hardcoded `ActionVariables: [2]`; reconcileValueTypesFromMPK never touched the list. The typed gen class (CustomWidgets$WidgetActionVariable) existed but the map-based template pipeline never fed it", "file": "modelsdk/widgets/mpk/mpk.go (ActionVariable, toActionVariables), modelsdk/widgets/augment.go (buildActionVariablesArray, actionVariablesMatch, createDefaultValueType, reconcileValueTypesFromMPK); tests actionvariables_test.go in both packages; example mdl-examples/bug-tests/1200-mpk-action-variables.mdl", "insight": "Third instance of the same shape after #716 (onChange) and #956 (defaultType): a widget.xml attribute/element that is part of the DEFINITION, never parsed, written as its empty default. Cheapest audit: diff every key Studio Pro stores on a WidgetValueType in the embedded templates against what createDefaultValueType derives from the .mpk — the embedded combobox.json already held the correct ActionVariables entry, i.e. the oracle was in the repo. `sdk/widgets/augment.go` has no importers; the live BSON path is modelsdk/widgets. When reconciling a list from the .mpk, rewrite only on disagreement, or an agreeing template's entry $IDs churn — the Combobox augment test is the no-change control (it fails if the rewrite is unconditional).", "refs": ["mendixlabs/mxcli#1200", "mendixlabs/mxcli#956", "#716"], "ce": ["CE0463"]} {"area": "modelsdk/canon", "date": "2026-09-26", "symptom": "A page rewrite still loses translations despite CarryTranslations: an empty caption's en_US '' vanishes (Texts$Text with no items), and a label's nl_NL 'Gebruikers' is dropped because its English 'Account Overview' is also the page title's", "cause": "Positional pairing needs the whole document's text paths unchanged — one DataGrid2 rebuilt from its template breaks that. Source pairing keys on (language, text): (en_US, '') is ambiguous on any real page, a shared English source is ambiguous, and a rebuilt EMPTY text has no translation to look up by at all", "file": "modelsdk/canon/translations.go", "insight": "Address a text by the named element that owns it — ($Type, Name, path from the element) — because a widget Name is unique per document. Exact only where the path from the element crosses no list index (a rebuilt pluggable widget reorders Properties; pairing there moves a translation onto the wrong property) and where the address occurs once in each document. Order: positional when the shape is unchanged, then owning element, then source", "refs": ["ako/mxcli#705"]} +{"date": "2026-10-01", "area": "modelsdk", "symptom": "mendixlabs/mxcli#849: `mxcli exec` writes the .mpr while Studio Pro has the project open; the write succeeds, mx check passes, and Studio Pro's next save silently discards it. No warning, no error.", "cause": "No write path looked for Studio Pro at all; the rule 'close Studio Pro first' existed only in prose (README, docs-site, skills).", "fix": "modelsdk/mpr/studiopro_lock.go: Writer.guardWrite refuses every write that would reach storage (updateUnit and WriteTransaction.WriteUnit after no-op elision, insertUnit, deleteUnit, MoveUnit, UpdateUnitContainer) with StudioProOpenError while `.mpr.lock` (matched case-insensitively) is beside the .mpr; `exec --force` / MXCLI_ALLOW_STUDIO_PRO_OPEN=1 override. Reads and elided no-op writes are never refused.", "insight": "Studio Pro's own generated .gitignore is the evidence for the signal and its spelling: it lists `testapp.mpr.lock` (lower-cased) beside `TestApp.mpr`, plus `mprcontents/mprjournal*`. Guarding at the storage layer after elision, not at command level, keeps twice-exec a no-op instead of an error and covers every command that writes.", "issue": "mendixlabs/mxcli#849", "file": "modelsdk/mpr/studiopro_lock.go"} diff --git a/.claude/skills/mendix/alter-page/SKILL.md b/.claude/skills/mendix/alter-page/SKILL.md index 7c6a9ce61a..6a029c0583 100644 --- a/.claude/skills/mendix/alter-page/SKILL.md +++ b/.claude/skills/mendix/alter-page/SKILL.md @@ -167,6 +167,13 @@ rebuilds the widget from what the statement says, so any property you do not restate — `ButtonStyle`, `Class`, design properties, tooltip — is dropped. `set` edits the one property and leaves the rest of the widget alone. +The exception is a **pluggable widget replaced by one of the same kind** (a combo +box by a combo box): there `replace` keeps every stored property the statement +does not change — a translated placeholder, `readOnlyStyle`, anything MDL has no +word for — and writes only what differs from the widget as `describe` prints it. +So a sort can be added to a combo box's options by restating its `describe` +line with `sort by` appended. + `set Action` is refused on a widget that has no action (a plain container, say), rather than writing a property the widget type does not define — Studio Pro refuses to open a document with an unknown property while MxBuild tolerates it, @@ -313,16 +320,19 @@ Removes widgets and their entire subtree from the page. ### REPLACE - Replace Widget Subtree ```sql --- Replace a single widget with new content -replace footer1 with { - footer newFooter { - actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary) - actionbutton btnCancel (caption: 'Cancel', action: cancel changes) +mdl 1; +-- Replace a data view's footer (it has no name: address it by its data view) +alter page MyModule.Customer_Edit { + replace dvMain.footer with { + footer { + actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary) + actionbutton btnCancel (caption: 'Cancel', action: cancel changes) + } } -} +}; ``` -Replaces the target widget with one or more new widgets. The new widgets use the same syntax as `create page`. +Replaces the target widget with one or more new widgets. The new widgets use the same syntax as `create page`, and may reuse the names of the widgets the replace removes. `insert into dvMain.footer { … }` appends to a footer and `drop dvMain.footer` empties it. ### DataGrid Column Operations @@ -428,8 +438,8 @@ alter page MyModule.Customer_Edit { ```sql mdl 1; alter page MyModule.Customer_Edit { - replace footer1 with { - footer newFooter { + replace dvMain.footer with { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: success) actionbutton btnDelete (caption: 'Delete', action: delete, buttonstyle: danger) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) @@ -565,8 +575,8 @@ adds. Both still fail at exec if they are genuinely wrong. ## Limitations — prefer binding at page creation (ledger finding #45) -`ALTER PAGE` is best for *content* edits (add/remove/retitle widgets). Three things -it cannot do; when you hit them, define the referenced microflows **before** the +`ALTER PAGE` is best for *content* edits (add/remove/retitle widgets). When you hit +the limit below, define the referenced microflows **before** the page and bind the buttons at creation time instead of rewiring afterwards: 1. **`SET` cannot rewire a button's action.** `set` accepts a fixed property list @@ -574,17 +584,11 @@ page and bind the buttons at creation time instead of rewiring afterwards: `set Action = call microflow … on btnSave` is a parse error. Set the button's action when the button is created (or `REPLACE` the button subtree). -2. **`REPLACE` cannot reuse a widget name that lives inside the subtree being - replaced.** The replacement is *built* (registering its widget names) before the - old subtree is removed, so reusing e.g. `btnSave` collides with the still-present - old `btnSave` ("duplicate widget name 'btnSave'"). Give the replacement widgets - fresh names, or rebuild the whole page with `create or modify page`. - -3. **A footer is not addressable by its author-given name.** A `footer myName { … }` - is a *marker*: its children are hoisted into the data view's footer and the - footer itself is serialized as `footer1`, so `drop myName` (and even - `drop footer1`) report "not found". To change footer contents, edit the - children by their own names, or `create or modify page`. +**A data view's footer has no name.** Its widgets are stored in the data + view, and the footer itself is not, so a name written on it is never kept + (MDL-DEPR005) and `describe` prints `footer { … }`. Address it by its data + view: `replace dvMain.footer with { footer { … } }`, `insert into + dvMain.footer { … }`, `drop dvMain.footer`. **Recommended pattern**: put save/reset microflows in a file that runs *before* the page definition, and bind the popup/footer buttons to them at creation. The diff --git a/.claude/skills/mendix/check-syntax/SKILL.md b/.claude/skills/mendix/check-syntax/SKILL.md index 235a9f6acd..2a45fb7e01 100644 --- a/.claude/skills/mendix/check-syntax/SKILL.md +++ b/.claude/skills/mendix/check-syntax/SKILL.md @@ -625,7 +625,9 @@ does not hot-reload when an external process changes the file. So after `mxcli e - `ped_read_document` / `ped_check_errors` will show the **stale** pre-exec model until Studio Pro re-scans — call `refresh_project` first (or reload the project in the UI). - **Hazard:** if Studio Pro later saves on its own, it overwrites mxcli's disk write with - its in-memory copy, silently discarding your MDL changes. + its in-memory copy, silently discarding your MDL changes. So a file-based write is + **refused** while Studio Pro's `.mpr.lock` is beside the `.mpr`; `exec --force` + (or `MXCLI_ALLOW_STUDIO_PRO_OPEN=1`) overrides it, e.g. for a lock left by a crash. **Safest practice:** don't keep the same project open-and-saving in Studio Pro while mxcli writes it. Either close (or don't save in) Studio Pro during MDL authoring, or diff --git a/.claude/skills/mendix/choose-edit-mode/SKILL.md b/.claude/skills/mendix/choose-edit-mode/SKILL.md index 7c61bcf41f..bd0cc81640 100644 --- a/.claude/skills/mendix/choose-edit-mode/SKILL.md +++ b/.claude/skills/mendix/choose-edit-mode/SKILL.md @@ -26,7 +26,9 @@ MDL has two ways to change a document, and each is safe in a different situation Pro since). On a Studio Pro-authored document this path is lossy even when you change nothing. On real projects it has flipped association storage from table to column, dropped page translations, dropped nanoflow annotation links, changed export - levels, and dropped a snippet's type. `mxcli diff` shows none of these. + levels, and dropped a snippet's type. `mxcli diff` runs the script on a scratch + copy and lists every unit it would write, so run it first and expect "exec would + write nothing" for a document you did not mean to change. 4. **Never `drop` and re-create an existing document to change it.** The new document gets new identities. For an entity, the runtime then drops its table and its rows. diff --git a/.claude/skills/mendix/create-page/SKILL.md b/.claude/skills/mendix/create-page/SKILL.md index 412205db30..55de2c087b 100644 --- a/.claude/skills/mendix/create-page/SKILL.md +++ b/.claude/skills/mendix/create-page/SKILL.md @@ -58,6 +58,8 @@ Both are optional and can be changed later with `alter page … { set (Class: ' - Default value: Mendix expression in single quotes - Referenced in expressions as `$varName` - Use for DataGrid2 column `visible:` (which hides/shows entire column, NOT per-row) +- An input binds to one directly: `checkbox cbShowAll (Label: 'Show all', Attribute: $ShowAll)` + — no data view needed. `$name` must be declared in the page's `Variables:` ### Key Syntax Elements diff --git a/.claude/skills/mendix/create-page/reference/examples.md b/.claude/skills/mendix/create-page/reference/examples.md index 36786d7b2e..11f5b3256b 100644 --- a/.claude/skills/mendix/create-page/reference/examples.md +++ b/.claude/skills/mendix/create-page/reference/examples.md @@ -28,7 +28,7 @@ create or modify page CRM.CustomerEdit textbox txtPhone (label: 'Phone', attribute: Phone) checkbox cbActive (label: 'Active', attribute: IsActive) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } @@ -99,7 +99,7 @@ create page CRM.Customer_MasterDetail textbox txtEmail (label: 'Email', attribute: Email) textbox txtPhone (label: 'Phone', attribute: Phone) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } diff --git a/.claude/skills/mendix/create-page/reference/widgets.md b/.claude/skills/mendix/create-page/reference/widgets.md index be36334b43..cbf40567df 100644 --- a/.claude/skills/mendix/create-page/reference/widgets.md +++ b/.claude/skills/mendix/create-page/reference/widgets.md @@ -447,7 +447,7 @@ dataview dvName (datasource: $VariableName) { textbox txtName (label: 'Name', attribute: Name) textarea txtDescription (label: 'Description', attribute: description) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } @@ -507,7 +507,7 @@ dataview dataView1 (datasource: $Customer) { checkbox cbActive (label: 'Active', attribute: IsActive) datepicker dpCreated (label: 'Created', attribute: CreateDate) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } @@ -531,7 +531,7 @@ the property is only needed when the two would disagree: ```sql dataview dv (datasource: $Customer, showFooter: true) -- empty footer, shown dataview dv (datasource: $Customer, showFooter: false) { -- widgets declared, hidden - footer f { dynamictext t (content: 'hidden') } + footer { dynamictext t (content: 'hidden') } } ``` @@ -979,10 +979,12 @@ wrapper per value (`ACT_Set1`…`ACT_Set9`), each calling the shared implementat ### FOOTER Widget -Container for form action buttons: +Container for form action buttons. Inside a data view it is the data view's +footer region, which stores no name — write `footer { … }` and address it as +`dvName.footer` in `alter page`: ```sql -footer footerName { +footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } diff --git a/.claude/skills/mendix/master-detail-pages/SKILL.md b/.claude/skills/mendix/master-detail-pages/SKILL.md index 1c2605ed4d..e0a4ae2ec7 100644 --- a/.claude/skills/mendix/master-detail-pages/SKILL.md +++ b/.claude/skills/mendix/master-detail-pages/SKILL.md @@ -39,7 +39,7 @@ create page Module.Entity_MasterDetail dataview entityDetail (datasource: selection entityList) { textbox txtName (label: 'Name', attribute: Name) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: success) } } @@ -125,7 +125,7 @@ create page CRM.Customer_MasterDetail textbox txtEmail (label: 'Email', attribute: Email) textbox txtPhone (label: 'Phone', attribute: Phone) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } diff --git a/.claude/skills/mendix/mdl-entities/SKILL.md b/.claude/skills/mendix/mdl-entities/SKILL.md index 6e1f6c3ce0..ad0e570a7c 100644 --- a/.claude/skills/mendix/mdl-entities/SKILL.md +++ b/.claude/skills/mendix/mdl-entities/SKILL.md @@ -346,8 +346,8 @@ alter enumeration Shop.OrderStatus add value Cancelled caption 'Cancelled'; ``` Re-running `create or modify` from `describe` on a Studio Pro association has flipped -its storage from table to column, which is a schema change, with `mxcli diff` reporting -no changes. Never `drop` and re-create an entity to change it: the new entity has a new +its storage from table to column, which is a schema change. Run `mxcli diff` first: it +runs the script on a scratch copy and lists every unit exec would write. Never `drop` and re-create an entity to change it: the new entity has a new identity, and the runtime drops the old table and its rows. ## Quick Reference diff --git a/.claude/skills/mendix/migrate-k2-nintex/SKILL.md b/.claude/skills/mendix/migrate-k2-nintex/SKILL.md index fd18ba3a5f..ca444ba1b6 100644 --- a/.claude/skills/mendix/migrate-k2-nintex/SKILL.md +++ b/.claude/skills/mendix/migrate-k2-nintex/SKILL.md @@ -198,7 +198,7 @@ create page CRM.Customer_Edit checkbox chkActive (label: 'Active', attribute: IsActive) -- Button bar (SmartForms action buttons) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } diff --git a/.claude/skills/mendix/migrate-oracle-forms/SKILL.md b/.claude/skills/mendix/migrate-oracle-forms/SKILL.md index c8e3a2103b..c31c12eac1 100644 --- a/.claude/skills/mendix/migrate-oracle-forms/SKILL.md +++ b/.claude/skills/mendix/migrate-oracle-forms/SKILL.md @@ -127,7 +127,7 @@ create page MyModule.Customer_Edit ( textbox txtCode (Label: 'Customer Code', Attribute: CustomerCode) textbox txtName (Label: 'Name', Attribute: CustomerName) textbox txtEmail (Label: 'Email', Attribute: Email) - footer footer1 { + footer { -- Reference to the microflow created in Phase 2 actionbutton btnSave (Caption: 'Save', Action: call microflow MyModule.ACT_Customer_Save(Customer = $Customer), ButtonStyle: Primary) actionbutton btnCancel (Caption: 'Cancel', Action: close page) diff --git a/.claude/skills/mendix/overview-pages/SKILL.md b/.claude/skills/mendix/overview-pages/SKILL.md index 91953f2f0c..f6cad29084 100644 --- a/.claude/skills/mendix/overview-pages/SKILL.md +++ b/.claude/skills/mendix/overview-pages/SKILL.md @@ -253,7 +253,7 @@ create page Module.Entity_NewEdit folder 'OverviewPages' datepicker dpDueDate (label: 'Due Date', attribute: DueDate) combobox cbStatus (label: 'Status', attribute: status) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } @@ -370,7 +370,7 @@ create page MdlTemplates.Store_NewEdit folder 'OverviewPages' textbox txtName (label: 'Name', attribute: Name) textbox txtLocation (label: 'Location', attribute: Location) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } @@ -428,7 +428,7 @@ create page MdlTemplates.Car_NewEdit folder 'OverviewPages' datepicker dpDate (label: 'Purchase date', attribute: PurchaseDate) radiobuttons rbType (label: 'Car type', attribute: CarType) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } diff --git a/.claude/skills/mendix/write-microflows/reference/pitfalls.md b/.claude/skills/mendix/write-microflows/reference/pitfalls.md index 69d609dacf..b3830b2002 100644 --- a/.claude/skills/mendix/write-microflows/reference/pitfalls.md +++ b/.claude/skills/mendix/write-microflows/reference/pitfalls.md @@ -484,6 +484,11 @@ end loop; > `$Customer/Name`, `$currentObject`, `retrieve … from $List`. Quoting the `$` token > (`"$Customer"`) breaks resolution. +> **In an XPath, quote names, never values.** `["Status" = 'Accepted']` is fine — the +> quotes come off the name, in a retrieve, a grant and a workflow targeting alike. A +> string is always single-quoted: `[Name = "Admin"]` is refused, because a name can +> never be the right-hand side of an XPath comparison (write `[Name = 'Admin']`). + ```mdl mdl 1; create persistent entity Module."item" ( diff --git a/.claude/skills/mendix/write-nanoflows/SKILL.md b/.claude/skills/mendix/write-nanoflows/SKILL.md index 38acc40c5e..4f8ceb08ce 100644 --- a/.claude/skills/mendix/write-nanoflows/SKILL.md +++ b/.claude/skills/mendix/write-nanoflows/SKILL.md @@ -573,19 +573,24 @@ IF $Location = empty THEN END IF; ``` -For per-action error handling without CONTINUE: +For per-action error handling on a call, use a handler WITHOUT rollback — the +only form a nanoflow call takes: ```mdl -$Result = CALL NANOFLOW Sales.NAV_Risky () ON ERROR ROLLBACK; +$Result = CALL NANOFLOW Sales.NAV_Risky () ON ERROR WITHOUT ROLLBACK BEGIN + LOG WARNING NODE 'Sales' 'Could not load the data.'; +END ERROR; ``` -### Most activities take NO error handling in a nanoflow +### Which activities take which clause in a nanoflow -An `ON ERROR` clause of **any** form is rejected on these six, with -**CE6035** "Error handling type is not supported" — measured on Mendix 11.14.0: +Measured on Mendix 11.14.0; every other form is **CE6035** "Error handling type +is not supported", and mxcli refuses it (MDL091): -| Refused in a nanoflow | Accepted | +| Activity in a nanoflow | Accepted clauses | |---|---| -| `CHANGE`, `LOG`, `SHOW PAGE`, `CLOSE PAGE`, `SHOW MESSAGE`, `VALIDATION FEEDBACK` | `DECLARE`, `SET` (the two *variable* activities) | +| `DECLARE`, `SET`, `RETRIEVE`, `DELETE` | every form | +| `CREATE`, `COMMIT`, `CALL NANOFLOW`, `CALL MICROFLOW` | only `ON ERROR WITHOUT ROLLBACK BEGIN … END ERROR` | +| `CHANGE`, `LOG`, `SHOW PAGE`, `CLOSE PAGE`, `SHOW MESSAGE`, `VALIDATION FEEDBACK` | none | mxcli refuses the clause rather than writing a nanoflow mxbuild rejects. The split is by activity, not by "client-side vs server-side" — `SHOW MESSAGE` is as diff --git a/.claude/skills/mendix/write-workflows/SKILL.md b/.claude/skills/mendix/write-workflows/SKILL.md index e2c149c852..e8689514f1 100644 --- a/.claude/skills/mendix/write-workflows/SKILL.md +++ b/.claude/skills/mendix/write-workflows/SKILL.md @@ -408,8 +408,11 @@ workflow / its tasks. They are easy to miss — there is no `complete task`: required**: a notify without one fails the build (CE0166, MDL-WF16). Name the element as `Module.Workflow.ElementName`; mxcli works out which kind it is and refuses one a notification cannot reach (a timer start, a user task). -- `open user task $Task`, `lock workflow $Wf`, and +- `open user task $Task`, `lock workflow $WfDef`, and `workflow operation abort|pause|restart|retry|continue $Wf` are also statements. + A lock or unlock names its workflow definition (`$WfDef` or `Module.Workflow`); + `pause all` / `unpause all` after it is Studio Pro's "Pause / Unpause instances". + A bare `lock workflow all` is refused (MDL-WF17) — it built as CE1825. A common shape: the task page's buttons call a microflow that does the change and then `set task outcome $Task ''`, leaving the workflow's outcome branch diff --git a/.claude/skills/mendix/xpath-constraints/SKILL.md b/.claude/skills/mendix/xpath-constraints/SKILL.md index 7577188b99..50299a1034 100644 --- a/.claude/skills/mendix/xpath-constraints/SKILL.md +++ b/.claude/skills/mendix/xpath-constraints/SKILL.md @@ -167,11 +167,12 @@ where [not(Module.Order_Customer/Module.Customer)] > bare name is not an attribute of the constrained entity **and** is a known > association, so attributes stay bare and XPath functions are never touched. -> **`= empty` does not work on associations (CE0161 / MDL047).** `= empty` tests -> *attribute* nullability only. To test whether an object *has no* associated -> object, use negated existence: `[not(Module.Order_Customer/Module.Customer)]` — -> **not** `[Module.Order_Customer = empty]`. `mxcli check` flags the association -> `= empty` form as **MDL047** before the build does. +> **`= empty` / `!= empty` do not work on associations (CE0161 / MDL047).** +> `empty` tests *attribute* nullability only. To test whether an object *has no* +> associated object, use negated existence: `[not(Module.Order_Customer/Module.Customer)]` +> — **not** `[Module.Order_Customer = empty]`; to test that it *has* one, the path +> itself: `[Module.Order_Customer/Module.Customer]` — **not** `!= empty`. +> `mxcli check` flags both forms as **MDL047** before the build does. ### Variable Paths @@ -213,6 +214,12 @@ where [Displayed = false()] Supported functions: `contains()`, `starts-with()`, `not()`, `true()`, `false()` +The expression functions `startsWith()` / `endsWith()` are not XPath: in a +constraint they are CE0161, and `check` reports them as **MDL091**. A member the +entity does not have is a reference error in `check -p`, and so is a system +member written the way `describe` prints the attribute: XPath spells it +`createdDate`, `changedDate`, `owner`, `changedBy` — `[CreatedDate > …]` is CE0161. + ### Tokens Mendix tokens provide runtime values. In an XPath constraint a token used as a diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a9513a234..b3e3b5a8f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,6 +52,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **`check -p` predicts what `exec` refuses, in seven more places** (ako/mxcli#906, ako/mxcli#558, ako/mxcli#563, mendixlabs/mxcli#1210 – #1213) — each passed `check` and then stopped `exec` part-way, with the statements before it written, or was accepted by both and broke the project. Each prediction calls exec's own decision, so the two cannot disagree. A `create demo user` for a user that exists, and an `alter module … add jar dependency` for a coordinate the module has, are reported like any other "already exists" (the jar actions are replayed in script order, so a drop then add is fine); `create … translations` into the project's source language, or a plain `create translations` of a language that has some, is reported with exec's message. A task queue the script creates earlier now satisfies a later `in queue` (it was "not found" and exec refused the whole script), and one created later is reported. A page or snippet whose widget calls or reads a microflow or nanoflow the script creates further down is **MDL-ORDER01** (a later `create or modify` is reported against the project). A variable passed to a Java action parameter of type `Microflow` is refused by check and exec under every language version: it was written as the reference's text, and the project no longer loaded. `referenceselector` is **MDL-WIDGET38**, naming `combobox` over the association instead of `widget init`, which cannot help with a built-in Forms widget. In a retrieve constraint, **MDL047** now also catches `Module.Assoc != empty` (and in a widget's data source), **MDL091** catches `startsWith()` / `endsWith()` (the XPath functions are `starts-with()` / `ends-with()`), and a member the entity does not have — including `CreatedDate` for the system member `createdDate` — is a reference error, for an entity in the project or declared by the script. Each XPath shape was measured CE0161 on mxbuild 11.13.0 against its control. +- **A published REST operation gets its query and body parameters, its mappings and its commit option** (ako/mxcli#571, mendixlabs/mxcli#1206) — `create published rest service` wrote only the path's `{name}` placeholders, each as a String, so a query or body microflow parameter failed `mx check` with CE0350 and an Integer `{id}` with CE6539; `import mapping`, `export mapping` and `commit` on an operation parsed and were thrown away (CE0350 on the body, CE0354 on a microflow returning an object). The parameters are now derived from the microflow as Studio Pro derives them — named in the path: path parameter; an object or a list: the body; `System.HttpRequest` / `HttpResponse`: none; anything else: query — each with the microflow parameter's type, and the bindings are written. `describe` prints the bindings (`commit` when it is not `Yes`, the value written without the clause) and names in a comment a parameter MDL cannot state (a header, a renamed or described parameter), which `create or modify` and `alter` keep. An unknown commit option is refused by `exec` and `check` (**MDL-REST03**). Executing the `describe` output of a Studio Pro service used to clear its mappings, retype its Integer path parameters, drop its body parameter, set Commit to Yes and turn its Basic and Session authentication off — measured on TestApp's `Services.OrdersRestApi`, 0 `mx check` errors before, 5 after; it now writes nothing. - **`create or modify` of a flow can change an activity's notes, and matches an explicit `on error rollback`** (ako/mxcli#859, part) — a statement that added, reworded or took off an `@annotation` on an activity (or dropped an annotated activity) was refused under the `mdl 1;` header ("the annotations on the replaced … change"; "… carries an annotation, which would be left behind") on every run, and rebuilt the whole flow without it. The stored notes of that activity are now replaced by the ones the statement states; a note shared with another activity (one `describe` gives an `id:`) is still refused. This is what made the Studio Pro-authored PedApp nanoflows `ACT_Feedback_TriggerScreenshotMode` and `ACT_Feedback_UploadImage` unrunnable when their plain description was put under `mdl 1;`: a note's `\r\n` is two characters under mdl 1, so it is a real change, now written once. Separately, `commit $E on error rollback` (any activity with a bare `on error rollback`, which `describe` never prints because it is the stored default) did not match its own stored activity, so it was dropped and written again whenever anything else in the flow changed. - **`describe microflow` prints a Show Page action's title override** (ako/mxcli#869) — `show page M.P with title = '…'` used to describe without its override, so describe → exec dropped it, and taking the override out of a `create or modify microflow` reported "Unchanged" and wrote nothing, under mdl 0 and mdl 1. - **Re-running a `call web service` statement unchanged writes nothing** (ako/mxcli#861) — the call's activity was re-spliced on every run, because its stored raw document holds the element IDs each build mints and was compared byte for byte; it is now compared with those IDs set aside. A stored call that lacks a key mxcli's structured form writes (such as `ErrorHandlingType`), or that refers to its service by ID, now describes as `call web service raw '…'` instead of a structured form that would write a different document. diff --git a/README.md b/README.md index 1f72824acd..80eb953c31 100644 --- a/README.md +++ b/README.md @@ -442,7 +442,7 @@ create page MyModule.Product_Edit textbox txtPrice (label: 'Price', attribute: Price) checkbox cbActive (label: 'Active', attribute: IsActive) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save_changes, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel_changes) } diff --git a/cmd/mxcli/cmd_check.go b/cmd/mxcli/cmd_check.go index 7f892f8fe4..4e004c216c 100644 --- a/cmd/mxcli/cmd_check.go +++ b/cmd/mxcli/cmd_check.go @@ -476,7 +476,7 @@ func parseScriptSet(files []string) []setScript { var out []setScript for _, f := range files { sc := setScript{Path: f} - if data, err := os.ReadFile(f); err == nil && !testrunner.IsTestFile(f) { + if data, err := readMDLSource(f); err == nil && !testrunner.IsTestFile(f) { sc.Source = string(data) if prog, errs := visitor.Build(sc.Source); len(errs) == 0 { sc.Prog = prog diff --git a/cmd/mxcli/cmd_diff.go b/cmd/mxcli/cmd_diff.go index 6a89c83d79..7cced952ab 100644 --- a/cmd/mxcli/cmd_diff.go +++ b/cmd/mxcli/cmd_diff.go @@ -4,20 +4,38 @@ package main import ( "fmt" + "io" "os" + "path/filepath" + "github.com/mendixlabs/mxcli/mdl/backend" + modelsdkbackend "github.com/mendixlabs/mxcli/mdl/backend/modelsdk" "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/scriptdiff" "github.com/mendixlabs/mxcli/mdl/visitor" "github.com/spf13/cobra" ) var diffCmd = &cobra.Command{ Use: "diff ", - Short: "Compare an MDL script against the current project state", - Long: `Compare an MDL script file against the current state of a Mendix project. + Short: "Show what executing an MDL script would change in the project", + Long: `Show what "mxcli exec" would change in a Mendix project, without changing it. -Shows the differences between what the script would create/modify and what -currently exists in the project. +The script is executed by exec itself, against a scratch copy of the project, +and the copy is then compared with the project unit by unit. What diff reports +is therefore exactly what exec would write: the documents it adds, rewrites, +moves or removes (an entity or association counts as its own document), and +the files next to the model it writes. Each change is shown as the DESCRIBE of +the document before and after. A unit exec rewrites whose description does +not change says which properties change instead. + +Like exec, diff runs the pre-flight checks first (skip them with --no-check), +runs the statements in order under the script's language header, and stops at +the first error (--continue-on-error runs every statement). A script exec +would refuse, or a statement it would stop at, is reported as Refused. + +The project itself is only read. The scratch copy leaves out .git, deployment, +releases and node_modules, and is deleted afterwards. Output Formats: unified - Traditional unified diff format (default) @@ -44,6 +62,10 @@ Examples: format, _ := cmd.Flags().GetString("format") useColor, _ := cmd.Flags().GetBool("color") width, _ := cmd.Flags().GetInt("width") + skipCheck, _ := cmd.Flags().GetBool("no-check") + continueOnError, _ := cmd.Flags().GetBool("continue-on-error") + showExecOutput, _ := cmd.Flags().GetBool("exec-output") + depPolicy := deprecationPolicy(cmd) refuseJSONFlag("diff", "--format unified|side|struct") if projectPath == "" { @@ -51,14 +73,12 @@ Examples: os.Exit(1) } - // Read the script file - content, err := os.ReadFile(filePath) + content, err := readMDLSource(filePath) if err != nil { fmt.Fprintf(os.Stderr, "Error reading file: %v\n", err) os.Exit(1) } - // Parse the script prog, errs := visitor.Build(string(content)) if len(errs) > 0 { fmt.Fprintf(os.Stderr, "Syntax errors found:\n") @@ -67,32 +87,36 @@ Examples: } os.Exit(1) } - - // Create executor and connect - exec, logger := newLoggedExecutor("subcommand") - defer logger.Close() - defer exec.Close() - exec.SetDescribeLanguage(mdlFlag(cmd)) - - connectProg, _ := visitor.Build(fmt.Sprintf("CONNECT LOCAL '%s'", visitor.QuoteString(projectPath))) - for _, stmt := range connectProg.Statements { - if err := exec.Execute(stmt); err != nil { - fmt.Fprintf(os.Stderr, "Error connecting: %v\n", err) - os.Exit(1) - } + if line, bad := unparsableInput(string(content), len(prog.Statements)); bad { + fmt.Fprintln(os.Stderr, unparsableInputError(filePath, line)) + os.Exit(1) } - // Run diff - opts := executor.DiffOptions{ - Format: executor.DiffFormat(format), - UseColor: useColor, - Width: width, + opts := scriptdiff.Options{ + // Always the file engine, whatever --mcp says: the script is + // executed for real, and only the scratch copy may receive it. + NewBackend: func() backend.FullBackend { return modelsdkbackend.New() }, + ContinueOnError: continueOnError, + Preflight: func(scratch *executor.Executor, w io.Writer) string { + return execPreflight(scratch, prog, projectPath, skipCheck, depPolicy, w, useColor) + }, + } + if filePath != "-" { + if abs, absErr := filepath.Abs(filePath); absErr == nil { + opts.ScriptDir = filepath.Dir(abs) + } } - if err := exec.DiffProgram(prog, opts); err != nil { + report, err := scriptdiff.Run(projectPath, prog, opts) + if err != nil { fmt.Fprintf(os.Stderr, "Error: %v\n", err) os.Exit(1) } + report.Write(os.Stdout, executor.DiffOptions{ + Format: executor.DiffFormat(format), + UseColor: useColor, + Width: width, + }, showExecOutput) }, } diff --git a/cmd/mxcli/cmd_exec.go b/cmd/mxcli/cmd_exec.go index d50f5a2d85..035e37d80c 100644 --- a/cmd/mxcli/cmd_exec.go +++ b/cmd/mxcli/cmd_exec.go @@ -9,8 +9,8 @@ import ( "path/filepath" "github.com/mendixlabs/mxcli/mdl/executor" - "github.com/mendixlabs/mxcli/mdl/linter" "github.com/mendixlabs/mxcli/mdl/visitor" + mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" "github.com/spf13/cobra" ) @@ -35,6 +35,13 @@ makes a partially-applied domain script re-runnable — the already-applied statements (e.g. "attribute already exists") error individually while the not- yet-applied ones still run — without a failure masking later work. +A write is refused while Studio Pro has the project open (its .mpr.lock +is beside the .mpr): Studio Pro does not reload the model from disk, and its next +save would silently discard the change. Close the project in Studio Pro, or route +writes through it with --mcp. --force writes anyway (for a lock left behind by a +crash); MXCLI_ALLOW_STUDIO_PRO_OPEN=1 does the same for every command. Reads, and +re-running a script whose statements change nothing, are never refused. + Pass "-" as the file to read the script from standard input, so MDL can be piped or written inline as a heredoc without a temporary file. @@ -53,6 +60,13 @@ Example: projectPath, _ := cmd.Flags().GetString("project") continueOnError, _ := cmd.Flags().GetBool("continue-on-error") skipCheck, _ := cmd.Flags().GetBool("no-check") + if force, _ := cmd.Flags().GetBool("force"); force { + mmpr.AllowWritesWhileStudioProOpen = true + if lock, _ := mmpr.StudioProLockFile(projectPath); lock != "" { + fmt.Fprintf(os.Stderr, "Warning: Studio Pro appears to have this project open (%s); writing anyway (--force). "+ + "Studio Pro's next save will discard these changes unless the project is closed or reloaded first.\n", lock) + } + } depPolicy := deprecationPolicy(cmd) // Read the script (a path, or "-" for stdin) @@ -102,98 +116,9 @@ Example: os.Exit(1) } - // Pre-flight: refuse a script whose semantic checks report an error, - // rather than writing part of it and leaving the model to mxbuild. - // exec is not transactional, so "run it and see" means a half-applied - // model. Warnings are printed and do not stop the run. - if !skipCheck { - violations := executor.ApplyDeprecationPolicy(executor.ValidateProgram(prog, projectPath), depPolicy) - if len(violations) > 0 { - formatter := linter.GetFormatter(linter.OutputFormatText, true) - formatter.Format(violations, os.Stderr) - } - if summary := linter.Summarize(violations); summary.Errors > 0 { - fmt.Fprintf(os.Stderr, - "\nRefusing to execute: %d error(s) above. Nothing was written.\n"+ - " exec applies statements one at a time and cannot roll back, so a script\n"+ - " with a known error would leave the model partly updated.\n"+ - " Fix them, or re-run with --no-check to apply the script anyway.\n", - summary.Errors) - os.Exit(1) - } - } - - // Second preflight pass: resolve every NAME against the connected - // project. The semantic pass above cannot do this — a missing module, - // entity, page or microflow needs a backend, not a path — so `mxcli - // check -p` ran it and `exec` did not (#607). - // - // MEASURED on the expr-checker fixture, running the pre-fix binary - // (`--no-check` reproduces it), because the failure mode is not the one - // the refusal above describes and the difference matters: - // - // create entity "NotAModule"."Thing" -> exit 0, "Created module: - // NotAModule". A misspelled - // module is SILENTLY CREATED. - // microflow retrieving a missing entity -> exit 0, both documents - // written. The dangling name - // reaches the model and is - // not reported until mxbuild - // rejects it (CE1613). - // - // So exec did not half-apply here — it completed, and wrote a model that - // only a 25s build would reject. That makes this a check-to-build parity - // fix (moving a build-tier error to the 2s tier) and a fix for the - // "no silent side effects on typos" rule in CLAUDE.md's checklist, which - // auto-creating a module on a misspelling violates outright. - // - // Safe to refuse on, because the pass skips references to objects the - // script itself creates: an error from it means the name resolves to - // nothing in the project AND is not created here, so exec would have - // failed on it regardless — later, and after writing. - // - // Only possible with -p. A script that connects with its own CONNECT - // statement has no backend until ExecuteProgram runs, which is the same - // condition `check` gates this on. - // - // CheckProjectConflicts is deliberately NOT run here, though `check` - // runs it alongside this pass: a plain CREATE over an existing document - // is worth reporting when validating a script, but it is ordinary for a - // re-run, and refusing it would break scripts that work today. Its one - // exception is: a create over a name ANOTHER kind already has in the - // module (ako/mxcli#793) is never a re-run — `or modify` of a kind that - // lacks the name still adds an element — and Mendix rejects the model - // (CE0122 / CE0065), so it is refused here, after the references. - if !skipCheck && projectPath != "" { - refErrs, refWarnings := exec.ValidateProgramWithWarnings(prog) - for _, w := range refWarnings { - fmt.Fprintf(os.Stderr, "Reference warning: %s\n", w) - } - if len(refErrs) > 0 { - for _, refErr := range refErrs { - fmt.Fprintf(os.Stderr, "Reference error: %v\n", refErr) - } - fmt.Fprintf(os.Stderr, - "\nRefusing to execute: %d unresolved reference(s) above. Nothing was written.\n"+ - " A name that resolves to nothing is written into the model as it stands and\n"+ - " is not reported until mxbuild rejects it (CE1613) — and a misspelled MODULE\n"+ - " is created rather than refused.\n"+ - " Fix them, or re-run with --no-check to apply the script anyway.\n", - len(refErrs)) - os.Exit(1) - } - if clashes := exec.CheckProjectNameClashes(prog); len(clashes) > 0 { - for _, c := range clashes { - fmt.Fprintf(os.Stderr, "Name clash: %v\n", c) - } - fmt.Fprintf(os.Stderr, - "\nRefusing to execute: %d name clash(es) above. Nothing was written.\n"+ - " Mendix rejects the model (CE0122 / CE0065) however the script continues.\n"+ - " Rename, or re-run with --no-check to apply the script anyway (each clashing\n"+ - " create is still refused when it runs).\n", - len(clashes)) - os.Exit(1) - } + if refusal := execPreflight(exec, prog, projectPath, skipCheck, depPolicy, os.Stderr, true); refusal != "" { + fmt.Fprint(os.Stderr, refusal) + os.Exit(1) } if continueOnError { @@ -222,6 +147,8 @@ Example: func init() { execCmd.Flags().Bool("no-check", false, "Skip the pre-flight semantic checks and apply the script even if mxcli check would report errors") + execCmd.Flags().Bool("force", false, + "Write even though Studio Pro appears to have the project open (its .mpr.lock is present) — e.g. a lock left behind by a crash") execCmd.Flags().Bool("continue-on-error", false, "Run every statement, reporting each failure instead of halting at the first (exits non-zero if any failed) — makes a partially-applied script re-runnable") } diff --git a/cmd/mxcli/cmd_fmt.go b/cmd/mxcli/cmd_fmt.go index 540c736449..62c5936948 100644 --- a/cmd/mxcli/cmd_fmt.go +++ b/cmd/mxcli/cmd_fmt.go @@ -148,9 +148,9 @@ func fmtFile(cmd *cobra.Command, args []string, declineHeader string) error { var data []byte var err error if fromStdin { - data, err = io.ReadAll(os.Stdin) + data, err = readMDLSource(stdinPath) } else { - data, err = os.ReadFile(filePath) + data, err = readMDLSource(filePath) } if err != nil { return fmt.Errorf("failed to read input: %w", err) diff --git a/cmd/mxcli/cmd_version.go b/cmd/mxcli/cmd_version.go new file mode 100644 index 0000000000..01530845e5 --- /dev/null +++ b/cmd/mxcli/cmd_version.go @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + + "github.com/spf13/cobra" +) + +// versionCmd is `mxcli version`, the bare-subcommand spelling every +// neighbouring tool (go, docker, gh, mx) accepts. It prints exactly what +// `--version` prints — the build version and build time compiled in through +// -X main.Version / -X main.BuildTime — so a bug report quoting either one +// identifies the same build (ako/mxcli#534). +var versionCmd = &cobra.Command{ + Use: "version", + Short: "Print the mxcli version and build time", + Args: cobra.NoArgs, + Run: func(cmd *cobra.Command, args []string) { + root := cmd.Root() + // Same text as cobra's default --version template. + fmt.Fprintf(cmd.OutOrStdout(), "%s version %s\n", root.Name(), root.Version) + }, +} + +func init() { + rootCmd.AddCommand(versionCmd) +} diff --git a/cmd/mxcli/cmd_version_test.go b/cmd/mxcli/cmd_version_test.go new file mode 100644 index 0000000000..411973aaf1 --- /dev/null +++ b/cmd/mxcli/cmd_version_test.go @@ -0,0 +1,40 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "strings" + "testing" +) + +// TestVersionSubcommand is ako/mxcli#534: `mxcli version` was an unknown +// command although --version worked and shouldSuppressWarning already +// expected a `version` argument. The subcommand must print exactly what +// --version prints, so a bug report quoting either one carries the same +// build identification. +func TestVersionSubcommand(t *testing.T) { + sub := runRootForTest(t, []string{"version"}) + flag := runRootForTest(t, []string{"--version"}) + if strings.TrimSpace(sub) == "" { + t.Fatal("`mxcli version` printed nothing") + } + if sub != flag { + t.Errorf("`mxcli version` = %q, `mxcli --version` = %q; want identical", sub, flag) + } + if !strings.Contains(sub, rootCmd.Version) { + t.Errorf("`mxcli version` = %q, does not carry the build version %q", sub, rootCmd.Version) + } +} + +func runRootForTest(t *testing.T, args []string) string { + t.Helper() + rootCmd.SetOut(nil) + out, err := captureStdout(t, func() error { + rootCmd.SetArgs(args) + return rootCmd.Execute() + }) + if err != nil { + t.Fatalf("`mxcli %v` failed: %v\n%s", args, err, out) + } + return out +} diff --git a/cmd/mxcli/deprecations_flag.go b/cmd/mxcli/deprecations_flag.go index db892c292e..07126be896 100644 --- a/cmd/mxcli/deprecations_flag.go +++ b/cmd/mxcli/deprecations_flag.go @@ -10,7 +10,7 @@ import ( "github.com/spf13/cobra" ) -// The --deprecations flag decides what `check` and `exec` do with a deprecated +// The --deprecations flag decides what `check`, `exec` and `diff` do with a deprecated // MDL spelling (an MDL-DEPRnnn warning, registry in mdl/deprecation). The // default warns; `error` fails the run, so CI over docs, skills and examples // can hold them to the canonical form. @@ -20,6 +20,7 @@ const deprecationsFlagUsage = "What to do with a deprecated MDL spelling (MDL-DE func init() { checkCmd.Flags().String("deprecations", "warn", deprecationsFlagUsage) execCmd.Flags().String("deprecations", "warn", deprecationsFlagUsage) + diffCmd.Flags().String("deprecations", "warn", deprecationsFlagUsage) } // deprecationPolicy reads --deprecations, exiting with a usage error on a value diff --git a/cmd/mxcli/exec_preflight.go b/cmd/mxcli/exec_preflight.go new file mode 100644 index 0000000000..61432cdfac --- /dev/null +++ b/cmd/mxcli/exec_preflight.go @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "io" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/deprecation" + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// execPreflight runs exec's pre-flight checks on prog, printing what they +// report to w, and returns why exec refuses to run the script — the message it +// prints before exiting — or "" when it runs it. `mxcli diff` runs the same +// checks, so it refuses exactly the scripts exec refuses (ako/mxcli#807). +// +// exec is the executor the script would run on, connected to projectPath ("" when +// the script connects itself). +func execPreflight(exec *executor.Executor, prog *ast.Program, projectPath string, skipCheck bool, depPolicy deprecation.Policy, w io.Writer, color bool) string { + // Pre-flight: refuse a script whose semantic checks report an error, + // rather than writing part of it and leaving the model to mxbuild. + // exec is not transactional, so "run it and see" means a half-applied + // model. Warnings are printed and do not stop the run. + if !skipCheck { + violations := executor.ApplyDeprecationPolicy(executor.ValidateProgram(prog, projectPath), depPolicy) + if len(violations) > 0 { + formatter := linter.GetFormatter(linter.OutputFormatText, color) + formatter.Format(violations, w) + } + if summary := linter.Summarize(violations); summary.Errors > 0 { + return fmt.Sprintf( + "\nRefusing to execute: %d error(s) above. Nothing was written.\n"+ + " exec applies statements one at a time and cannot roll back, so a script\n"+ + " with a known error would leave the model partly updated.\n"+ + " Fix them, or re-run with --no-check to apply the script anyway.\n", + summary.Errors) + } + } + + // Second preflight pass: resolve every NAME against the connected + // project. The semantic pass above cannot do this — a missing module, + // entity, page or microflow needs a backend, not a path — so `mxcli + // check -p` ran it and `exec` did not (#607). + // + // MEASURED on the expr-checker fixture, running the pre-fix binary + // (`--no-check` reproduces it), because the failure mode is not the one + // the refusal above describes and the difference matters: + // + // create entity "NotAModule"."Thing" -> exit 0, "Created module: + // NotAModule". A misspelled + // module is SILENTLY CREATED. + // microflow retrieving a missing entity -> exit 0, both documents + // written. The dangling name + // reaches the model and is + // not reported until mxbuild + // rejects it (CE1613). + // + // So exec did not half-apply here — it completed, and wrote a model that + // only a 25s build would reject. That makes this a check-to-build parity + // fix (moving a build-tier error to the 2s tier) and a fix for the + // "no silent side effects on typos" rule in CLAUDE.md's checklist, which + // auto-creating a module on a misspelling violates outright. + // + // Safe to refuse on, because the pass skips references to objects the + // script itself creates: an error from it means the name resolves to + // nothing in the project AND is not created here, so exec would have + // failed on it regardless — later, and after writing. + // + // Only possible with -p. A script that connects with its own CONNECT + // statement has no backend until ExecuteProgram runs, which is the same + // condition `check` gates this on. + // + // CheckProjectConflicts is deliberately NOT run here, though `check` + // runs it alongside this pass: a plain CREATE over an existing document + // is worth reporting when validating a script, but it is ordinary for a + // re-run, and refusing it would break scripts that work today. Its one + // exception is: a create over a name ANOTHER kind already has in the + // module (ako/mxcli#793) is never a re-run — `or modify` of a kind that + // lacks the name still adds an element — and Mendix rejects the model + // (CE0122 / CE0065), so it is refused here, after the references. + if !skipCheck && projectPath != "" { + refErrs, refWarnings := exec.ValidateProgramWithWarnings(prog) + for _, warning := range refWarnings { + fmt.Fprintf(w, "Reference warning: %s\n", warning) + } + if len(refErrs) > 0 { + for _, refErr := range refErrs { + fmt.Fprintf(w, "Reference error: %v\n", refErr) + } + return fmt.Sprintf( + "\nRefusing to execute: %d unresolved reference(s) above. Nothing was written.\n"+ + " A name that resolves to nothing is written into the model as it stands and\n"+ + " is not reported until mxbuild rejects it (CE1613) — and a misspelled MODULE\n"+ + " is created rather than refused.\n"+ + " Fix them, or re-run with --no-check to apply the script anyway.\n", + len(refErrs)) + } + if clashes := exec.CheckProjectNameClashes(prog); len(clashes) > 0 { + for _, c := range clashes { + fmt.Fprintf(w, "Name clash: %v\n", c) + } + return fmt.Sprintf( + "\nRefusing to execute: %d name clash(es) above. Nothing was written.\n"+ + " Mendix rejects the model (CE0122 / CE0065) however the script continues.\n"+ + " Rename, or re-run with --no-check to apply the script anyway (each clashing\n"+ + " create is still refused when it runs).\n", + len(clashes)) + } + } + return "" +} diff --git a/cmd/mxcli/lsp_diagnostics.go b/cmd/mxcli/lsp_diagnostics.go index 684fe6695a..8ad4c37238 100644 --- a/cmd/mxcli/lsp_diagnostics.go +++ b/cmd/mxcli/lsp_diagnostics.go @@ -29,7 +29,7 @@ var errLineRegexp = regexp.MustCompile(`^line (\d+):(\d+) (.+)$`) // (mendixlabs/mxcli#1103). A test block is a microflow body; testrunner renders // it as one, on the same lines, so the positions below need no adjustment. func checkableDocument(docURI uri.URI, text string) (string, []protocol.Diagnostic) { - path := docURI.Filename() + path := documentPath(docURI) if !testrunner.IsTestFile(path) { return text, nil } @@ -196,7 +196,11 @@ func (s *mdlServer) runSemanticCheck(ctx context.Context, docURI uri.URI, text s return } - // Write to a temp file path based on the URI + // The check reads the document from disk; a virtual document + // (mendix-mdl:, mendixlabs/mxcli#1245) has no file to read. + if !isFileURI(docURI) { + return + } filePath := uriToPath(string(docURI)) if filePath == "" { return diff --git a/cmd/mxcli/lsp_helpers.go b/cmd/mxcli/lsp_helpers.go index 13751a9e45..df20b3fe47 100644 --- a/cmd/mxcli/lsp_helpers.go +++ b/cmd/mxcli/lsp_helpers.go @@ -7,6 +7,8 @@ import ( "context" "encoding/json" "fmt" + "go.lsp.dev/uri" + "net/url" "os" "os/exec" "path/filepath" @@ -220,3 +222,29 @@ func parseTableOutput(output string) []projectElement { } return elems } + +// documentPath is the path a document's URI names, for deciding what kind of +// document it is (a `.test.mdl` file, say). uri.URI.Filename panics on any +// scheme but file, and the VS Code extension opens describe previews as +// `mendix-mdl:` virtual documents, so calling it directly took the server down +// on every didOpen (mendixlabs/mxcli#1245). A non-file URI yields its path +// component; it is a name, not a file on disk. +func documentPath(docURI uri.URI) string { + if isFileURI(docURI) { + return docURI.Filename() + } + if u, err := url.Parse(string(docURI)); err == nil { + if u.Path != "" { + return u.Path + } + return u.Opaque + } + return "" +} + +// isFileURI reports whether a document lives on disk. Only such a document can +// be handed to `mxcli check` by path. +func isFileURI(docURI uri.URI) bool { + u, err := url.Parse(string(docURI)) + return err == nil && u.Scheme == uri.FileScheme +} diff --git a/cmd/mxcli/lsp_language.go b/cmd/mxcli/lsp_language.go index ed37703eb4..72a06082dc 100644 --- a/cmd/mxcli/lsp_language.go +++ b/cmd/mxcli/lsp_language.go @@ -192,7 +192,7 @@ func (s *mdlServer) migrationHover(docURI uri.URI, text string, pos protocol.Pos // `fmt --upgrade` is offered too. func (s *mdlServer) CodeAction(ctx context.Context, params *protocol.CodeActionParams) ([]protocol.CodeAction, error) { docURI := uri.URI(params.TextDocument.URI) - if testrunner.IsTestFile(docURI.Filename()) { + if testrunner.IsTestFile(documentPath(docURI)) { return nil, nil // diagnosed as the MDL rendered from it, not as written } s.mu.Lock() diff --git a/cmd/mxcli/lsp_virtual_uri_test.go b/cmd/mxcli/lsp_virtual_uri_test.go new file mode 100644 index 0000000000..f2fddad2f2 --- /dev/null +++ b/cmd/mxcli/lsp_virtual_uri_test.go @@ -0,0 +1,46 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "testing" + + "go.lsp.dev/protocol" + "go.lsp.dev/uri" +) + +// mendixlabs/mxcli#1245: the VS Code extension opens describe previews as +// `mendix-mdl:` virtual documents, and uri.URI.Filename panics on any scheme +// but file — "panic: only file URIs are supported, got mendix-mdl". The server +// exited on every didOpen until VS Code gave up restarting it. A virtual +// document is diagnosed in memory like any other; the file control proves the +// same text yields the same diagnostics, so the virtual case is not passing by +// being skipped. +func TestLSPVirtualDocumentDoesNotPanic(t *testing.T) { + const virtual = uri.URI("mendix-mdl:/MyModule/Foo.mdl") + s := newMDLServer(nil) + + defer func() { + if r := recover(); r != nil { + t.Fatalf("a %s document panicked the server: %v", virtual, r) + } + }() + + got := s.documentDiagnostics(virtual, replaceEntity) + want := s.documentDiagnostics(lspTestURI, replaceEntity) + if len(diagsWithCode(got, "MDL-DEPR001")) != 1 || len(got) != len(want) { + t.Errorf("virtual document diagnostics %+v, file control %+v", got, want) + } + + if _, err := s.CodeAction(context.Background(), &protocol.CodeActionParams{ + TextDocument: protocol.TextDocumentIdentifier{URI: protocol.DocumentURI(virtual)}, + }); err != nil { + t.Errorf("CodeAction on a virtual document: %v", err) + } + + // A virtual .test.mdl is still recognised as a test file by its name. + if p := documentPath(uri.URI("mendix-mdl:/MyModule/Foo.test.mdl")); p != "/MyModule/Foo.test.mdl" { + t.Errorf("documentPath = %q", p) + } +} diff --git a/cmd/mxcli/main.go b/cmd/mxcli/main.go index 317be7b5bf..62355e6104 100644 --- a/cmd/mxcli/main.go +++ b/cmd/mxcli/main.go @@ -13,7 +13,6 @@ import ( "github.com/mendixlabs/mxcli/mdl/backend" "github.com/mendixlabs/mxcli/mdl/diaglog" "github.com/mendixlabs/mxcli/mdl/executor" - "github.com/mendixlabs/mxcli/mdl/langver" "github.com/mendixlabs/mxcli/mdl/repl" "github.com/mendixlabs/mxcli/mdl/visitor" "github.com/spf13/cobra" @@ -143,51 +142,13 @@ beta language; --mdl 0 starts them in the alpha language, and in the REPL an commands, _ := cmd.Flags().GetString("command") projectPath, _ := cmd.Flags().GetString("project") - if commands != "" { - // Execute commands from -c flag - exec, logger := newLoggedExecutor("batch") - defer logger.Close() - defer exec.Close() - - // Suppress status messages when stdout is a pipe so that - // output can be piped directly to other tools (e.g. mxcli fmt). - if fi, statErr := os.Stdout.Stat(); statErr == nil && (fi.Mode()&os.ModeCharDevice) == 0 { - exec.SetQuiet(true) - } - - // The one-liner's language (freeze decision 6): mdl 1 unless - // --mdl 0, or a header the commands state themselves. It is what - // headerless input is read as and what describe writes. - lang := mdlFlag(cmd) - if v, written := langver.ScanWrittenHeader(commands); written { - lang = v - } - exec.SetDescribeLanguage(lang) - - // Auto-connect if project specified. CONNECT runs on its own so - // that a header in the commands stays their first statement. - if projectPath != "" { - connectProg, _ := visitor.Build(fmt.Sprintf("CONNECT LOCAL '%s';", visitor.QuoteString(projectPath))) - if err := exec.ExecuteProgram(connectProg); err != nil { - fmt.Fprintf(os.Stderr, "Error: %v\n", err) - os.Exit(1) - } - } - - prog, errs := visitor.BuildSession(commands, lang) - if len(errs) > 0 { - for _, err := range errs { - fmt.Fprintf(os.Stderr, "Parse error: %v\n", err) - } - os.Exit(1) - } - - if err := exec.ExecuteProgram(prog); err != nil { - if errors.Is(err, executor.ErrExit) { - return - } - fmt.Fprintf(os.Stderr, "Error: %v\n", err) - os.Exit(1) + // -c given at all — even empty — is a one-liner run, never the REPL: + // an empty -c from a generator used to open the interactive prompt and + // hang its caller (mendixlabs/mxcli#1218). + if cmd.Flags().Changed("command") { + continueOnError, _ := cmd.Flags().GetBool("continue-on-error") + if code := runCommandLine(cmd, commands, projectPath, continueOnError, os.Stderr); code != 0 { + os.Exit(code) } } else { // Start interactive REPL @@ -353,7 +314,9 @@ func init() { rootCmd.PersistentFlags().Bool("mcp-verbose", false, "Print each PED tool call the MCP backend makes (requires --mcp)") rootCmd.PersistentFlags().Bool("mcp-trace", false, "Print each MDL command with the PED tool calls it makes (implies --mcp-verbose; requires --mcp)") rootCmd.PersistentFlags().String("engine", "", "Deprecated and ignored: there is one model engine. Kept so scripts pinning the old one keep running.") - rootCmd.Flags().StringP("command", "c", "", "Execute MDL command(s) and exit") + rootCmd.Flags().StringP("command", "c", "", "Execute MDL command(s) and exit (an empty value is an error, never the REPL)") + rootCmd.Flags().Bool("continue-on-error", false, + "With -c: run every statement, reporting each failure instead of stopping at the first (exits non-zero if any failed)") // Check command flags checkCmd.Flags().BoolP("references", "r", false, "Validate references against the project (implied by -p; kept for compatibility)") @@ -364,6 +327,9 @@ func init() { diffCmd.Flags().StringP("format", "f", "unified", "Output format: unified, side, struct") diffCmd.Flags().BoolP("color", "", false, "Use colored output") diffCmd.Flags().IntP("width", "w", 120, "Terminal width for side-by-side format") + diffCmd.Flags().Bool("no-check", false, "Skip exec's pre-flight checks, as exec --no-check does") + diffCmd.Flags().Bool("continue-on-error", false, "Run every statement, as exec --continue-on-error does, instead of stopping at the first error") + diffCmd.Flags().Bool("exec-output", false, "Also print what exec reports while it runs the script on the scratch copy") // Diff-local command flags diffLocalCmd.Flags().StringP("ref", "r", "HEAD", "Git ref or range (e.g., HEAD, main, main..feature)") diff --git a/cmd/mxcli/mdlsource.go b/cmd/mxcli/mdlsource.go index e2fb5c854f..2511e0ed90 100644 --- a/cmd/mxcli/mdlsource.go +++ b/cmd/mxcli/mdlsource.go @@ -6,6 +6,8 @@ import ( "fmt" "io" "os" + + "github.com/mendixlabs/mxcli/mdl/srctext" ) // stdinPath is the conventional spelling for "read from standard input". @@ -18,15 +20,27 @@ const stdinPath = "-" // `-` is how every other Unix tool spells it — without this the dash was taken // literally and the command failed with "open -: no such file or directory", // forcing a temp file. (mxcli-todo findings #5) +// +// The bytes are decoded by srctext.Decode: a leading UTF-8 byte-order mark is +// dropped and a UTF-16 file is decoded, so a script saved by Windows +// PowerShell 5.1 or Notepad reads the same as one saved without a BOM +// (mendixlabs/mxcli#1253). func readMDLSource(path string) ([]byte, error) { + var content []byte + var err error if path == stdinPath { - content, err := io.ReadAll(os.Stdin) + content, err = io.ReadAll(os.Stdin) if err != nil { return nil, fmt.Errorf("reading MDL from stdin: %w", err) } - return content, nil + } else if content, err = os.ReadFile(path); err != nil { + return nil, err + } + decoded, err := srctext.Decode(content) + if err != nil { + return nil, fmt.Errorf("%s: %w", mdlSourceLabel(path), err) } - return os.ReadFile(path) + return decoded, nil } // mdlSourceLabel names the source in messages: a real path, or "". diff --git a/cmd/mxcli/mdlsource_encoding_test.go b/cmd/mxcli/mdlsource_encoding_test.go new file mode 100644 index 0000000000..03836c29e1 --- /dev/null +++ b/cmd/mxcli/mdlsource_encoding_test.go @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "os" + "path/filepath" + "strings" + "testing" + "unicode/utf16" + + "github.com/mendixlabs/mxcli/cmd/mxcli/testrunner" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +func encodeUTF16LE(s string) []byte { + out := []byte{0xFF, 0xFE} + for _, u := range utf16.Encode([]rune(s)) { + out = append(out, byte(u), byte(u>>8)) + } + return out +} + +// mendixlabs/mxcli#1253: a script written by Windows PowerShell 5.1 — a UTF-8 +// BOM from `Set-Content -Encoding UTF8`, UTF-16LE from `>` — failed check and +// exec with "token recognition error at: 'U+FEFF'". Every script reader the +// CLI has must hand the parser the decoded text. The plain-UTF-8 file is the +// control: it must parse under the same readers, or a pass proves nothing. +func TestScriptReadersDecodeBOMAndUTF16(t *testing.T) { + const script = "mdl 1;\r\ncreate module BomTest;\r\n" + dir := t.TempDir() + files := map[string][]byte{ + "plain.mdl": []byte(script), + "bom.mdl": append([]byte{0xEF, 0xBB, 0xBF}, script...), + "utf16.mdl": encodeUTF16LE(script), + } + for name, body := range files { + p := filepath.Join(dir, name) + if err := os.WriteFile(p, body, 0o644); err != nil { + t.Fatal(err) + } + + // check / exec / fmt / diff + src, err := readMDLSource(p) + if err != nil { + t.Fatalf("%s: readMDLSource: %v", name, err) + } + if _, errs := visitor.Build(string(src)); len(errs) > 0 { + t.Errorf("%s: readMDLSource text does not parse: %v", name, errs) + } + if !strings.HasPrefix(string(src), "mdl 1;") { + t.Errorf("%s: the header is not the first thing in the text: %q", name, src) + } + + // the multi-file check's set-level pass + set := parseScriptSet([]string{p}) + if len(set) != 1 || set[0].Prog == nil { + t.Errorf("%s: parseScriptSet did not parse the file", name) + } + } +} + +// The test runner reads .test.mdl files on its own path. +func TestTestRunnerDecodesBOM(t *testing.T) { + const body = "/**\n * @test bom\n */\nbegin\n return true;\nend;\n" + dir := t.TempDir() + plain := filepath.Join(dir, "plain.test.mdl") + bom := filepath.Join(dir, "bom.test.mdl") + _ = os.WriteFile(plain, []byte(body), 0o644) + _ = os.WriteFile(bom, append([]byte{0xEF, 0xBB, 0xBF}, body...), 0o644) + want, err := testrunner.ParseTestFile(plain) + if err != nil { + t.Fatal(err) + } + got, err := testrunner.ParseTestFile(bom) + if err != nil { + t.Fatal(err) + } + if len(got.Tests) != len(want.Tests) || len(want.Tests) == 0 { + t.Fatalf("bom file: %d tests, plain control: %d", len(got.Tests), len(want.Tests)) + } + if got.Tests[0].MDL != want.Tests[0].MDL { + t.Errorf("bom file test body %q, plain %q", got.Tests[0].MDL, want.Tests[0].MDL) + } +} diff --git a/cmd/mxcli/oneliner.go b/cmd/mxcli/oneliner.go new file mode 100644 index 0000000000..fef5bf5532 --- /dev/null +++ b/cmd/mxcli/oneliner.go @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "errors" + "fmt" + "io" + "os" + "strings" + + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/langver" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/spf13/cobra" +) + +// runCommandLine runs the statements of `mxcli -c` and returns the process exit +// code. Failures are written to errOut. +// +// The failure semantics are those of `mxcli exec` (mendixlabs/mxcli#1218): +// +// - An empty or whitespace-only -c is an error. It used to fall through to the +// interactive REPL, which a generator that produced an empty statement list +// experienced as a hang. +// - By default the run stops at the first failing statement, read-only or not +// — a later statement may depend on an earlier one, and a script that +// carries on past a failure can write on a wrong premise. The error says +// which statement failed and how many were not run, so the stop is never +// silent. +// - --continue-on-error attempts every statement, reports each failure with +// its statement number, and exits non-zero if any failed. +func runCommandLine(cmd *cobra.Command, commands, projectPath string, continueOnError bool, errOut io.Writer) int { + if strings.TrimSpace(commands) == "" { + fmt.Fprintln(errOut, "Error: -c was given no MDL to run. Pass one or more statements, "+ + "e.g. -c \"list modules;\"; run mxcli without -c for the interactive REPL.") + return 1 + } + + exec, logger := newLoggedExecutor("batch") + defer logger.Close() + defer exec.Close() + + // Suppress status messages when stdout is a pipe so that + // output can be piped directly to other tools (e.g. mxcli fmt). + if fi, statErr := os.Stdout.Stat(); statErr == nil && (fi.Mode()&os.ModeCharDevice) == 0 { + exec.SetQuiet(true) + } + + // The one-liner's language (freeze decision 6): mdl 1 unless + // --mdl 0, or a header the commands state themselves. It is what + // headerless input is read as and what describe writes. + lang := mdlFlag(cmd) + if v, written := langver.ScanWrittenHeader(commands); written { + lang = v + } + exec.SetDescribeLanguage(lang) + + // Auto-connect if project specified. CONNECT runs on its own so + // that a header in the commands stays their first statement. + if projectPath != "" { + connectProg, _ := visitor.Build(fmt.Sprintf("CONNECT LOCAL '%s';", visitor.QuoteString(projectPath))) + if err := exec.ExecuteProgram(connectProg); err != nil { + fmt.Fprintf(errOut, "Error: %v\n", err) + return 1 + } + } + + prog, errs := visitor.BuildSession(commands, lang) + if len(errs) > 0 { + for _, err := range errs { + fmt.Fprintf(errOut, "Parse error: %v\n", err) + } + return 1 + } + + if continueOnError { + res, err := exec.ExecuteProgramContinueOnError(prog, errOut) + if err != nil && !errors.Is(err, executor.ErrExit) { + fmt.Fprintf(errOut, "Error: %v\n", err) + return 1 + } + if res.Failed > 0 { + fmt.Fprintf(errOut, "%d statements: %d succeeded, %d failed\n", res.Total, res.Succeeded, res.Failed) + return 1 + } + return 0 + } + + stoppedAt, err := exec.ExecuteProgramReportingStop(prog) + if err == nil || errors.Is(err, executor.ErrExit) { + return 0 + } + total := len(prog.Statements) + if stoppedAt >= 0 && total > 1 { + fmt.Fprintf(errOut, "Error: statement %d of %d: %v\n", stoppedAt+1, total, err) + if skipped := total - stoppedAt - 1; skipped > 0 { + fmt.Fprintf(errOut, "Stopped: %d later statement(s) not run. --continue-on-error runs every statement.\n", skipped) + } + return 1 + } + fmt.Fprintf(errOut, "Error: %v\n", err) + return 1 +} diff --git a/cmd/mxcli/oneliner_test.go b/cmd/mxcli/oneliner_test.go new file mode 100644 index 0000000000..967c66f468 --- /dev/null +++ b/cmd/mxcli/oneliner_test.go @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "strings" + "testing" +) + +// runOneLinerForTest runs `mxcli -c ` without a project and returns +// the exit code and everything written to the error stream. +func runOneLinerForTest(t *testing.T, commands string, continueOnError bool) (int, string, string) { + t.Helper() + t.Setenv("HOME", t.TempDir()) // the session log is not the developer's + var errOut bytes.Buffer + var code int + out, _ := captureStdout(t, func() error { + code = runCommandLine(rootCmd, commands, "", continueOnError, &errOut) + return nil + }) + return code, out, errOut.String() +} + +// mendixlabs/mxcli#1218 (1): `-c ""` fell through to the interactive REPL, +// which hangs a generator that spawned mxcli with an open stdin. Empty or +// whitespace-only input is an error, and nothing waits on stdin. +func TestOneLinerEmptyIsAnError(t *testing.T) { + for _, c := range []string{"", " ", "\n\t"} { + code, _, errOut := runOneLinerForTest(t, c, false) + if code == 0 { + t.Errorf("-c %q: exit 0, want an error", c) + } + if !strings.Contains(errOut, "-c was given no MDL") { + t.Errorf("-c %q: error output %q does not say the command was empty", c, errOut) + } + } +} + +// mendixlabs/mxcli#1218 (2): a failing statement in a ;-separated -c stopped +// the run with nothing saying the later statements were skipped. The default +// stays fail-fast (as exec) but the stop is reported with the statement's +// position; --continue-on-error runs the rest. +// +// No project: `describe entity` fails (not connected) while `show features for +// version` needs none, so statement 3 is observable. +func TestOneLinerFailureSemantics(t *testing.T) { + const batch = "show features for version 10.0; describe entity String; show features for version 11.0;" + + code, out, errOut := runOneLinerForTest(t, batch, false) + if code == 0 { + t.Fatalf("exit 0 with a failing statement; stderr %q", errOut) + } + if !strings.Contains(out, "Features for Mendix 10.0") { + t.Errorf("statement 1 did not run (control): %q", out) + } + if strings.Contains(out, "Features for Mendix 11.0") { + t.Errorf("fail-fast ran statement 3 after statement 2 failed") + } + for _, want := range []string{"statement 2 of 3", "1 later statement(s) not run", "--continue-on-error"} { + if !strings.Contains(errOut, want) { + t.Errorf("stderr %q does not contain %q", errOut, want) + } + } + + code, out, errOut = runOneLinerForTest(t, batch, true) + if code == 0 { + t.Errorf("--continue-on-error: exit 0 although statement 2 failed") + } + if !strings.Contains(out, "Features for Mendix 11.0") { + t.Errorf("--continue-on-error did not run statement 3: %q", out) + } + if !strings.Contains(errOut, "statement 2:") || !strings.Contains(errOut, "1 failed") { + t.Errorf("--continue-on-error stderr %q does not report statement 2 and the tally", errOut) + } +} diff --git a/cmd/mxcli/syntax/features_integration.go b/cmd/mxcli/syntax/features_integration.go index 13be374c17..f09c2aaf71 100644 --- a/cmd/mxcli/syntax/features_integration.go +++ b/cmd/mxcli/syntax/features_integration.go @@ -318,9 +318,18 @@ func init() { Keywords: []string{ "create published rest", "publish rest", "rest resource", "rest operation", "microflow", "path parameter", + "query parameter", "body parameter", "import mapping", "export mapping", "commit", "grant access", "revoke access", }, - Syntax: "CREATE [OR MODIFY] PUBLISHED REST SERVICE Module.Name (\n Path: 'rest/api/v1',\n Version: '1.0.0',\n ServiceName: 'My API'\n)\n{\n RESOURCE 'name' {\n GET '' MICROFLOW Module.GetAll;\n GET '{id}' MICROFLOW Module.GetById;\n POST '' MICROFLOW Module.Create;\n }\n};\n\nALTER PUBLISHED REST SERVICE Module.Name SET Version = '2.0.0';\nALTER PUBLISHED REST SERVICE Module.Name ADD RESOURCE 'items' { ... };\nALTER PUBLISHED REST SERVICE Module.Name DROP RESOURCE 'legacy';\nDROP PUBLISHED REST SERVICE Module.Name;", + Syntax: "CREATE [OR MODIFY] PUBLISHED REST SERVICE Module.Name (\n Path: 'rest/api/v1',\n Version: '1.0.0',\n ServiceName: 'My API'\n)\n{\n RESOURCE 'name' {\n GET '' MICROFLOW Module.GetAll;\n GET '{id}' MICROFLOW Module.GetById;\n POST '' MICROFLOW Module.Create\n [IMPORT MAPPING Module.IMM] [EXPORT MAPPING Module.EMM]\n [COMMIT Yes | YesWithoutEvents | No]; -- no COMMIT clause: Yes\n }\n};\n\n" + + "-- Operation parameters come from the microflow, as Studio Pro derives them:\n" + + "-- a parameter named in the path ('{id}') -> path parameter\n" + + "-- an object or a list -> the body\n" + + "-- System.HttpRequest / HttpResponse -> none (the request and response)\n" + + "-- anything else -> query parameter\n" + + "-- each with the microflow parameter's type. Create the microflow first.\n" + + "-- A header parameter, a renamed one or a description set in Studio Pro has\n" + + "-- no MDL spelling: describe notes it, CREATE OR MODIFY / ALTER keep it.\n\nALTER PUBLISHED REST SERVICE Module.Name SET Version = '2.0.0';\nALTER PUBLISHED REST SERVICE Module.Name ADD RESOURCE 'items' { ... };\nALTER PUBLISHED REST SERVICE Module.Name DROP RESOURCE 'legacy';\nDROP PUBLISHED REST SERVICE Module.Name;", Example: "mdl 1;\nCREATE PUBLISHED REST SERVICE Module.OrderAPI (\n Path: 'rest/orders/v1',\n Version: '1.0.0',\n ServiceName: 'Order API'\n)\n{\n RESOURCE 'orders' {\n GET '' MICROFLOW Module.GetAllOrders;\n GET '{id}' MICROFLOW Module.GetOrderById;\n POST '' MICROFLOW Module.CreateOrder;\n DELETE '{id}' MICROFLOW Module.DeleteOrder;\n }\n};\n\nGRANT ACCESS ON PUBLISHED REST SERVICE Module.OrderAPI\n TO Module.User, Module.Admin;", SeeAlso: []string{"rest", "rest.consumed"}, }) diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index 26111d4009..78aaecefbc 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -180,17 +180,20 @@ func init() { "-- Two limits, both enforced rather than silently ignored:\n" + "--\n" + "-- ON ERROR CONTINUE is rejected by Mendix (CE6035) on CREATE, CHANGE,\n" + - "-- COMMIT, LOG, SHOW PAGE, CLOSE PAGE, SHOW MESSAGE and VALIDATION\n" + - "-- FEEDBACK -> MDL076. A custom handler IS accepted on all of them, and\n" + - "-- CONTINUE is fine on DECLARE, SET, RETRIEVE, DELETE and CALL MICROFLOW.\n" + + "-- COMMIT, LOG, SHOW PAGE, CLOSE PAGE, SHOW MESSAGE, VALIDATION\n" + + "-- FEEDBACK and CALL WORKFLOW -> MDL076. A custom handler IS accepted on\n" + + "-- all of them, and CONTINUE is fine on DECLARE, SET, RETRIEVE, DELETE\n" + + "-- and CALL MICROFLOW.\n" + "--\n" + "-- List operations and aggregates ($x = head $l, $n = count $l) have\n" + "-- no error handling in Mendix at all -> MDL077.\n" + "--\n" + - "-- IN A NANOFLOW only DECLARE and SET take a clause at all. CHANGE, LOG,\n" + - "-- SHOW PAGE, CLOSE PAGE, SHOW MESSAGE and VALIDATION FEEDBACK are CE6035\n" + - "-- there in EVERY form, and are refused: a nanoflow activity aborts the\n" + - "-- flow on error by default and has no transaction to roll back.\n" + + "-- IN A NANOFLOW a statement with no clause aborts the flow on error, and\n" + + "-- there is no transaction to roll back. DECLARE, SET, RETRIEVE and DELETE\n" + + "-- take every clause. CREATE, COMMIT, CALL NANOFLOW and CALL MICROFLOW take\n" + + "-- only ON ERROR WITHOUT ROLLBACK BEGIN ... END ERROR. CHANGE, LOG, SHOW\n" + + "-- PAGE, CLOSE PAGE, SHOW MESSAGE and VALIDATION FEEDBACK take none. Every\n" + + "-- other form is CE6035 and is refused (MDL091).\n" + "--\n" + "-- A handler that does NOT end in RETURN/RAISE ERROR merges back into the main\n" + "-- flow, so a variable created after the merge is out of scope on the error\n" + @@ -633,10 +636,18 @@ func init() { "@merge(x, y) -- the implicit merge that closes a split\n" + "@anchor(from: bottom, to: top, true: (…), false: (…)) -- on an IF: to = its incoming flow,\n" + " -- from = the flow leaving its closing merge\n" + + "@anchor(true: (to: top)) -- one branch's edge; either side may be left out\n" + "@caption 'text'\n@color Green\n@annotation 'a note'\n@excluded\n" + "@applyentityaccess | @applyentityaccess(false) -- DOCUMENT-level, before CREATE MICROFLOW/RULE\n" + "@annotation(id: n1, text: 'a note', position: (x, y), size: (w, h))\n" + "@annotation(id: n1) -- attaches THAT note to another activity\n\n" + + "Inside ON ERROR ... BEGIN ... END ERROR the annotations mean what they mean\n" + + "outside it: @anchor(to:) on the handler's first statement is the side the\n" + + "error edge enters, @anchor(from:) and @curve shape the edge leaving a\n" + + "statement — for the last one, the edge that rejoins the main flow.\n\n" + + "@curve has no per-branch form: on a split it curves every outgoing edge, and\n" + + "@curve(true: …) is refused (MDL060). An @anchor parameter that is not a\n" + + "side mxcli knows is refused too (MDL092) rather than left on the default.\n\n" + "An unrecognised @name is an error (MDL059): it would parse and do nothing,\n" + "so a typo of @position would silently discard the layout. That covers\n" + "DOCUMENT annotations too — a typo, or one on a document kind that does not\n" + diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index be56cade3d..46f1dee75f 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -13,7 +13,7 @@ func init() { "widget", "layout", "screen", }, Syntax: "CREATE PAGE Module.Name [FOLDER 'FolderPath']\n (\n Title: 'Page Title',\n Layout: Module.LayoutName\n [, Params: ( $Param: Module.Entity )]\n [, Url: 'page-url']\n [, Variables: ( $var: Boolean = 'true' )]\n [, PopupWidth: 800, PopupHeight: 480, PopupResizable: true]\n [, PopupCloseAction: cancelButton1]\n [, Class: 'css-class', Style: 'css: rule']\n )\n {\n -- widgets\n }", - Example: "CREATE PAGE MyModule.EditCustomer\n (\n Params: ( $Customer: MyModule.Customer ),\n Title: 'Edit Customer',\n Layout: Atlas_Core.PopupLayout,\n Class: 'container-fluid'\n )\n {\n DATAVIEW dvCustomer (DataSource: $Customer) {\n TEXTBOX txtName (Label: 'Name', Attribute: Name)\n FOOTER footer1 {\n ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary)\n ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES)\n }\n }\n };", + Example: "CREATE PAGE MyModule.EditCustomer\n (\n Params: ( $Customer: MyModule.Customer ),\n Title: 'Edit Customer',\n Layout: Atlas_Core.PopupLayout,\n Class: 'container-fluid'\n )\n {\n DATAVIEW dvCustomer (DataSource: $Customer) {\n TEXTBOX txtName (Label: 'Name', Attribute: Name)\n FOOTER {\n ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary)\n ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES)\n }\n }\n };", SeeAlso: []string{"page.create", "page.widgets", "page.alter", "snippet"}, }) @@ -129,7 +129,7 @@ LIST IMPACT OF htmlelement; "legacydatagrid", "visible", "editable", "conditional visibility", }, Syntax: "-- Containers\nLAYOUTGRID name { ROW r { COLUMN c (DesktopWidth: 6) { ... } } }\nCONTAINER name (Class: 'cls') { ... }\nCONTAINER name (OnClick: CALL MICROFLOW Module.MF) { ... } -- clickable container\nCUSTOMCONTAINER name (Class: 'cls') { ... }\nGROUPBOX name (Caption: 'C') { ... }\nTABCONTAINER name { TABPAGE tp (Caption: 'One') { ... } TABPAGE tp2 (Caption: 'Two') { ... } }\n\n" + - "-- Data widgets\nDATAVIEW name (DataSource: $Param) { ... FOOTER f { ... } }\nDATAGRID name (DataSource: DATABASE Module.Entity) { COLUMN c (Attribute: A) }\nGALLERY name (DataSource: DATABASE Module.Entity, DesktopColumns: 3) { ... }\nLISTVIEW name (DataSource: DATABASE Module.Entity) { ... }\nLISTVIEW name (...) { ... TEMPLATE FOR Module.Specialization { ... } }\n\n" + + "-- Data widgets\nDATAVIEW name (DataSource: $Param) { ... FOOTER { ... } }\nDATAGRID name (DataSource: DATABASE Module.Entity) { COLUMN c (Attribute: A) }\nGALLERY name (DataSource: DATABASE Module.Entity, DesktopColumns: 3) { ... }\nLISTVIEW name (DataSource: DATABASE Module.Entity) { ... }\nLISTVIEW name (...) { ... TEMPLATE FOR Module.Specialization { ... } }\n\n" + "-- Data grid 2 column filters go INSIDE the column's own braces\nDATAGRID dg (...) { COLUMN c (Attribute: A) { TEXTFILTER tf (Attribute: A) } }\nTEXTFILTER | NUMBERFILTER | DATEFILTER | DROPDOWNFILTER | DROPDOWNSORT\n" + "-- Match the filter to the column's type, or MxBuild refuses it: String ->\n" + "-- TEXTFILTER, Integer/Long/Decimal -> NUMBERFILTER, Date and time -> DATEFILTER,\n" + @@ -199,7 +199,7 @@ LIST IMPACT OF htmlelement; "-- Use DYNAMICTEXT with a literal Content.)\n" + "-- REFERENCESELECTOR (unsupported widget type)\n" + "-- LEGACYDATAGRID (use DATAGRID for the pluggable equivalent on Mendix 11+)", - Example: "DATAVIEW dvCustomer (DataSource: $Customer) {\n TEXTBOX txtName (Label: 'Name', Attribute: Name)\n COMBOBOX cbStatus (Label: 'Status', Attribute: Status)\n FOOTER footer1 {\n ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary)\n ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES)\n }\n}", + Example: "DATAVIEW dvCustomer (DataSource: $Customer) {\n TEXTBOX txtName (Label: 'Name', Attribute: Name)\n COMBOBOX cbStatus (Label: 'Status', Attribute: Status)\n FOOTER {\n ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary)\n ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES)\n }\n}", SeeAlso: []string{"page.create", "page.datasource"}, }) diff --git a/cmd/mxcli/testrunner/parser.go b/cmd/mxcli/testrunner/parser.go index c431bb10be..adcf2297e5 100644 --- a/cmd/mxcli/testrunner/parser.go +++ b/cmd/mxcli/testrunner/parser.go @@ -11,6 +11,8 @@ import ( "path/filepath" "regexp" "strings" + + "github.com/mendixlabs/mxcli/mdl/srctext" ) // TestCase represents a single test extracted from a test file. @@ -97,7 +99,12 @@ type FileError struct { // ParseTestFile parses a test file (.test.mdl or .test.md) and extracts test cases. func ParseTestFile(path string) (*TestSuite, error) { - content, err := os.ReadFile(path) + raw, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("reading test file: %w", err) + } + // A BOM or UTF-16 is decoded as for every other script (mendixlabs/mxcli#1253). + content, err := srctext.Decode(raw) if err != nil { return nil, fmt.Errorf("reading test file: %w", err) } diff --git a/docs-site/src/appendixes/quick-reference.md b/docs-site/src/appendixes/quick-reference.md index 7c4ccd93ca..6aefe5b177 100644 --- a/docs-site/src/appendixes/quick-reference.md +++ b/docs-site/src/appendixes/quick-reference.md @@ -403,7 +403,7 @@ CREATE PAGE MyModule.Customer_Edit TEXTBOX txtEmail (Label: 'Email', Attribute: Email) COMBOBOX cbStatus (Label: 'Status', Attribute: Status) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) } @@ -541,6 +541,7 @@ Cross-reference commands require `REFRESH CATALOG FULL` to populate reference da |---------|--------|-------| | Interactive REPL | `mxcli` | Interactive MDL shell | | Inline commands | `mxcli -p app.mpr -c "CMD1; CMD2; CMD3"` | Semicolons batch multiple commands | +| Inline, keep going | `mxcli -p app.mpr -c "CMD1; CMD2" --continue-on-error` | Without it a batch stops at the first failing statement and says how many were not run; an empty `-c` is an error | | Execute script | `mxcli exec script.mdl -p app.mpr` | Script file | | Stdin piping | `echo "CMD" \| mxcli -p app.mpr` | Quiet mode, pipe-friendly | | Check syntax | `mxcli check script.mdl` | Parse-only validation | @@ -555,6 +556,7 @@ Cross-reference commands require `REFRESH CATALOG FULL` to populate reference da | External SQL | `mxcli sql --driver postgres --dsn '...' "SELECT 1"` | Direct database query | | Docker build | `mxcli docker build -p app.mpr` | Build with PAD patching | | Docker check | `mxcli docker check -p app.mpr` | Validate with `mx check` | +| Version | `mxcli version` (or `--version`) | Build version and build time | | Diagnostics | `mxcli diag [--bundle]` | Session logs, version info | | Loop report | `mxcli diag loop-report [--json]` | Which mxcli commands a session actually ran, how often, how long | | New project | `mxcli new --version X.Y.Z` | Create project from scratch with all tooling | diff --git a/docs-site/src/examples/alter-page.md b/docs-site/src/examples/alter-page.md index ee9784e8c4..66d9d19378 100644 --- a/docs-site/src/examples/alter-page.md +++ b/docs-site/src/examples/alter-page.md @@ -44,8 +44,8 @@ ALTER PAGE CRM.Customer_Edit { ```sql ALTER PAGE CRM.Customer_Edit { - REPLACE footer1 WITH { - FOOTER newFooter { + REPLACE dvMain.footer WITH { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Success) ACTIONBUTTON btnDelete (Caption: 'Delete', Action: DELETE, ButtonStyle: Danger) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) diff --git a/docs-site/src/examples/crm-module.md b/docs-site/src/examples/crm-module.md index 4cb5ec40c2..311780eadc 100644 --- a/docs-site/src/examples/crm-module.md +++ b/docs-site/src/examples/crm-module.md @@ -142,7 +142,7 @@ CREATE PAGE CRM.Customer_NewEdit ( TEXTBOX txtEmail (Label: 'Email', Attribute: Email) TEXTBOX txtPhone (Label: 'Phone', Attribute: Phone) TEXTAREA txtNotes (Label: 'Notes', Attribute: Notes) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave ( Caption: 'Save', Action: CALL MICROFLOW CRM.ACT_Customer_Save, diff --git a/docs-site/src/examples/master-detail.md b/docs-site/src/examples/master-detail.md index 8c6f342f45..458c2db5d2 100644 --- a/docs-site/src/examples/master-detail.md +++ b/docs-site/src/examples/master-detail.md @@ -35,7 +35,7 @@ CREATE PAGE CRM.Customer_MasterDetail ( TEXTBOX txtEmail (Label: 'Email', Attribute: Email) TEXTBOX txtPhone (Label: 'Phone', Attribute: Phone) TEXTAREA txtNotes (Label: 'Notes', Attribute: Notes) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave ( Caption: 'Save', Action: SAVE CHANGES, diff --git a/docs-site/src/examples/rest-integration.md b/docs-site/src/examples/rest-integration.md index 28dbdcdee9..0f8c8309f0 100644 --- a/docs-site/src/examples/rest-integration.md +++ b/docs-site/src/examples/rest-integration.md @@ -325,7 +325,37 @@ CREATE PUBLISHED REST SERVICE Module.OrderAPI ( }; ``` -**Operation paths:** Use empty string `''` for the root, `'{paramName}'` for path parameters. Do NOT start or end with `/`. Path parameters must match a microflow parameter name exactly (case-sensitive) — e.g., `'{id}'` requires the microflow to declare `$id: String`. +**Operation paths:** Use empty string `''` for the root, `'{paramName}'` for path parameters. Do NOT start or end with `/`. Path parameters must match a microflow parameter name exactly (case-sensitive) — `'{id}'` binds the microflow's `$id`, whatever its type. + +**Operation parameters** come from the microflow, the way Studio Pro derives them. Create the microflow before the service: + +| Microflow parameter | Operation parameter | +|---|---| +| named in the path (`'{id}'`) | a path parameter | +| an object or a list | the body | +| `System.HttpRequest`, `System.HttpResponse` | none — they are the request and the response | +| anything else | a query parameter | + +Each gets the microflow parameter's type. A header parameter, a renamed parameter or a description set in Studio Pro has no MDL spelling: `describe` notes it in a comment, and `CREATE OR MODIFY` / `ALTER` on that project keep it. + +**Mappings and commit:** a body that is not a file document needs an import mapping, and a microflow returning an object or a list needs an export mapping: + +```sql +mdl 1; +CREATE OR MODIFY PUBLISHED REST SERVICE Module.OrderAPI ( + Path: 'rest/orders/v1', + Version: '1.0.0', + ServiceName: 'Order API' +) +{ + RESOURCE 'orders' { + -- COMMIT: Yes (the default without the clause) | YesWithoutEvents | No + POST '' MICROFLOW Module.PRS_CreateOrder + IMPORT MAPPING Module.IMM_Order EXPORT MAPPING Module.EMM_Order + COMMIT YesWithoutEvents; + } +}; +``` ### Multiple Resources diff --git a/docs-site/src/examples/validation.md b/docs-site/src/examples/validation.md index f34bc58b98..9cd987e622 100644 --- a/docs-site/src/examples/validation.md +++ b/docs-site/src/examples/validation.md @@ -84,7 +84,7 @@ CREATE PAGE Sales.Order_Edit ( DATEPICKER dpDelivery (Label: 'Delivery Date', Attribute: DeliveryDate) TEXTBOX txtQuantity (Label: 'Quantity', Attribute: Quantity) TEXTBOX txtDiscount (Label: 'Discount %', Attribute: DiscountPercent) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave ( Caption: 'Save', Action: CALL MICROFLOW Sales.ACT_Order_Save, diff --git a/docs-site/src/language/nanoflows.md b/docs-site/src/language/nanoflows.md index b06c8d5497..64a0734c7e 100644 --- a/docs-site/src/language/nanoflows.md +++ b/docs-site/src/language/nanoflows.md @@ -124,7 +124,7 @@ The following activities are server-only and cannot be used in nanoflows: - `CALL EXTERNAL ACTION` — external actions are server-side - All **workflow actions** (call/open workflow, set task outcome, user task, etc.) -> **Note:** Per-action error handling (`on error continue`) IS supported in nanoflows. Only `ErrorEvent` (raise error as a standalone flow action) is forbidden. Note that `on error rollback` is syntactically valid but only rolls back in-memory changes — nanoflows have no database transactions. +> **Note:** Per-action error handling is supported in nanoflows, per activity. Only `ErrorEvent` (raise error as a standalone flow action) is forbidden. A nanoflow has no database transaction, so an activity without a clause aborts the flow on error. `declare`, `set`, `retrieve` and `delete` take every clause; `create`, `commit`, `call nanoflow` and `call microflow` take only `on error without rollback begin … end error`; `change`, `log`, `show page`, `close page`, `show message` and `validation feedback` take none. Every other form is CE6035 at build time, and mxcli refuses it (MDL091) — measured on Mendix 11.14.0. ## SHOW and DESCRIBE diff --git a/docs-site/src/language/page-patterns.md b/docs-site/src/language/page-patterns.md index db6a0eab6a..2e1b000022 100644 --- a/docs-site/src/language/page-patterns.md +++ b/docs-site/src/language/page-patterns.md @@ -63,7 +63,7 @@ CREATE PAGE MyModule.Customer_Edit FOLDER 'Customers' TEXTBOX txtPhone (Label: 'Phone', Attribute: Phone) COMBOBOX cbStatus (Label: 'Status', Attribute: Status) CHECKBOX cbActive (Label: 'Active', Attribute: IsActive) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) } @@ -101,7 +101,7 @@ CREATE PAGE MyModule.Customer_Detail FOLDER 'Customers' } } } - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnEdit ( Caption: 'Edit', Action: SHOW PAGE MyModule.Customer_Edit, @@ -148,7 +148,7 @@ CREATE PAGE MyModule.Product_MasterDetail FOLDER 'Products' TEXTBOX txtDescription (Label: 'Description', Attribute: Description) TEXTBOX txtPrice (Label: 'Price', Attribute: Price) COMBOBOX cbCategory (Label: 'Category', Attribute: Category) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) } } @@ -191,7 +191,7 @@ CREATE PAGE MyModule.Order_MasterDetail FOLDER 'Orders' COLUMN (Attribute: LineTotal, Caption: 'Total', Alignment: right) } - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnEdit ( Caption: 'Edit Order', Action: SHOW PAGE MyModule.Order_Edit, @@ -252,7 +252,7 @@ CREATE PAGE HR.Employee_Edit FOLDER 'Employees' TEXTBOX txtDept (Label: 'Department', Attribute: Department) DATEPICKER dpHireDate (Label: 'Hire Date', Attribute: HireDate) TEXTBOX txtEmail (Label: 'Email', Attribute: Email) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) } diff --git a/docs-site/src/language/page-structure.md b/docs-site/src/language/page-structure.md index dd6a367e92..24a4364d7c 100644 --- a/docs-site/src/language/page-structure.md +++ b/docs-site/src/language/page-structure.md @@ -48,7 +48,7 @@ CREATE PAGE MyModule.Customer_Edit { DATAVIEW dvCustomer (DataSource: $Customer) { TEXTBOX txtName (Label: 'Name', Attribute: Name) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) } diff --git a/docs-site/src/language/pages.md b/docs-site/src/language/pages.md index b85405f7c6..82ca06ba78 100644 --- a/docs-site/src/language/pages.md +++ b/docs-site/src/language/pages.md @@ -58,7 +58,7 @@ CREATE PAGE MyModule.Customer_Edit DATAVIEW dvCustomer (DataSource: $Customer) { TEXTBOX txtName (Label: 'Name', Attribute: Name) TEXTBOX txtEmail (Label: 'Email', Attribute: Email) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) } diff --git a/docs-site/src/language/widget-types.md b/docs-site/src/language/widget-types.md index 86a4cf83bb..c6d5a70e9e 100644 --- a/docs-site/src/language/widget-types.md +++ b/docs-site/src/language/widget-types.md @@ -87,7 +87,7 @@ DATAVIEW dvCustomer (DataSource: $Customer) { TEXTBOX txtName (Label: 'Name', Attribute: Name) TEXTBOX txtEmail (Label: 'Email', Attribute: Email) COMBOBOX cbStatus (Label: 'Status', Attribute: Status) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) } @@ -313,11 +313,11 @@ COMBOBOX cbStatus (Label: 'Status', Attribute: Status) ### REFERENCESELECTOR -Dropdown for selecting an associated object via a reference association: - -```sql -REFERENCESELECTOR rsCategory (Label: 'Category', Attribute: Category) -``` +The classic reference selector (`Forms$ReferenceSelector`) is a built-in Forms +widget mxcli has no writer for. The keyword parses, and `check` refuses it as +**MDL-WIDGET38** rather than letting `exec` stop at the page. Select an +associated object with a `COMBOBOX` over the association instead; it needs its +own `datasource:` (CE0642 without one) and a caption attribute. ## Display Widgets @@ -433,16 +433,17 @@ DATAVIEW dvOrder (DataSource: $Order) { DYNAMICTEXT txtOrderTitle (Content: 'Order #{1}', Attribute: OrderId) } TEXTBOX txtStatus (Label: 'Status', Attribute: Status) - FOOTER ftr1 { ... } + FOOTER { ... } } ``` ### FOOTER -Footer section of a DataView. Typically contains save/cancel buttons: +Footer section of a DataView. Typically contains save/cancel buttons. It +stores no name of its own; `ALTER PAGE` addresses it as `.footer`: ```sql -FOOTER footer1 { +FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) } diff --git a/docs-site/src/reference/capabilities.md b/docs-site/src/reference/capabilities.md index 063d7038fa..790bf5f53e 100644 --- a/docs-site/src/reference/capabilities.md +++ b/docs-site/src/reference/capabilities.md @@ -142,7 +142,7 @@ Everything mxcli can do, organized by use case. |---|---|---| | Design properties (Atlas v3) | Requires Mendix 11.0+ | Use CSS classes on 10.x | | REST query parameters | Requires Mendix 11.0+ | Build query string manually on 10.x | -| Concurrent editing | Not supported | Close Studio Pro before mxcli writes | +| Concurrent editing | Not supported — a file-based write is refused while Studio Pro has the project open (`.mpr.lock` present) | Close Studio Pro before mxcli writes, or write through it with `--mcp`; `exec --force` overrides | | Widget template drift | CE0463 on version mismatch | MPK augmentation handles most cases | | Marketplace module update | Existing modules are reported, not updated in place | Update via Studio Pro (preserves local edits and entity IDs) | | 47 of 52 metamodel domains | Not yet implemented | REST, OData write, etc. pending | diff --git a/docs-site/src/reference/page/create-page.md b/docs-site/src/reference/page/create-page.md index 829fde60b6..09260ab888 100644 --- a/docs-site/src/reference/page/create-page.md +++ b/docs-site/src/reference/page/create-page.md @@ -258,7 +258,7 @@ CREATE PAGE MyModule.Customer_Edit TEXTBOX txtEmail (Label: 'Email', Attribute: Email) COMBOBOX cbStatus (Label: 'Status', Attribute: Status) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) } @@ -378,7 +378,7 @@ CREATE PAGE MyModule.AdvancedForm CONTAINER cntAdvanced (Visible: '$showAdvanced') { TEXTAREA taNotes (Label: 'Notes', Attribute: Notes) } - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) } diff --git a/docs-site/src/reference/page/create-snippet.md b/docs-site/src/reference/page/create-snippet.md index 39b07e991d..99623d447f 100644 --- a/docs-site/src/reference/page/create-snippet.md +++ b/docs-site/src/reference/page/create-snippet.md @@ -102,7 +102,7 @@ CREATE PAGE MyModule.Customer_Edit { DATAVIEW dvCustomer (DataSource: $Customer) { SNIPPETCALL snpAddress (Snippet: MyModule.AddressFields) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) } } diff --git a/docs-site/src/reference/query/describe-page.md b/docs-site/src/reference/query/describe-page.md index c6c91421b2..7f2fabd52e 100644 --- a/docs-site/src/reference/query/describe-page.md +++ b/docs-site/src/reference/query/describe-page.md @@ -38,7 +38,7 @@ CREATE PAGE Sales.Customer_Edit DATAVIEW dvCustomer (DataSource: $Customer) { TEXTBOX txtName (Label: 'Name', Attribute: Name) TEXTBOX txtEmail (Label: 'Email', Attribute: Email) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) } diff --git a/docs-site/src/tools/diff.md b/docs-site/src/tools/diff.md index 1729a56dbb..ceced6462a 100644 --- a/docs-site/src/tools/diff.md +++ b/docs-site/src/tools/diff.md @@ -4,7 +4,8 @@ mxcli provides two diff commands for comparing MDL scripts against project state ## mxcli diff -Compares an MDL script against the current project state, showing what would change if the script were executed. This is a dry-run preview. +Shows what `mxcli exec` would change in the project if it ran the script, without +changing the project. **Usage:** @@ -12,82 +13,89 @@ Compares an MDL script against the current project state, showing what would cha mxcli diff -p app.mpr changes.mdl ``` -This shows: -- Elements that would be created (new entities, microflows, pages) -- Elements that would be modified (changed attributes, altered properties) -- Elements that would be removed (DROP statements) +### How it works: diff runs exec -Use `mxcli diff` to review changes before applying them, especially when working with AI-generated scripts. +`diff` copies the project to a scratch folder, runs the script there with exec's +own code — the same pre-flight checks, the statements in order, under the +script's language header — and then compares the copy with the project unit by +unit. What it reports is therefore exactly what `exec` would write: there is no +second judgement to disagree with exec's. The project itself is only read, and +the scratch copy is deleted afterwards (it leaves out `.git`, `deployment`, +`releases` and `node_modules`). -### What diff does not compare - -`diff` compares entities, view entities, enumerations, associations, microflows -and nanoflows. Every other statement — `grant`, `create constant`, pages, -navigation, settings — is listed under **Not compared** after the summary: +Each unit exec would add, rewrite, move or remove is shown as the `describe` of +the document before and after. A domain model is shown per entity and +association, module security per module role, and project security per user +role and demo user. This covers every document kind — pages, snippets, +layouts, navigation, settings, security, folders — and statements of every +kind, including `grant`, `alter`, `move` and `drop`. ``` -Summary: 0 new, 0 modified, 1 unchanged +--- Page.MyFirstModule.Home_Web (current) ++++ Page.MyFirstModule.Home_Web (script) +@@ -1,6 +1,6 @@ + create or modify page MyFirstModule.Home_Web ( + Title: 'Homepage', +- Layout: Atlas_Core.Atlas_TopBar, ++ Layout: Atlas_Core.Atlas_Default, + PopupResizable: true + ) { + +Summary: 0 new, 1 modified, 0 removed — exec would write 1 unit(s) +``` + +A unit exec rewrites although its description does not change — a property +`describe` does not print, a translation, a move to another folder — is listed +on one line with what changes, so a write is never hidden behind identical MDL: -Not compared (2 statement(s)) — diff has no comparison for these, -so they are absent from the summary above, not unchanged: - create constant x1 - grant microflow access x1 +``` +Modified: Page MyFirstModule.Home_Web: changed: CanvasHeight, CanvasWidth +Modified: Microflow Shop.ACT_Apply: moved to 'Shop/Archive' ``` -Read that list. The counts describe only the statements diff understands, so a -script made entirely of the others summarises as all zeros — which means "not -examined", not "no change". Those statements were previously skipped without a -word, so the summary looked like a clean bill of health for a script that would -add documents (#997). - -### Both sides go through one renderer - -The project side and the script side are rendered by the same describer -`describe microflow` uses, so an unmodified `describe` dump diffs as -**unchanged**. Before this, the script side had a renderer of its own that -covered 18 of 43 activity types and silently emitted nothing for the rest, so a -java-action call, a `download file` or a canvas annotation appeared as a -deletion in a script that changed nothing at all. - -### `create or modify` of a flow: diff reaches exec's verdict - -`exec` applies `create or modify microflow|nanoflow` on a flow that exists as a -patch spliced into the stored flow, and under `mdl 1;` it refuses a change the -splice cannot make (a change inside a loop body, a redrawn connector) rather -than rebuild the flow. `diff` runs the same patch planning, under the script's -own language header, without writing, so the two agree: - -- a statement `exec` would refuse is listed as **Refused**, with the reason, - and counted as refused in the summary; -- a statement whose patch is empty — nothing `exec` would write — is - **unchanged**, whatever the two renderings differ in (surrounding whitespace, - a bracketed `where [ … ]` against the bare form describe prints). An - expression is compared by its tokens: the case of a keyword (`AND`, `and`) - and the whitespace and line breaks between tokens — a member list laid out - over several lines — are not a change, and neither is `Long` for the one - Integer/Long type a flow stores (ako/mxcli#886); what is inside a string - literal is compared as written. A statement - that builds the flow that is stored is unchanged however that flow is - described: a guard clause `if … then return …; end if; return …;` that - describe prints as `if … then … else … end if`, a nested guard it prints with - a `join`/`merge` pair, a long flow it prints as crossed branches - (ako/mxcli#859); -- a statement `exec` would write although its MDL renders as stored — a - `folder` clause naming another folder, which `exec` applies as a move — is - **modified**, with what `exec` would write: - - ``` - Modified: Microflow Shop.ACT_Apply: its MDL renders as stored, but exec would write it: moved to folder 'Archive' - ``` +Files exec writes next to the model (Java sources, theme files) are listed as +`New file:` / `Modified file:`. + +### What diff no longer reports + +Because the verdict is exec's own, a script exec has already applied diffs as +`exec would write nothing`. Before (ako/mxcli#907), diff compared the script's +text with the stored document's description and reported phantom changes +exec never makes — `Boolean` against the stored `Boolean default false`, +`String` against `String(unlimited)`, a position, a re-laid-out flow — and +did not compare pages, translations or layouts at all. + +### Refusals and errors + +- A script exec's pre-flight refuses (a `check` error, an unresolved + reference, a name clash) is reported as **Refused**, with the same report + exec prints, and nothing is written. `--no-check` skips the pre-flight, as + it does for exec. +- A statement exec stops at — a plain `create` of a document that exists + (ako/mxcli#807), a `create or modify` of a flow the splice cannot make + under `mdl 1;` — is reported as **Refused** with exec's error, after the + changes exec makes before it. `--continue-on-error` runs every statement, + as it does for exec. +- Statements that depend on earlier ones (a flow calling a microflow the + script creates first) are diffed like any other, because the earlier ones + run first (ako/mxcli#856). ``` -Refused: Microflow Shop.ACT_Apply: exec would refuse this statement and write nothing: the Loop at (700, 200) changes inside its body; … +Refused: exec would stop at this error, having written nothing: entity already exists: Shop.Order — … -Summary: 0 new, 0 modified, 0 unchanged, 1 refused +Summary: 0 new, 0 modified, 0 removed — exec would write nothing ``` -Before this, `diff` compared renderings only and said "unchanged" for a -statement `exec` then refused (ako/mxcli#839). +`diff` always runs the script with the file engine on the scratch copy, also +under `--mcp`: the script is executed for real, and only the copy may receive +it. `--exec-output` prints what exec reports while it runs. + +For the same reason a script may not reach past the copy. A `connect` to the +`-p` project (a headerless script may hold one, and so may a script it runs +with `execute script`) is followed on the copy; a `connect` to any other +project, a `sql ` and an `import from` are refused with an error, +because diff has no copy of that project or database and does not run them for +real. ## mxcli diff-local diff --git a/docs-site/src/tutorial/create-page.md b/docs-site/src/tutorial/create-page.md index 8b9e69fdff..8250ab0a71 100644 --- a/docs-site/src/tutorial/create-page.md +++ b/docs-site/src/tutorial/create-page.md @@ -84,7 +84,7 @@ CREATE PAGE MyModule.Product_Edit TEXTBOX txtPrice (Label: 'Price', Attribute: Price) CHECKBOX cbActive (Label: 'Active', Attribute: IsActive) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) } diff --git a/docs-site/src/tutorial/describe-search.md b/docs-site/src/tutorial/describe-search.md index 694667b7a8..5128f3d9c5 100644 --- a/docs-site/src/tutorial/describe-search.md +++ b/docs-site/src/tutorial/describe-search.md @@ -93,7 +93,7 @@ CREATE PAGE MyFirstModule.Customer_Edit TEXTBOX txtEmail (Label: 'Email', Attribute: Email) TEXTBOX txtPhone (Label: 'Phone', Attribute: Phone) - FOOTER footer1 { + FOOTER { ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE CHANGES, ButtonStyle: Primary) ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL CHANGES) } diff --git a/docs-site/src/tutorial/opening-project.md b/docs-site/src/tutorial/opening-project.md index e7e3bdcf50..6c7469803a 100644 --- a/docs-site/src/tutorial/opening-project.md +++ b/docs-site/src/tutorial/opening-project.md @@ -14,6 +14,8 @@ mxcli -p /path/to/app.mpr -c "LIST MODULES" This opens the project in read-only mode, runs the command, and exits. The `-p` flag works with all mxcli subcommands. +`-c` takes several statements separated by `;`. Like `mxcli exec`, it stops at the first statement that fails, prints `statement 2 of 3: …` and how many later statements were not run, and exits non-zero; add `--continue-on-error` to run every statement and report each failure. An empty `-c ""` is an error: it never opens the interactive REPL, so a script that generates an empty command list cannot hang waiting on input. + ## Opening in the REPL You can also open a project from inside the interactive REPL: diff --git a/docs-site/src/tutorial/quickstart.md b/docs-site/src/tutorial/quickstart.md index 0e841f9ca8..5248711305 100644 --- a/docs-site/src/tutorial/quickstart.md +++ b/docs-site/src/tutorial/quickstart.md @@ -136,4 +136,4 @@ mxcli setup mxbuild -p your-app.mpr **"CGO not available"** -- mxcli uses pure Go SQLite. No C compiler needed. If you see CGO errors, ensure you're using the official pre-built binary or a `make build` from source. -**Project won't open in Studio Pro after changes** -- Close Studio Pro before running mxcli write commands, then reopen. See [F4 sync support](../appendixes/version-compatibility.md) for details. +**"refusing to write …: Studio Pro has this project open"** -- Close the project in Studio Pro before running mxcli write commands, then reopen it. Studio Pro does not reload the model from disk, so its next save would silently discard mxcli's changes. If Studio Pro is not running, the `.mpr.lock` was left behind by a crash: delete it, or pass `--force` to `exec`. diff --git a/docs-site/src/tutorial/validation.md b/docs-site/src/tutorial/validation.md index af79093f85..4da454b946 100644 --- a/docs-site/src/tutorial/validation.md +++ b/docs-site/src/tutorial/validation.md @@ -173,7 +173,7 @@ Before executing, you can preview what a script would change: mxcli diff -p app.mpr changes.mdl ``` -This compares the script against the current project state and shows what would be created, modified, or left unchanged. It does not modify the project. +This runs the script with exec's own code on a scratch copy of the project and shows every document exec would create, modify or remove, each as its `describe` before and after. It does not modify the project. ## What mxcli check catches automatically diff --git a/docs-wiki/bug-patterns/duplicate-resolver-drift.md b/docs-wiki/bug-patterns/duplicate-resolver-drift.md index ca5fb1856e..d190a4276a 100644 --- a/docs-wiki/bug-patterns/duplicate-resolver-drift.md +++ b/docs-wiki/bug-patterns/duplicate-resolver-drift.md @@ -66,6 +66,15 @@ interface already exposed the accessor the switches were standing in for (`GetFlow`), which is the tell for this variant: a hand-written switch over an interface's implementations answers a question the interface answers already. +*A preview versus the command it previews.* `mxcli diff` predicted exec by +rendering each statement as MDL and comparing it with the stored document's +description. Every normalisation exec applies — implicit defaults, positions, +the splice verdict, write elision — had to be re-derived by that renderer, and +each round of fixes (#997, #794, #839) synchronised one more spot while the +next rehearsal found phantoms on 66 scripts (#907). The remedy was the general +one taken literally: diff now runs exec itself on a scratch copy and compares +units, so there is no second answer left to drift. + **The tell is that the fix for the reported instance is obviously incomplete.** When a symptom's cause is "this switch was missing a case", the next question is how many other switches answer the same question — the answer has repeatedly been diff --git a/docs-wiki/bug-patterns/mutator-addressing.md b/docs-wiki/bug-patterns/mutator-addressing.md index cd4bab9b31..77ee8bb618 100644 --- a/docs-wiki/bug-patterns/mutator-addressing.md +++ b/docs-wiki/bug-patterns/mutator-addressing.md @@ -45,7 +45,12 @@ invalid document. positionally (`layoutContainer.top`), reusing the dotted reference that also serves columns and disambiguated the same way. Only `INSERT INTO` accepts one: BEFORE and AFTER position a widget among siblings, and treating them as INTO -would silently put widgets somewhere the script did not ask for. +would silently put widgets somewhere the script did not ask for. A data view's +footer is the same thing (`dvMain.footer`); it went unaddressable for months +because the builder accepted a name for it and dropped it, and the grammar's +own example used describe's invented `footer1` — a name nothing resolved. A +name written where the model keeps none should be refused or deprecated at +parse time (MDL-DEPR005), not silently discarded by the builder. **Property lookup is per-shape, and the shapes differ.** A button's text is a `CaptionTemplate`, not a `Caption`. A column's value kind comes from the diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 4ae3eb277a..8f2c0a7941 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -1575,7 +1575,7 @@ create page MyModule.Customer_Edit textbox txtEmail (label: 'Email', attribute: Email) combobox cbStatus (label: 'Status', attribute: status) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } diff --git a/docs/05-mdl-specification/01-language-reference.md b/docs/05-mdl-specification/01-language-reference.md index f90249a433..9e43142819 100644 --- a/docs/05-mdl-specification/01-language-reference.md +++ b/docs/05-mdl-specification/01-language-reference.md @@ -796,7 +796,7 @@ create page MyModule.Customer_Edit dataview dvCustomer (datasource: $Customer) { textbox txtName (label: 'Name', attribute: Name) textbox txtEmail (label: 'Email', attribute: Email) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save_changes, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel_changes) } diff --git a/mdl-examples/bug-tests/1140-flow-arg-page-parameter.mdl b/mdl-examples/bug-tests/1140-flow-arg-page-parameter.mdl index 3edf470750..6b4bf129d2 100644 --- a/mdl-examples/bug-tests/1140-flow-arg-page-parameter.mdl +++ b/mdl-examples/bug-tests/1140-flow-arg-page-parameter.mdl @@ -86,7 +86,7 @@ create or modify page CustomModule.PANEL_BufferDefinition_Edit ( ) { placeholder Main { dataview dvEdit (DataSource: $Dto, FormOrientation: Vertical) { - footer footer1 { + footer { actionbutton btnSave ( Caption: 'Save', Action: call nanoflow CustomModule.ACT_BufferDefinition_SaveEdit_NF( diff --git a/mdl-examples/bug-tests/571-published-rest-parameters-and-mappings.mdl b/mdl-examples/bug-tests/571-published-rest-parameters-and-mappings.mdl new file mode 100644 index 0000000000..600bff301e --- /dev/null +++ b/mdl-examples/bug-tests/571-published-rest-parameters-and-mappings.mdl @@ -0,0 +1,94 @@ +mdl 1; +-- ============================================================================ +-- ako/mxcli#571, mendixlabs/mxcli#1206: published REST operation parameters +-- and mapping bindings +-- ============================================================================ +-- +-- Before the fix only the path's {name} placeholders were written, each as a +-- String: the query and body parameters failed mx check with CE0350, an +-- Integer {id} with CE6539. The import mapping, export mapping and commit +-- clauses parsed and were thrown away, so a body that is not a file document +-- failed with CE0350 and a microflow returning an object with CE0354. +-- +-- Usage: +-- mxcli exec mdl-examples/bug-tests/571-published-rest-parameters-and-mappings.mdl -p app.mpr +-- mxcli docker check -p app.mpr -> 0 errors +-- mxcli exec (the same script again) -> writes nothing +-- +-- The operation parameters are derived from each microflow, as Studio Pro +-- derives them: a parameter named in the path is a path parameter, an object +-- or a list is the body, System.HttpRequest / HttpResponse are left out, and +-- anything else is a query parameter, each with the microflow parameter's type. + +create module RestQ; + +create persistent entity RestQ.Upload extends System.FileDocument ( + Label: String(100) +); + +create persistent entity RestQ.Item ( + Code: String(20), + Quantity: Integer +); + +create json structure RestQ.JSON_Item + sample '{"code": "A1", "quantity": 3}'; + +create import mapping RestQ.IMM_Item + with json structure RestQ.JSON_Item +{ + create RestQ.Item { + Code = code, + Quantity = quantity + } +}; + +create export mapping RestQ.EMM_Item + with json structure RestQ.JSON_Item +{ + RestQ.Item { + code = Code, + quantity = Quantity + } +}; + +create microflow RestQ.GetStatus ($orderNumber: String, $count: Integer, $httpRequest: System.HttpRequest) +returns String as $Result +begin + declare $Result String = $orderNumber + ':' + toString($count); + return $Result; +end; + +create microflow RestQ.GetById ($id: Integer, $verbose: Boolean) +returns String as $Result +begin + declare $Result String = toString($id); + return $Result; +end; + +create microflow RestQ.PutFile ($file: RestQ.Upload) +returns String as $Result +begin + declare $Result String = 'ok'; + return $Result; +end; + +create microflow RestQ.PostItem ($item: RestQ.Item) +returns RestQ.Item as $item +begin + return $item; +end; + +create published rest service RestQ.Orders ( + Path: 'rest/orders/v1', + Version: '1.0.0', + ServiceName: 'Orders' +) +{ + resource 'orders' { + get 'status' microflow RestQ.GetStatus; + get 'items/{id}' microflow RestQ.GetById; + post 'upload' microflow RestQ.PutFile; + post 'item' microflow RestQ.PostItem import mapping RestQ.IMM_Item export mapping RestQ.EMM_Item commit YesWithoutEvents; + } +}; diff --git a/mdl-examples/bug-tests/762-813-dataview-properties.mdl b/mdl-examples/bug-tests/762-813-dataview-properties.mdl index bb89536389..67d45f4cb5 100644 --- a/mdl-examples/bug-tests/762-813-dataview-properties.mdl +++ b/mdl-examples/bug-tests/762-813-dataview-properties.mdl @@ -67,7 +67,7 @@ create or modify page Issue762.DVTest ( -- wins over the `footer { … }` block, and hiding must not discard the widgets. dataview dvHidden (datasource: $Thing, showFooter: false) { textbox t5 (attribute: Name) - footer f1 { + footer { dynamictext ft1 (content: 'hidden footer') } } diff --git a/mdl-examples/bug-tests/microflow-workflow-action-position-roundtrip.mdl b/mdl-examples/bug-tests/microflow-workflow-action-position-roundtrip.mdl index cbbdb544a6..b8f2bfc6d5 100644 --- a/mdl-examples/bug-tests/microflow-workflow-action-position-roundtrip.mdl +++ b/mdl-examples/bug-tests/microflow-workflow-action-position-roundtrip.mdl @@ -39,7 +39,7 @@ end; create microflow BugWfPos.Mixed ( $Workflow: System.Workflow ) returns boolean as $Ok begin lock workflow $Workflow; - unlock workflow all; + unlock workflow $Workflow unpause all; workflow operation pause $Workflow; return true; end; diff --git a/mdl-examples/bug-tests/move-entity-503-preserves-storage-guids.mdl b/mdl-examples/bug-tests/move-entity-503-preserves-storage-guids.mdl index 898dec6bbd..fd072be987 100644 --- a/mdl-examples/bug-tests/move-entity-503-preserves-storage-guids.mdl +++ b/mdl-examples/bug-tests/move-entity-503-preserves-storage-guids.mdl @@ -69,9 +69,9 @@ CREATE MODULE BugMoveGuid503B; -- -- Measured with a binary predating both fixes here, so it is a third defect on this -- command and not something these fixes introduced — but it is why this file keeps --- one endpoint of each pair put. Filed as ako/mxcli#628: MoveEntity scans --- AssociationsItems() and never CrossAssociationsItems(), so a cross-association --- that already exists is invisible to the move. +-- one endpoint of each pair put. Filed and fixed as ako/mxcli#628 (MoveEntity +-- never looked at CrossAssociationsItems()); the both-endpoints sequences are +-- exercised in move-entity-628-existing-cross-associations.mdl. -- Pair 1 — the TO side moves, so the cross-association STAYS in the source module -- and its qualified name does not change. diff --git a/mdl-examples/bug-tests/move-entity-628-existing-cross-associations.mdl b/mdl-examples/bug-tests/move-entity-628-existing-cross-associations.mdl new file mode 100644 index 0000000000..c25d1eacb9 --- /dev/null +++ b/mdl-examples/bug-tests/move-entity-628-existing-cross-associations.mdl @@ -0,0 +1,58 @@ +mdl 1; +-- ============================================================================ +-- ako/mxcli#628 — MOVE ENTITY must see the cross-associations that already exist +-- ============================================================================ +-- +-- MoveEntity converted the plain associations of the moved entity and never +-- looked at existing cross-associations. Moving the SECOND endpoint of an +-- association to another module — the normal way to reorganise a module one +-- entity at a time — left a cross-association whose ParentPointer named an +-- element absent from its unit, and Studio Pro could no longer open the project: +-- +-- System.AggregateException: The given key '' was not present in the dictionary. +-- +-- Three shapes, each below: +-- 1. both endpoints end up in one module -> a plain association again; +-- 2. the FROM end moves on -> the cross-association travels with it; +-- 3. the TO end moves on -> the cross-association elsewhere is re-pointed. +-- +-- mxcli exec move-entity-628-existing-cross-associations.mdl -p app.mpr +-- mxcli docker check -p app.mpr +-- +-- Expected: every statement succeeds and the project checks clean. Afterwards +-- describe association Bug628B.Child_Parent; -> from Bug628B.Child to Bug628B.Parent +-- describe association Bug628E.C2_P2; -> from Bug628E.C2 to Bug628D.P2 +-- describe association Bug628C.C3_P3; -> from Bug628C.C3 to Bug628E.P3 +-- +-- The storage GUID carry through each conversion is asserted on Studio +-- Pro-authored content in mdl/backend/modelsdk/issue628_move_cross_assoc_test.go; +-- in a script like this one every element is mxcli-created (GUID == $ID), so a +-- lost GUID is not observable here. +-- ============================================================================ + +create module Bug628A; +create module Bug628B; +create module Bug628C; +create module Bug628D; +create module Bug628E; + +-- Shape 1 (the issue's reproduction). +create or modify persistent entity Bug628A.Parent ( Code: String(50) ); +create or modify persistent entity Bug628A.Child ( Descr: String(100) ); +create or modify association Bug628A.Child_Parent from Bug628A.Child to Bug628A.Parent type Reference; +move entity Bug628A.Parent to Bug628B; +move entity Bug628A.Child to Bug628B; + +-- Shape 2. +create or modify persistent entity Bug628C.P2 ( Code: String(50) ); +create or modify persistent entity Bug628C.C2 ( Descr: String(100) ); +create or modify association Bug628C.C2_P2 from Bug628C.C2 to Bug628C.P2 type Reference; +move entity Bug628C.P2 to Bug628D; +move entity Bug628C.C2 to Bug628E; + +-- Shape 3. +create or modify persistent entity Bug628C.P3 ( Code: String(50) ); +create or modify persistent entity Bug628C.C3 ( Descr: String(100) ); +create or modify association Bug628C.C3_P3 from Bug628C.C3 to Bug628C.P3 type Reference; +move entity Bug628C.P3 to Bug628D; +move entity Bug628D.P3 to Bug628E; diff --git a/mdl-examples/bug-tests/pages-541-roundtrip-property-drift.mdl b/mdl-examples/bug-tests/pages-541-roundtrip-property-drift.mdl index 5ff4d2ed31..d33546fbfa 100644 --- a/mdl-examples/bug-tests/pages-541-roundtrip-property-drift.mdl +++ b/mdl-examples/bug-tests/pages-541-roundtrip-property-drift.mdl @@ -89,7 +89,7 @@ create or modify page Issue541.RuleAction_NewEdit ( -- All four actions whose DisabledDuringExecution was missing, plus -- save_changes' SyncAutomatically (class 2). - footer footerButtons { + footer { actionbutton btnSave (caption: 'Save', action: save changes close page, buttonstyle: Success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes close page) actionbutton btnClose (caption: 'Close', action: close page) diff --git a/mdl-examples/bug-tests/pages-550-describe-input-properties.mdl b/mdl-examples/bug-tests/pages-550-describe-input-properties.mdl index a7ef137bf2..c0e6f9ab24 100644 --- a/mdl-examples/bug-tests/pages-550-describe-input-properties.mdl +++ b/mdl-examples/bug-tests/pages-550-describe-input-properties.mdl @@ -80,7 +80,7 @@ create or modify page Issue550.ChangePasswordForm ( ValidationMessage: 'The password cannot be empty.' ) - footer footerButtons { + footer { actionbutton okButton1 (caption: 'OK', action: call microflow Issue550.ACT_ChangePassword) actionbutton cancelButton1 (caption: 'Cancel', action: cancel changes close page) } diff --git a/mdl-examples/bug-tests/workflow-actions-describe.mdl b/mdl-examples/bug-tests/workflow-actions-describe.mdl index fbbd5c7be3..8908b17139 100644 --- a/mdl-examples/bug-tests/workflow-actions-describe.mdl +++ b/mdl-examples/bug-tests/workflow-actions-describe.mdl @@ -23,12 +23,11 @@ mdl 1; -- and DESCRIBE re-quoted what it read: 'cancelled' became '''cancelled''', -- then seven quotes a side, doubling on every round trip. -- --- NOT covered here: `lock workflow all` / `unlock workflow all`. Those write an --- activity mxbuild rejects with CE1825 "The 'Workflow' property is required" — --- a lock always needs a specific workflow definition. Writing the selection --- anyway does not help, because with `all` there is no workflow to name. That --- is a separate, still-open defect; the variable form below is the one that --- builds. +-- `lock workflow all` / `unlock workflow all` are refused (MDL-WF17): they wrote +-- an activity mxbuild rejects with CE1825 "The 'Workflow' property is +-- required" — a lock always names its workflow definition. Studio Pro's "Pause +-- instances" / "Unpause instances" options are `pause all` / `unpause all` after +-- the workflow (mendixlabs/mxcli#870). -- -- Verified on Mendix 11.13.0: 0 errors, and describe→exec→describe is stable. -- ============================================================================ @@ -67,7 +66,11 @@ begin $Records = get workflow activity records $Wf; open workflow $Wf; lock workflow $WfDef; + lock workflow $WfDef pause all; unlock workflow $WfDef; + unlock workflow $WfDef unpause all; + lock workflow W.ApproveOrder pause all; + unlock workflow W.ApproveOrder; workflow operation pause $Wf; workflow operation abort $Wf reason 'cancelled by user'; end; diff --git a/mdl-examples/doctype-tests/03-page-examples.mdl b/mdl-examples/doctype-tests/03-page-examples.mdl index 55c7fc3a60..e4d69081fb 100644 --- a/mdl-examples/doctype-tests/03-page-examples.mdl +++ b/mdl-examples/doctype-tests/03-page-examples.mdl @@ -280,7 +280,7 @@ create page PgTest.P006_DataForm folder 'Forms' radiobuttons radioButtons1 (label: 'Is active', attribute: IsActive) datepicker datePicker1 (label: 'Create date', attribute: CreateDate) - footer footer1 { + footer { actionbutton actionButton1 (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton actionButton2 (caption: 'Cancel', action: cancel changes, buttonstyle: default) } @@ -387,7 +387,7 @@ create page PgTest.P007_DataForm_2 folder 'Forms' textbox txtName (label: 'Name', attribute: Name) textbox txtEmail (label: 'Email', attribute: Email) - footer footer1 { + footer { actionbutton actionButton1 (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton actionButton2 (caption: 'Cancel', action: cancel changes, buttonstyle: default) } @@ -716,7 +716,7 @@ create page PgTest.P012_Product_Manage_Edit folder 'DataGrid' textbox txtCode (label: 'Product Code', attribute: Code) textarea txtDescription (label: 'Description', attribute: description) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes close page, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes close page, buttonstyle: default) } @@ -1009,7 +1009,7 @@ create or modify page PgTest.P014_Product_Edit folder 'DataView' ( } } } - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } @@ -1044,7 +1044,7 @@ create page PgTest.P015_Product_EditFull folder 'DataView' } } } - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes, buttonstyle: default) } @@ -1083,7 +1083,7 @@ create or modify page PgTest.P018_DataView_FormOrientation_Vertical folder 'Data dataview ProductForm (datasource: $Product, FormOrientation: Vertical) { textbox txtName (label: 'Product Name', attribute: Name) textbox txtCode (label: 'Product Code', attribute: Code) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } @@ -1103,7 +1103,7 @@ create or modify page PgTest.P018_DataView_FormOrientation_Horizontal folder 'Da dataview ProductForm (datasource: $Product, FormOrientation: Horizontal) { textbox txtName (label: 'Product Name', attribute: Name) textbox txtCode (label: 'Product Code', attribute: Code) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } @@ -1123,7 +1123,7 @@ create or modify page PgTest.P018_DataView_LabelWidth folder 'DataView' ( dataview ProductForm (datasource: $Product, LabelWidth: 5) { textbox txtName (label: 'Product Name', attribute: Name) textbox txtCode (label: 'Product Code', attribute: Code) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } @@ -1477,7 +1477,7 @@ create page PgTest.P020_Master_Detail folder 'MasterDetail' textbox txtAddress (label: 'Address', attribute: Address) textbox txtCity (label: 'City', attribute: City) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes, buttonstyle: default) } @@ -1820,7 +1820,7 @@ create page PgTest.P023c_Detail_MicroflowAction folder 'MicroflowActions' textbox txtPrice (label: 'Price', attribute: Price) textbox txtStock (label: 'Stock', attribute: Stock) - footer footer1 { + footer { -- Microflow action with page parameter passes the DataView's object actionbutton btnProcess ( caption: 'Process Product', @@ -1942,7 +1942,7 @@ create page PgTest.StyledPage folder 'Styling' dynamictext dtCity (content: 'City: {1}', contentparams: ({1} = City), class: 'text-muted') } - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary, class: 'btn-block') actionbutton btnCancel (caption: 'Cancel', action: cancel changes, class: 'btn-outline') } @@ -2299,7 +2299,7 @@ create page PgTest.P034_ComboBox_Enum folder 'ComboBox' -- ComboBox (pluggable widget) selecting from the Country enumeration combobox cmbCountry (label: 'Country (ComboBox)', attribute: Country) }}} - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes close page, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes close page, buttonstyle: default) } @@ -2332,7 +2332,7 @@ create page PgTest.P035_Order_Edit folder 'ComboBox' -- TEXTBOX txtStatus (Label: 'Status', Attribute: Status) textarea txtNotes (label: 'Notes', attribute: Notes) }}} - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes close page, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes close page, buttonstyle: default) } @@ -2422,7 +2422,7 @@ create page PgTest.P037_GroupBox_Example folder 'GroupBox' datepicker dpCreateDate (label: 'Created On', attribute: CreateDate) } - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes close page, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes close page, buttonstyle: default) } @@ -2460,7 +2460,7 @@ create page PgTest.P036_ComboBox_Association folder 'ComboBox' textbox txtTotal (label: 'Total Amount', attribute: TotalAmount) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes close page, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes close page, buttonstyle: default) } diff --git a/mdl-examples/doctype-tests/12-styling-examples.mdl b/mdl-examples/doctype-tests/12-styling-examples.mdl index 2873a5d396..16ca64a351 100644 --- a/mdl-examples/doctype-tests/12-styling-examples.mdl +++ b/mdl-examples/doctype-tests/12-styling-examples.mdl @@ -86,7 +86,7 @@ create page StyleTest.P001_CSS_Classes textbox txtName (label: 'Name', attribute: Name, class: 'form-control-lg') textbox txtEmail (label: 'Email', attribute: Email, class: 'text-muted') - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary, class: 'btn-lg') actionbutton btnCancel (caption: 'Cancel', action: cancel changes, class: 'btn-sm') } @@ -241,7 +241,7 @@ create page StyleTest.P003_Design_Properties datepicker dpDue (label: 'Due Date', attribute: DueDate) checkbox chkDone (label: 'Completed', attribute: IsCompleted) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary, designproperties: ('Full width': on)) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } @@ -342,7 +342,7 @@ create page StyleTest.P004_Combined_Styling } } - footer footer1 { + footer { actionbutton btnSave ( caption: 'Save Employee', action: save changes, @@ -543,7 +543,7 @@ create page StyleTest.P006_Roundtrip textbox txtEmail (label: 'Email', attribute: Email) checkbox chkActive (label: 'Active', attribute: IsActive) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } @@ -582,7 +582,7 @@ create or modify page StyleTest.P006_Roundtrip textbox txtEmail (label: 'Email', attribute: Email) checkbox chkActive (label: 'Active', attribute: IsActive) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } diff --git a/mdl-examples/doctype-tests/15-fragment-examples.mdl b/mdl-examples/doctype-tests/15-fragment-examples.mdl index 6d5111e5e8..f159160392 100644 --- a/mdl-examples/doctype-tests/15-fragment-examples.mdl +++ b/mdl-examples/doctype-tests/15-fragment-examples.mdl @@ -236,7 +236,7 @@ create page FragTest.P005_Quick_View dataview dvCustomer (datasource: $Customer) { dynamictext txtHeader (content: 'Customer Details', rendermode: H3) use fragment CustomerFields - footer footer1 { + footer { actionbutton btnClose (caption: 'Close', action: close page) } } @@ -319,8 +319,8 @@ create snippet FragTest.CustomerCard -- Replace a widget subtree -- ALTER PAGE FragTest.P001_Customer_Edit { --- REPLACE footer1 WITH { --- FOOTER newFooter { +-- REPLACE dvMain.footer WITH { +-- FOOTER { -- ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE_CHANGES, ButtonStyle: Primary) -- ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL_CHANGES) -- ACTIONBUTTON btnDelete (Caption: 'Delete', Action: CANCEL_CHANGES, ButtonStyle: Danger) diff --git a/mdl-examples/doctype-tests/17-custom-widget-examples.mdl b/mdl-examples/doctype-tests/17-custom-widget-examples.mdl index 8c3761e011..1250ab53db 100644 --- a/mdl-examples/doctype-tests/17-custom-widget-examples.mdl +++ b/mdl-examples/doctype-tests/17-custom-widget-examples.mdl @@ -158,7 +158,7 @@ create page CWTest.P_ComboBox_Enum folder 'CustomWidgets' } } } - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes close page, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes close page, buttonstyle: default) } @@ -201,7 +201,7 @@ create page CWTest.P_ComboBox_Assoc folder 'CustomWidgets' } } } - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes close page, buttonstyle: success) actionbutton btnCancel (caption: 'Cancel', action: cancel changes close page, buttonstyle: default) } diff --git a/mdl-examples/doctype-tests/29-datagrid-examples.mdl b/mdl-examples/doctype-tests/29-datagrid-examples.mdl index fd10e9e1d4..bfedd2e856 100644 --- a/mdl-examples/doctype-tests/29-datagrid-examples.mdl +++ b/mdl-examples/doctype-tests/29-datagrid-examples.mdl @@ -87,7 +87,7 @@ create page DgTest.Product_Edit folder 'DataGrid' ( dataview dvProduct (datasource: $Product) { textbox txtName (label: 'Name', attribute: Name) textbox txtCode (label: 'Code', attribute: Code) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes close page, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel changes close page, buttonstyle: default) } diff --git a/mdl-examples/use-cases/02-agentic-search.mdl b/mdl-examples/use-cases/02-agentic-search.mdl index e70a5ca21f..fdd680a3f3 100644 --- a/mdl-examples/use-cases/02-agentic-search.mdl +++ b/mdl-examples/use-cases/02-agentic-search.mdl @@ -395,7 +395,7 @@ create page TaskMgmt.Task_Detail datepicker dpDue (label: 'Due Date', attribute: DueDate) textbox txtHours (label: 'Estimated Hours', attribute: EstimatedHours) - footer footer1 { + footer { actionbutton btnSave (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton btnCancel (caption: 'Cancel', action: cancel changes) } diff --git a/mdl-examples/widgetdemo/03-showcase-page.mdl b/mdl-examples/widgetdemo/03-showcase-page.mdl index 857186439d..c415600aeb 100644 --- a/mdl-examples/widgetdemo/03-showcase-page.mdl +++ b/mdl-examples/widgetdemo/03-showcase-page.mdl @@ -156,7 +156,7 @@ create or modify page WidgetDemo.WidgetDemo_Showcase ( } } } - footer ftr1 { + footer { actionbutton btnSv (caption: 'Save', action: save changes, buttonstyle: primary) actionbutton btnCn (caption: 'Cancel', action: cancel changes) } diff --git a/mdl/ast/ast_microflow.go b/mdl/ast/ast_microflow.go index b878a6e254..dd593b3821 100644 --- a/mdl/ast/ast_microflow.go +++ b/mdl/ast/ast_microflow.go @@ -407,6 +407,12 @@ type ActivityAnnotations struct { // than silently straightening the edge. InvalidCurves []string + // InvalidAnchors holds the raw text of any @anchor parameter the visitor + // could not use — an unknown key, or a side that is not top/right/bottom/ + // left — so validation can refuse it rather than leave the edge on the + // builder's default sides in silence (mendixlabs/mxcli#992). + InvalidAnchors []string + // InvalidNotes holds the raw text of any `@annotation(...)` parameter // the visitor could not use — an unknown key, or a malformed `position:`/`size:` // pair — so validation can refuse it. Dropping it would lose the note diff --git a/mdl/ast/ast_microflow_workflow.go b/mdl/ast/ast_microflow_workflow.go index 425f94bdb1..ac9fdd9a1a 100644 --- a/mdl/ast/ast_microflow_workflow.go +++ b/mdl/ast/ast_microflow_workflow.go @@ -100,7 +100,13 @@ func (*OpenWorkflowStmt) isMicroflowStatement() {} // LockWorkflowStmt represents: LOCK WORKFLOW ($WorkflowVar | ALL) type LockWorkflowStmt struct { - WorkflowVariable string + WorkflowVariable string + // Workflow is the qualified name of the workflow definition, the other + // selection Studio Pro offers ("Input type: workflow document"). + Workflow string + // PauseAllWorkflows is `pause all`, Studio Pro's "Pause instances". The bare + // `lock workflow all` also sets it, with no selection — the form check and + // exec refuse (mendixlabs/mxcli#870). PauseAllWorkflows bool ErrorHandling *ErrorHandlingClause Annotations *ActivityAnnotations @@ -110,7 +116,9 @@ func (*LockWorkflowStmt) isMicroflowStatement() {} // UnlockWorkflowStmt represents: UNLOCK WORKFLOW ($WorkflowVar | ALL) type UnlockWorkflowStmt struct { - WorkflowVariable string + WorkflowVariable string + Workflow string // qualified name; see LockWorkflowStmt + // ResumeAllPausedWorkflows is `unpause all`, "Unpause instances". ResumeAllPausedWorkflows bool ErrorHandling *ErrorHandlingClause Annotations *ActivityAnnotations diff --git a/mdl/backend/mcp/microflow.go b/mdl/backend/mcp/microflow.go index 56e7a17868..b53c54305c 100644 --- a/mdl/backend/mcp/microflow.go +++ b/mdl/backend/mcp/microflow.go @@ -4,6 +4,7 @@ package mcp import ( "fmt" + "reflect" "strconv" "strings" @@ -293,6 +294,12 @@ func (b *Backend) buildFlowDocContent(kind, name string, params []*microflows.Mi if !ok1 || !ok2 { return nil, fmt.Errorf("%s %q: a sequence flow references an object that is not supported yet", kind, name) } + if f.IsErrorHandler { + // PED's SequenceFlow has no property marking an error-handler + // flow, so this edge would be written as an ordinary one — a + // second normal exit from the activity (mendixlabs/mxcli#698). + return nil, fmt.Errorf("%s %q: an error handler flow cannot be authored over MCP (Studio Pro's PED API has no error-handler flow) — run without --mcp", kind, name) + } pf := map[string]any{ "originId": fmt.Sprintf("$id(%s)", op), "destinationId": fmt.Sprintf("$id(%s)", dp), @@ -437,6 +444,9 @@ func (b *Backend) mapObjectTree(o microflows.MicroflowObject, path string, idPat if err != nil { return nil, err } + if err := carryErrorHandlingType(obj.Action, action); err != nil { + return nil, err + } return map[string]any{ "$Type": "Microflows$ActionActivity", "relativeMiddlePoint": pos, @@ -1115,3 +1125,46 @@ func mfEnumName(dt microflows.DataType) string { } return "" } + +// carryErrorHandlingType writes an action's error-handling type onto its PED +// map. mapMicroflowAction built each action without it, so `on error rollback` +// or `on error continue` authored over --mcp landed with PED's default — the +// clause silently gone (mendixlabs/mxcli#698). PED declares errorHandlingType on +// every action element ('Rollback' | 'Custom' | 'CustomWithoutRollBack' | +// 'Continue' | 'Abort', ped_get_schema on Studio Pro 11.14). +// +// The two custom forms are refused: they need an error-handler flow, which +// PED's SequenceFlow cannot express. +func carryErrorHandlingType(a microflows.MicroflowAction, m map[string]any) error { + eh := actionErrorHandlingType(a) + switch eh { + case "": + return nil + case microflows.ErrorHandlingTypeCustom, microflows.ErrorHandlingTypeCustomWithoutRollback: + return fmt.Errorf("a custom error handler (`on error … begin … end error`) cannot be authored over MCP: Studio Pro's PED API has no error-handler flow — run without --mcp") + } + m["errorHandlingType"] = string(eh) + return nil +} + +var errorHandlingTypeType = reflect.TypeOf(microflows.ErrorHandlingType("")) + +// actionErrorHandlingType reads the ErrorHandlingType field every action that +// has one carries; "" for one that has none. +func actionErrorHandlingType(a microflows.MicroflowAction) microflows.ErrorHandlingType { + v := reflect.ValueOf(a) + if v.Kind() == reflect.Ptr { + if v.IsNil() { + return "" + } + v = v.Elem() + } + if v.Kind() != reflect.Struct { + return "" + } + f := v.FieldByName("ErrorHandlingType") + if !f.IsValid() || f.Type() != errorHandlingTypeType { + return "" + } + return microflows.ErrorHandlingType(f.String()) +} diff --git a/mdl/backend/mcp/microflow_error_handling_test.go b/mdl/backend/mcp/microflow_error_handling_test.go new file mode 100644 index 0000000000..db348133c8 --- /dev/null +++ b/mdl/backend/mcp/microflow_error_handling_test.go @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mcp + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +func callActivity(eh microflows.ErrorHandlingType) *microflows.ActionActivity { + a := µflows.ActionActivity{Action: µflows.MicroflowCallAction{ + ErrorHandlingType: eh, + MicroflowCall: µflows.MicroflowCall{Microflow: "M.SUB"}, + }} + a.ID = "act-1" + return a +} + +// mendixlabs/mxcli#698. The MCP mapper built the action without its +// errorHandlingType, so `call microflow … on error rollback|continue` authored +// over --mcp landed with PED's default — the clause silently lost. The PED +// schema (ped_get_schema, Studio Pro 11.14) declares errorHandlingType on every +// action element: 'Rollback' | 'Custom' | 'CustomWithoutRollBack' | 'Continue' +// | 'Abort'. +func TestMapObjectTree_CarriesErrorHandlingType(t *testing.T) { + b := &Backend{} + for _, eh := range []microflows.ErrorHandlingType{microflows.ErrorHandlingTypeRollback, microflows.ErrorHandlingTypeContinue} { + m, err := b.mapObjectTree(callActivity(eh), "/objects/0", map[model.ID]string{}) + if err != nil { + t.Fatalf("%s: %v", eh, err) + } + action := m["action"].(map[string]any) + if action["errorHandlingType"] != string(eh) { + t.Errorf("%s: action errorHandlingType = %v", eh, action["errorHandlingType"]) + } + } + // No clause: nothing written, PED's default applies — as before. + m, err := b.mapObjectTree(callActivity(""), "/objects/0", map[model.ID]string{}) + if err != nil { + t.Fatal(err) + } + if _, ok := m["action"].(map[string]any)["errorHandlingType"]; ok { + t.Error("an action with no error handling wrote one") + } +} + +// A custom handler needs an error-handler flow, and PED's SequenceFlow has no +// property for one (ped_get_schema: originId, destinationId, sides, caseValue). +// Written anyway, the handler's error edge became an ordinary flow. Refused +// rather than dropped. +func TestMapObjectTree_RefusesACustomErrorHandler(t *testing.T) { + b := &Backend{} + for _, eh := range []microflows.ErrorHandlingType{microflows.ErrorHandlingTypeCustom, microflows.ErrorHandlingTypeCustomWithoutRollback} { + if _, err := b.mapObjectTree(callActivity(eh), "/objects/0", map[model.ID]string{}); err == nil || !strings.Contains(err.Error(), "error handler") { + t.Errorf("%s: accepted (%v)", eh, err) + } + } + oc := µflows.MicroflowObjectCollection{ + Objects: []microflows.MicroflowObject{callActivity(microflows.ErrorHandlingTypeRollback), func() microflows.MicroflowObject { + e := µflows.EndEvent{} + e.ID = "end-1" + return e + }()}, + Flows: []*microflows.SequenceFlow{{OriginID: "act-1", DestinationID: "end-1", IsErrorHandler: true}}, + } + if _, err := b.buildFlowDocContent("microflow", "MF", nil, oc, µflows.VoidType{}); err == nil || !strings.Contains(err.Error(), "error handler") { + t.Errorf("an error-handler flow was written as an ordinary one (%v)", err) + } +} diff --git a/mdl/backend/modelsdk/association_move_cross.go b/mdl/backend/modelsdk/association_move_cross.go new file mode 100644 index 0000000000..eec264a5cb --- /dev/null +++ b/mdl/backend/modelsdk/association_move_cross.go @@ -0,0 +1,242 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "fmt" + "strings" + + "go.mongodb.org/mongo-driver/v2/bson" + "go.mongodb.org/mongo-driver/v2/x/bsonx/bsoncore" + + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/modelsdk/element" + genDm "github.com/mendixlabs/mxcli/modelsdk/gen/domainmodels" + "github.com/mendixlabs/mxcli/modelsdk/meta" + mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// A MOVE ENTITY has to look at the cross-associations that already exist, not +// only at the plain associations it converts (ako/mxcli#628). A cross-association +// lives in the unit of its FROM entity, holds that entity by id (ParentPointer) +// and the TO entity by qualified name (Child). Moving entity E from S to T leaves +// three shapes that are not the same case: +// +// 1. Both endpoints end up in one module. The cross-association becomes a plain +// DomainModels$Association again in that module — the inverse of +// crossAssocRawFromAssoc. This is what moving the second endpoint of an +// association across, one entity at a time, produces. +// 2. E is the FROM entity and the TO entity is elsewhere. The cross-association +// travels with E to T (its ParentPointer would not resolve in S), so its +// qualified name changes S.X -> T.X. +// 3. E is the TO entity of a cross-association in ANOTHER module. That module's +// unit names `S.E`; it is re-pointed to `T.E`. +// +// Left alone, shape 1 and 2 leave a ParentPointer naming an element absent from +// its unit — Studio Pro fails to load the project ("The given key was not present +// in the dictionary") — and shape 3 leaves a dangling qualified name. Every +// conversion carries the stored document, GUID included: the runtime keys the +// association's data on that GUID (CLAUDE.md, "A GUID Is the Database's Identity"). + +// moveOwnCrossAssociations handles shapes 1 and 2: the cross-associations in the +// source unit whose FROM entity is the moved one. It removes them from sourceDM +// and adds them to targetDM, as plain associations when their TO entity lives in +// the target module. +func moveOwnCrossAssociations(sourceDM, targetDM *genDm.DomainModel, entityID model.ID, sourceModuleName, targetModuleName string) []types.MovedAssociation { + targetEntities := entityIDsByName(targetDM) + var out []types.MovedAssociation + var removeIdx []int + for i, el := range sourceDM.CrossAssociationsItems() { + ca, ok := el.(*genDm.CrossAssociation) + if !ok || string(ca.ParentRefID()) != string(entityID) { + continue + } + moved := types.MovedAssociation{ + Name: ca.Name(), + OldQualifiedName: sourceModuleName + "." + ca.Name(), + NewQualifiedName: targetModuleName + "." + ca.Name(), + } + mod, ent, _ := strings.Cut(ca.ChildQualifiedName(), ".") + if toID, ok := targetEntities[ent]; ok && mod == targetModuleName { + targetDM.AddAssociations(assocFromGenCrossAssoc(ca, toID)) + moved.SameModule = true + } else { + targetDM.AddCrossAssociations(cloneCrossAssoc(ca)) + } + removeIdx = append(removeIdx, i) + out = append(out, moved) + } + for i := len(removeIdx) - 1; i >= 0; i-- { + sourceDM.RemoveCrossAssociations(removeIdx[i]) + } + return out +} + +// convertIncomingCrossAssociations handles shape 1 seen from the other side: a +// cross-association in the TARGET unit whose TO entity is the moved one. Both +// endpoints are now in the target module, so it becomes a plain association +// there. Its qualified name does not change. +func convertIncomingCrossAssociations(targetDM *genDm.DomainModel, entityID model.ID, oldEntityQN, targetModuleName string) []types.MovedAssociation { + var out []types.MovedAssociation + var removeIdx []int + for i, el := range targetDM.CrossAssociationsItems() { + ca, ok := el.(*genDm.CrossAssociation) + if !ok || ca.ChildQualifiedName() != oldEntityQN { + continue + } + targetDM.AddAssociations(assocFromGenCrossAssoc(ca, string(entityID))) + removeIdx = append(removeIdx, i) + qn := targetModuleName + "." + ca.Name() + out = append(out, types.MovedAssociation{Name: ca.Name(), OldQualifiedName: qn, NewQualifiedName: qn, SameModule: true}) + } + for i := len(removeIdx) - 1; i >= 0; i-- { + targetDM.RemoveCrossAssociations(removeIdx[i]) + } + return out +} + +// repointCrossAssociationsElsewhere handles shape 3: every cross-association in a +// domain model other than the source and target units that names the moved entity +// as its TO entity is re-pointed to the entity's new qualified name. +func (b *Backend) repointCrossAssociationsElsewhere(sourceDMID, targetDMID model.ID, oldEntityQN, newEntityQN string) error { + dms, err := b.ListDomainModels() + if err != nil { + return fmt.Errorf("list domain models: %w", err) + } + for _, info := range dms { + if info.ID == sourceDMID || info.ID == targetDMID || string(info.ID) == meta.SystemDomainModelID { + continue + } + gdm, err := b.loadDomainModelGen(info.ID) + if err != nil { + return err + } + changed := false + for _, el := range gdm.CrossAssociationsItems() { + if ca, ok := el.(*genDm.CrossAssociation); ok && ca.ChildQualifiedName() == oldEntityQN { + ca.SetChildQualifiedName(newEntityQN) + changed = true + } + } + if changed { + if err := b.persistDM(info.ID, gdm); err != nil { + return err + } + } + } + return nil +} + +func entityIDsByName(dm *genDm.DomainModel) map[string]string { + out := map[string]string{} + for _, el := range dm.EntitiesItems() { + if e, ok := el.(*genDm.Entity); ok { + out[e.Name()] = string(e.ID()) + } + } + return out +} + +// cloneCrossAssoc re-homes a stored cross-association in another unit unchanged: +// a clean element over the stored bytes, so the encoder passes the whole document +// — GUID included — through verbatim. +func cloneCrossAssoc(ca *genDm.CrossAssociation) *genDm.CrossAssociation { + if raw := ca.Raw(); raw != nil { + out := genDm.NewCrossAssociation() + out.SetRaw(raw) + out.InitFromRaw(raw) + out.SetID(ca.ID()) + return out + } + return ca +} + +// assocFromGenCrossAssoc converts a cross-association back into the plain +// association both of whose endpoints now share its unit; childID is the TO +// entity's element id. Like the forward conversion it prefers a raw transform of +// the stored document, so the GUID and every untouched property survive; the +// property build is the fallback for a cross-association never persisted. +func assocFromGenCrossAssoc(ca *genDm.CrossAssociation, childID string) *genDm.Association { + if raw, ok := assocRawFromCrossAssoc(ca, childID); ok { + out := genDm.NewAssociation() + out.SetRaw(raw) + out.InitFromRaw(raw) + out.SetID(ca.ID()) + return out + } + + out := genDm.NewAssociation() + out.SetID(ca.ID()) + out.SetName(ca.Name()) + out.SetDocumentation(ca.Documentation()) + out.SetExportLevel(orDefault(ca.ExportLevel(), "Hidden")) + out.SetParentID(ca.ParentRefID()) + out.SetChildID(element.ID(childID)) + out.SetType(ca.Type()) + out.SetOwner(ca.Owner()) + out.SetStorageFormat(orDefault(ca.StorageFormat(), "Column")) + out.SetParentConnection(domainmodel.DefaultParentConnection) + out.SetChildConnection(domainmodel.DefaultChildConnection) + pdb, cdb := "DeleteMeButKeepReferences", "DeleteMeButKeepReferences" + if odb, ok := ca.DeleteBehavior().(*genDm.AssociationDeleteBehavior); ok { + pdb, cdb = orDefault(odb.ParentDeleteBehavior(), pdb), orDefault(odb.ChildDeleteBehavior(), cdb) + } + out.SetDeleteBehavior(deleteBehaviorToGen(pdb, cdb)) + if src, ok := ca.Source().(*genDm.OqlViewAssociationSource); ok && src != nil { + out.SetSource(oqlViewAssociationSourceToGen(src.Reference())) + } + assignID(out.DeleteBehavior()) + return out +} + +// assocRawFromCrossAssoc is the inverse of crossAssocRawFromAssoc: $Type becomes +// DomainModels$Association, Child (a qualified name) becomes ChildPointer (the TO +// entity's 16-byte id, resolvable now that it shares the unit), and the two +// connection points a plain association declares — the line's on-canvas anchors — +// are added with the defaults mxcli and Studio Pro write for a new association. +// Everything else, the GUID above all, passes through. The key set is the one +// `generated/metamodel` declares for DomainModelsAssociation. +func assocRawFromCrossAssoc(ca *genDm.CrossAssociation, childID string) (bson.Raw, bool) { + raw := ca.Raw() + if raw == nil { + return nil, false + } + elems, err := bsoncore.Document(raw).Elements() + if err != nil { + return nil, false + } + out := make(bson.D, 0, len(elems)+2) + child, parent := false, false + for _, e := range elems { + switch e.Key() { + case "$Type": + out = append(out, bson.E{Key: "$Type", Value: "DomainModels$Association"}) + case "Child": + out = append(out, + bson.E{Key: "ChildConnection", Value: domainmodel.DefaultChildConnection}, + bson.E{Key: "ChildPointer", Value: mmpr.IDToBsonBinary(childID)}) + child = true + case "ParentPointer": + v := e.Value() + out = append(out, + bson.E{Key: "ParentConnection", Value: domainmodel.DefaultParentConnection}, + bson.E{Key: "ParentPointer", Value: bson.RawValue{Type: bson.Type(v.Type), Value: v.Data}}) + parent = true + case "ChildConnection", "ParentConnection": + // Not declared on a cross-association; never carry a stray one twice. + default: + v := e.Value() + out = append(out, bson.E{Key: e.Key(), Value: bson.RawValue{Type: bson.Type(v.Type), Value: v.Data}}) + } + } + if !child || !parent { + return nil, false + } + b, err := bson.Marshal(out) + if err != nil { + return nil, false + } + return bson.Raw(b), true +} diff --git a/mdl/backend/modelsdk/association_move_write.go b/mdl/backend/modelsdk/association_move_write.go index 4f1e883845..2e84dbd91d 100644 --- a/mdl/backend/modelsdk/association_move_write.go +++ b/mdl/backend/modelsdk/association_move_write.go @@ -368,6 +368,15 @@ func (b *Backend) MoveEntity(entity *domainmodel.Entity, sourceDMID, targetDMID sourceDM.RemoveAssociations(removeIdx[i]) } + // The cross-associations that already exist (ako/mxcli#628): the ones the moved + // entity is the FROM end of travel with it — or become plain associations again + // when their TO entity is already in the target — and the ones in the target + // that point at it become plain associations there. See association_move_cross.go. + oldEntityQN := sourceModuleName + "." + entity.Name + newEntityQN := targetModuleName + "." + entity.Name + converted = append(converted, moveOwnCrossAssociations(sourceDM, targetDM, entity.ID, sourceModuleName, targetModuleName)...) + converted = append(converted, convertIncomingCrossAssociations(targetDM, entity.ID, oldEntityQN, targetModuleName)...) + // Rewrite the moved entity's module-qualified refs (view source + validations). oldPrefix, newPrefix := sourceModuleName+".", targetModuleName+"." if entity.Source == "DomainModels$OqlViewEntitySource" && strings.HasPrefix(entity.SourceDocumentRef, oldPrefix) { @@ -390,9 +399,9 @@ func (b *Backend) MoveEntity(entity *domainmodel.Entity, sourceDMID, targetDMID // // An ATTRIBUTE reference always follows the entity, so its module prefix is // rewritten unconditionally. An ASSOCIATION reference is rewritten only for the - // associations this move actually sent to the target module: a pre-existing - // cross-association whose parent is the moved entity is not in the conversion - // list and does not travel, so a blanket prefix swap would break it. + // associations this move actually sent to the target module — a converted one, + // or a pre-existing cross-association the moved entity is the FROM end of + // (#628) — never by a blanket prefix swap. assocRenames := make(map[string]string, len(converted)) for _, m := range converted { if m.Moved() { @@ -438,5 +447,10 @@ func (b *Backend) MoveEntity(entity *domainmodel.Entity, sourceDMID, targetDMID if err := b.persistDM(targetDMID, targetDM); err != nil { return nil, fmt.Errorf("MoveEntity: persist target: %w", err) } + // A cross-association in any OTHER module that names the moved entity is + // re-pointed; nothing else in the move rewrites that module's unit (#628). + if err := b.repointCrossAssociationsElsewhere(sourceDMID, targetDMID, oldEntityQN, newEntityQN); err != nil { + return nil, fmt.Errorf("MoveEntity: re-point cross-associations: %w", err) + } return converted, nil } diff --git a/mdl/backend/modelsdk/domainmodel_alter.go b/mdl/backend/modelsdk/domainmodel_alter.go index ccd3abf86d..bcc65de6e2 100644 --- a/mdl/backend/modelsdk/domainmodel_alter.go +++ b/mdl/backend/modelsdk/domainmodel_alter.go @@ -774,6 +774,14 @@ func (b *Backend) UpdateAttribute(domainModelID, entityID model.ID, attr *domain next := attributeToGen(attr, entityIsExternal(ent)) next.SetID(items[idx].ID()) + // attributeToGen returns raw == nil, which the codec reads as a NEW element + // and mints GUID = $ID for — the runtime keys mendixsystem$attribute.id on + // that GUID and would drop the column (ako/mxcli#627, the #1119 class). + // Attribute -> Attribute keeps the $Type, so carrying the stored raw bytes + // is enough; no raw transform is needed. + if stored, ok := items[idx].(*genDm.Attribute); ok { + carryStoredAttribute(next, stored) + } // The generated list offers only Append and Remove, so an in-place replace // means rebuilding it. Order is worth the rebuild: it is the order Studio diff --git a/mdl/backend/modelsdk/domainmodel_child_identity.go b/mdl/backend/modelsdk/domainmodel_child_identity.go index b07c04bb98..ca0d91a38c 100644 --- a/mdl/backend/modelsdk/domainmodel_child_identity.go +++ b/mdl/backend/modelsdk/domainmodel_child_identity.go @@ -82,15 +82,7 @@ func carryAttributeIdentity(ge, orig *genDm.Entity, entity *domainmodel.Entity) claimed := make(map[string]bool, len(storedByID)) carry := func(ga, sa *genDm.Attribute) { - ga.SetID(sa.ID()) - ga.SetRaw(sa.Raw()) - // attributeToGen sets ExportLevel "Hidden", and a property the rebuild - // sets wins over the carried raw bytes — so without this an API attribute - // became Hidden on every rewrite of its entity (ako/mxcli#801). The - // semantic attribute has no export level to take it from. - if lvl := sa.ExportLevel(); lvl != "" { - ga.SetExportLevel(lvl) - } + carryStoredAttribute(ga, sa) claimed[string(sa.ID())] = true } @@ -170,3 +162,22 @@ func carryIndexIdentity(ge, orig *genDm.Entity, entity *domainmodel.Entity) { claimed[string(si.ID())] = true } } + +// carryStoredAttribute makes the rebuilt attribute ga the stored attribute sa as +// far as identity goes: the stored $ID and raw bytes (so the GUID passes through +// instead of being re-minted as $ID — #1119, ako/mxcli#627) and the stored export +// level. Both rebuild sites use it: the entity rewrite (carryAttributeIdentity) +// and the single-attribute rewrite (Backend.UpdateAttribute). +func carryStoredAttribute(ga, sa *genDm.Attribute) { + ga.SetID(sa.ID()) + if raw := sa.Raw(); raw != nil { + ga.SetRaw(raw) + } + // attributeToGen sets ExportLevel "Hidden", and a property the rebuild + // sets wins over the carried raw bytes — so without this an API attribute + // became Hidden on every rewrite of its entity (ako/mxcli#801). The + // semantic attribute has no export level to take it from. + if lvl := sa.ExportLevel(); lvl != "" { + ga.SetExportLevel(lvl) + } +} diff --git a/mdl/backend/modelsdk/export_level_carry.go b/mdl/backend/modelsdk/export_level_carry.go index 7f1d2c9d67..5bea6df5c6 100644 --- a/mdl/backend/modelsdk/export_level_carry.go +++ b/mdl/backend/modelsdk/export_level_carry.go @@ -4,6 +4,8 @@ package modelsdkbackend import ( "fmt" + "sort" + "strings" "go.mongodb.org/mongo-driver/v2/x/bsonx/bsoncore" ) @@ -97,3 +99,69 @@ func withExportLevel(doc []byte, lvl string) ([]byte, error) { } return out, nil } + +// keepStoredTopLevel returns contents with each top-level key in keys set to +// the value the stored unit holds, for a document kind whose statement cannot +// state those properties and whose writer therefore emits constants for them. +// A key the stored unit lacks is left as written; a key only the stored unit +// has is inserted before the first written key that sorts after it, the order +// Studio Pro writes. +func (b *Backend) keepStoredTopLevel(unitID string, contents []byte, keys []string) ([]byte, error) { + if b.reader == nil || unitID == "" || len(contents) == 0 { + return contents, nil + } + stored, err := b.reader.GetRawUnitBytes(unitID) + if err != nil || len(stored) == 0 { + return contents, nil + } + return withStoredTopLevel(contents, stored, keys) +} + +// withStoredTopLevel is keepStoredTopLevel on bytes. +func withStoredTopLevel(doc, stored []byte, keys []string) ([]byte, error) { + carry := map[string]bsoncore.Value{} + for _, k := range keys { + if v, err := bsoncore.Document(stored).LookupErr(k); err == nil { + carry[k] = v + } + } + if len(carry) == 0 { + return doc, nil + } + elems, err := bsoncore.Document(doc).Elements() + if err != nil { + return nil, fmt.Errorf("carry stored properties: %w", err) + } + present := map[string]bool{} + for _, el := range elems { + present[el.Key()] = true + } + var missing []string + for k := range carry { + if !present[k] { + missing = append(missing, k) + } + } + sort.Strings(missing) + idx, out := bsoncore.AppendDocumentStart(nil) + for _, el := range elems { + key := el.Key() + for len(missing) > 0 && missing[0] < key && !strings.HasPrefix(key, "$") { + out = bsoncore.AppendValueElement(out, missing[0], carry[missing[0]]) + missing = missing[1:] + } + if v, ok := carry[key]; ok { + out = bsoncore.AppendValueElement(out, key, v) + continue + } + out = append(out, el...) + } + for _, k := range missing { + out = bsoncore.AppendValueElement(out, k, carry[k]) + } + out, err = bsoncore.AppendDocumentEnd(out, idx) + if err != nil { + return nil, fmt.Errorf("carry stored properties: %w", err) + } + return out, nil +} diff --git a/mdl/backend/modelsdk/integration_read.go b/mdl/backend/modelsdk/integration_read.go index 0ce146a12a..49d7591742 100644 --- a/mdl/backend/modelsdk/integration_read.go +++ b/mdl/backend/modelsdk/integration_read.go @@ -10,6 +10,7 @@ import ( "github.com/mendixlabs/mxcli/modelsdk/element" genBe "github.com/mendixlabs/mxcli/modelsdk/gen/businessevents" genDb "github.com/mendixlabs/mxcli/modelsdk/gen/databaseconnector" + genDT "github.com/mendixlabs/mxcli/modelsdk/gen/datatypes" genExportMappings "github.com/mendixlabs/mxcli/modelsdk/gen/exportmappings" genImportMappings "github.com/mendixlabs/mxcli/modelsdk/gen/importmappings" genOp "github.com/mendixlabs/mxcli/modelsdk/gen/odatapublish" @@ -427,11 +428,21 @@ func (b *Backend) ListPublishedRestServices() ([]*model.PublishedRestService, er continue } operation := &model.PublishedRestOperation{ - Path: op.Path(), - HTTPMethod: op.HttpMethod(), - Summary: op.Summary(), - Microflow: op.MicroflowQualifiedName(), - Deprecated: op.Deprecated(), + Path: op.Path(), + HTTPMethod: op.HttpMethod(), + Summary: op.Summary(), + Microflow: op.MicroflowQualifiedName(), + Deprecated: op.Deprecated(), + Documentation: op.Documentation(), + ImportMapping: op.ImportMappingQualifiedName(), + ExportMapping: op.ExportMappingQualifiedName(), + Commit: op.Commit(), + ObjectHandlingBackup: op.ObjectHandlingBackup(), + } + for _, pEl := range op.ParametersItems() { + if p, ok := pEl.(*genRest.RestOperationParameter); ok { + operation.OperationParameters = append(operation.OperationParameters, publishedRestParameterFromGen(p)) + } } operation.ID = model.ID(op.ID()) operation.TypeName = "Rest$PublishedRestServiceOperation" @@ -712,3 +723,25 @@ func rawQueryTypeName(q *genDb.DatabaseQuery) string { v, _ := q.Raw().Lookup(dbconnector.TypeKey).StringValueOK() return v } + +// publishedRestParameterFromGen reads one Rest$RestOperationParameter. +func publishedRestParameterFromGen(p *genRest.RestOperationParameter) *model.PublishedRestOperationParameter { + param := &model.PublishedRestOperationParameter{ + Name: p.Name(), + ParameterType: p.ParameterType(), + MicroflowParameter: p.MicroflowParameterQualifiedName(), + Description: p.Description(), + } + if t := p.Type(); t != nil { + param.DataType = strings.TrimSuffix(strings.TrimPrefix(t.TypeName(), "DataTypes$"), "Type") + switch g := t.(type) { + case *genDT.ObjectType: + param.QualifiedName = g.EntityQualifiedName() + case *genDT.ListType: + param.QualifiedName = g.EntityQualifiedName() + case *genDT.EnumerationType: + param.QualifiedName = g.EnumerationQualifiedName() + } + } + return param +} diff --git a/mdl/backend/modelsdk/issue627_update_attribute_guid_test.go b/mdl/backend/modelsdk/issue627_update_attribute_guid_test.go new file mode 100644 index 0000000000..f73ef0c0b8 --- /dev/null +++ b/mdl/backend/modelsdk/issue627_update_attribute_guid_test.go @@ -0,0 +1,138 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "os" + "path/filepath" + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// copyPedApp copies the Studio Pro-authored PedApp fixture into a temp dir. Its +// elements have GUID != $ID, which is the only subject on which a re-minted +// storage GUID is observable (CLAUDE.md, "A GUID Is the Database's Identity"). +func copyPedApp(t *testing.T) string { + t.Helper() + dst := t.TempDir() + if err := os.CopyFS(dst, os.DirFS("../../../testdata/pedapp")); err != nil { + t.Fatalf("copy PedApp fixture: %v", err) + } + return filepath.Join(dst, "PedApp.mpr") +} + +// TestIssue627_UpdateAttributePreservesGUID guards ako/mxcli#627: the +// AttributeModifier path of the fluent API (Backend.UpdateAttribute) rebuilt the +// attribute from the semantic model with raw == nil, so the codec minted +// GUID = $ID — the #1119 data-loss class, the column dropped on the next deploy. +// The #1119 write guard refused it, which left the API unusable on any Studio +// Pro-authored attribute. +// +// The subject is a PedApp string attribute whose stored GUID differs from its +// $ID; the comparison is against the GUID read before the write, never against a +// previous run (a re-mint is stable, so a second write is elided). +func TestIssue627_UpdateAttributePreservesGUID(t *testing.T) { + proj := copyPedApp(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + + dmID, ent, attr := studioProStringAttribute(t, b) + before := attributeGUIDs(t, b, dmID, ent.ID) + ids := attributeIDs(t, b, dmID, ent.ID) + want := before[attr.Name] + if want == "" || want == ids[attr.Name] { + t.Fatalf("subject %s.%s: GUID %q, $ID %q — need a stored GUID that differs from $ID", + ent.Name, attr.Name, want, ids[attr.Name]) + } + + st := attr.Type.(*domainmodel.StringAttributeType) + st.Length += 7 + if err := b.UpdateAttribute(dmID, ent.ID, attr); err != nil { + t.Fatalf("UpdateAttribute: %v", err) + } + if err := b.Disconnect(); err != nil { + t.Fatalf("disconnect: %v", err) + } + + b2 := New() + if err := b2.Connect(proj); err != nil { + t.Fatalf("reconnect: %v", err) + } + t.Cleanup(func() { _ = b2.Disconnect() }) + + after := attributeGUIDs(t, b2, dmID, ent.ID) + if got := after[attr.Name]; got != want { + t.Errorf("attribute %s.%s: storage GUID changed %s -> %s", ent.Name, attr.Name, want, got) + } + // The siblings pass through as stored bytes; assert it so a future rebuild + // of the whole list is caught here too. + for name, g := range before { + if after[name] != g { + t.Errorf("sibling attribute %s: storage GUID changed %s -> %s", name, g, after[name]) + } + } + // And the edit itself landed — otherwise "GUID unchanged" proves nothing. + dm, err := b2.GetDomainModel(moduleOfDM(t, b2, dmID)) + if err != nil { + t.Fatalf("GetDomainModel: %v", err) + } + var gotLen int + for _, e := range dm.Entities { + if e.ID != ent.ID { + continue + } + for _, a := range e.Attributes { + if a.Name == attr.Name { + if s, ok := a.Type.(*domainmodel.StringAttributeType); ok { + gotLen = s.Length + } + } + } + } + if gotLen != st.Length { + t.Errorf("attribute %s.%s: length = %d after UpdateAttribute, want %d", ent.Name, attr.Name, gotLen, st.Length) + } +} + +// studioProStringAttribute picks the first string attribute with a non-zero +// length in a loadable, non-System domain model. +func studioProStringAttribute(t *testing.T, b *Backend) (model.ID, *domainmodel.Entity, *domainmodel.Attribute) { + t.Helper() + dms, err := b.ListDomainModels() + if err != nil { + t.Fatalf("ListDomainModels: %v", err) + } + for _, d := range dms { + if _, err := b.loadDomainModelGen(d.ID); err != nil { + continue + } + for _, e := range d.Entities { + for _, a := range e.Attributes { + if s, ok := a.Type.(*domainmodel.StringAttributeType); ok && s.Length > 0 { + return d.ID, e, a + } + } + } + } + t.Fatal("no string attribute in a loadable PedApp domain model") + return "", nil, nil +} + +func moduleOfDM(t *testing.T, b *Backend, dmID model.ID) model.ID { + t.Helper() + dms, err := b.ListDomainModels() + if err != nil { + t.Fatalf("ListDomainModels: %v", err) + } + for _, d := range dms { + if d.ID == dmID { + return d.ContainerID + } + } + t.Fatalf("domain model %s not found", dmID) + return "" +} diff --git a/mdl/backend/modelsdk/issue628_move_cross_assoc_test.go b/mdl/backend/modelsdk/issue628_move_cross_assoc_test.go new file mode 100644 index 0000000000..180a4dc9ad --- /dev/null +++ b/mdl/backend/modelsdk/issue628_move_cross_assoc_test.go @@ -0,0 +1,305 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/model" + genDm "github.com/mendixlabs/mxcli/modelsdk/gen/domainmodels" +) + +// TestIssue628_MoveEntitySeesExistingCrossAssociations guards ako/mxcli#628: +// MoveEntity converted the plain associations of the moved entity and never +// looked at the cross-associations that already existed. Moving the second +// endpoint of an association across — the normal way to reorganise a module one +// entity at a time — left a cross-association whose ParentPointer named an +// element absent from its unit, and Studio Pro could no longer load the project +// ("The given key was not present in the dictionary"). +// +// The issue's honest assertion is on the stored documents: after every sequence, +// no association pointer may name an element absent from the unit it lives in, +// and no cross-association may name an entity that is not where it says. The +// association's storage GUID must survive each conversion, which only a Studio +// Pro-authored subject (PedApp, GUID != $ID) can detect. +func TestIssue628_MoveEntitySeesExistingCrossAssociations(t *testing.T) { + type step struct { + moveFrom bool // move the FROM (parent) entity; otherwise the TO (child) entity + to int // index into the three other modules + } + for _, tc := range []struct { + name string + steps []step + // wantPlainIn is the module index (into others) whose unit must hold the + // association as a plain association afterwards; -1 means it stays a + // cross-association. + wantPlainIn int + // wantCrossIn / wantChild, when the association stays cross-module: the + // unit holding it (-1 = the original module) and the module of its TO end. + wantCrossIn, wantChildIn int + }{ + // The issue's reproduction: shape 1 seen from the FROM end. + {name: "ToThenFrom_BackToPlain", steps: []step{{false, 0}, {true, 0}}, wantPlainIn: 0}, + // Shape 1 seen from the TO end: the cross-association is in the target. + {name: "FromThenTo_BackToPlain", steps: []step{{true, 0}, {false, 0}}, wantPlainIn: 0}, + // Shape 2: the FROM end moves on, the cross-association travels with it. + {name: "ToThenFromElsewhere_Travels", steps: []step{{false, 0}, {true, 1}}, wantPlainIn: -1, wantCrossIn: 1, wantChildIn: 0}, + // Shape 3: the TO end moves again; the cross-association stays in a module + // that is neither source nor target of the second move and is re-pointed. + {name: "ToThenToElsewhere_Repointed", steps: []step{{false, 0}, {false, 1}}, wantPlainIn: -1, wantCrossIn: -1, wantChildIn: 1}, + } { + t.Run(tc.name, func(t *testing.T) { + proj := copyPedAppFixture(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + srcID, srcMod, assocName, guid, childID, parentID := associationSubjectDistinct(t, b) + if guid == "" || guid == assocRawID(t, b, srcID, assocName) { + t.Fatalf("subject %s.%s: GUID %q must exist and differ from $ID", srcMod, assocName, guid) + } + others := otherDomainModels(t, b, srcID, 2) + + where := map[model.ID]int{childID: -1, parentID: -1} // -1 = srcMod + dmOf := func(i int) (model.ID, string) { + if i < 0 { + return srcID, srcMod + } + return others[i].id, others[i].name + } + for _, s := range tc.steps { + entID := childID + if s.moveFrom { + entID = parentID + } + fromDM, fromMod := dmOf(where[entID]) + toDM, toMod := dmOf(s.to) + ent := entityByID(t, b, fromDM, entID) + if _, err := b.MoveEntity(ent, fromDM, toDM, fromMod, toMod); err != nil { + t.Fatalf("MoveEntity %s.%s -> %s: %v", fromMod, ent.Name, toMod, err) + } + where[entID] = s.to + } + if err := b.Disconnect(); err != nil { + t.Fatalf("disconnect: %v", err) + } + + b2 := New() + if err := b2.Connect(proj); err != nil { + t.Fatalf("reconnect: %v", err) + } + t.Cleanup(func() { _ = b2.Disconnect() }) + + assertNoDanglingAssociations(t, b2) + + if tc.wantPlainIn >= 0 { + dmID, mod := dmOf(tc.wantPlainIn) + if got := crossAssocGUID(t, b2, dmID, assocName); got != "" { + t.Errorf("%s is still a cross-association in %s; both endpoints are there", assocName, mod) + } + got, ok := plainAssocGUID(t, b2, dmID, assocName) + if !ok { + t.Fatalf("%s is not a plain association in %s after both endpoints moved there", assocName, mod) + } + if got != guid { + t.Errorf("%s: storage GUID changed %s -> %s", assocName, guid, got) + } + return + } + dmID, mod := dmOf(tc.wantCrossIn) + got := crossAssocGUID(t, b2, dmID, assocName) + if got != guid { + t.Errorf("%s in %s: storage GUID %q, want %s", assocName, mod, got, guid) + } + _, childMod := dmOf(tc.wantChildIn) + if ref := crossAssocChild(t, b2, dmID, assocName); !strings.HasPrefix(ref, childMod+".") { + t.Errorf("%s in %s names TO entity %q, want one in %s", assocName, mod, ref, childMod) + } + }) + } +} + +type dmRef struct { + id model.ID + name string +} + +// associationSubjectDistinct is associationSubject restricted to an association +// between two different entities, with no other association between the pair. +func associationSubjectDistinct(t *testing.T, b *Backend) (dmID model.ID, moduleName, assocName, assocGUID string, childID, parentID model.ID) { + t.Helper() + dms, err := b.ListDomainModels() + if err != nil { + t.Fatalf("ListDomainModels: %v", err) + } + for _, d := range dms { + gdm, err := b.loadDomainModelGen(d.ID) + if err != nil { + continue + } + mod, err := b.GetModule(d.ContainerID) + if err != nil || mod == nil { + continue + } + for _, el := range gdm.AssociationsItems() { + a, ok := el.(*genDm.Association) + if !ok || a.Raw() == nil || a.ChildRefID() == a.ParentRefID() { + continue + } + return d.ID, mod.Name, a.Name(), rawKeyHex(t, a.Raw(), "GUID"), + model.ID(a.ChildRefID()), model.ID(a.ParentRefID()) + } + } + t.Fatal("no loadable domain model with an association between two entities") + return +} + +func otherDomainModels(t *testing.T, b *Backend, exclude model.ID, n int) []dmRef { + t.Helper() + dms, err := b.ListDomainModels() + if err != nil { + t.Fatalf("ListDomainModels: %v", err) + } + var out []dmRef + for _, d := range dms { + if d.ID == exclude { + continue + } + if _, err := b.loadDomainModelGen(d.ID); err != nil { + continue + } + mod, err := b.GetModule(d.ContainerID) + if err != nil || mod == nil { + continue + } + out = append(out, dmRef{d.ID, mod.Name}) + if len(out) == n { + return out + } + } + t.Fatalf("fixture has fewer than %d other loadable domain models", n) + return nil +} + +func assocRawID(t *testing.T, b *Backend, dmID model.ID, name string) string { + t.Helper() + gdm, err := b.loadDomainModelGen(dmID) + if err != nil { + t.Fatalf("loadDomainModelGen: %v", err) + } + for _, el := range gdm.AssociationsItems() { + if a, ok := el.(*genDm.Association); ok && a.Name() == name { + return rawKeyHex(t, a.Raw(), "$ID") + } + } + return "" +} + +func plainAssocGUID(t *testing.T, b *Backend, dmID model.ID, name string) (string, bool) { + t.Helper() + gdm, err := b.loadDomainModelGen(dmID) + if err != nil { + t.Fatalf("loadDomainModelGen: %v", err) + } + for _, el := range gdm.AssociationsItems() { + if a, ok := el.(*genDm.Association); ok && a.Name() == name { + return rawKeyHex(t, a.Raw(), "GUID"), true + } + } + return "", false +} + +func crossAssocChild(t *testing.T, b *Backend, dmID model.ID, name string) string { + t.Helper() + gdm, err := b.loadDomainModelGen(dmID) + if err != nil { + t.Fatalf("loadDomainModelGen: %v", err) + } + for _, el := range gdm.CrossAssociationsItems() { + if ca, ok := el.(*genDm.CrossAssociation); ok && ca.Name() == name { + return ca.ChildQualifiedName() + } + } + return "" +} + +// assertNoDanglingAssociations is the stored-document form of "the project +// opens": every association pointer resolves inside its own unit, and every +// cross-association's TO name resolves to an entity in the module it names. +func assertNoDanglingAssociations(t *testing.T, b *Backend) { + t.Helper() + dms, err := b.ListDomainModels() + if err != nil { + t.Fatalf("ListDomainModels: %v", err) + } + type unit struct { + mod string + dm *genDm.DomainModel + } + var units []unit + entitiesByModule := map[string]map[string]bool{} + for _, d := range dms { + gdm, err := b.loadDomainModelGen(d.ID) + if err != nil { + continue + } + mod, err := b.GetModule(d.ContainerID) + if err != nil || mod == nil { + continue + } + units = append(units, unit{mod.Name, gdm}) + names := map[string]bool{} + for _, el := range gdm.EntitiesItems() { + if e, ok := el.(*genDm.Entity); ok { + names[e.Name()] = true + } + } + entitiesByModule[mod.Name] = names + } + for _, u := range units { + ids := map[string]bool{} + for _, el := range u.dm.EntitiesItems() { + ids[string(el.ID())] = true + } + for _, el := range u.dm.AssociationsItems() { + if a, ok := el.(*genDm.Association); ok { + if !ids[string(a.ParentRefID())] || !ids[string(a.ChildRefID())] { + t.Errorf("association %s.%s points at an entity outside its unit", u.mod, a.Name()) + } + } + } + for _, el := range u.dm.CrossAssociationsItems() { + ca, ok := el.(*genDm.CrossAssociation) + if !ok { + continue + } + if !ids[string(ca.ParentRefID())] { + t.Errorf("cross-association %s.%s: ParentPointer names an entity absent from its unit", u.mod, ca.Name()) + } + mod, ent, _ := strings.Cut(ca.ChildQualifiedName(), ".") + if mod == "System" { + continue + } + if !entitiesByModule[mod][ent] { + t.Errorf("cross-association %s.%s: Child %q does not resolve", u.mod, ca.Name(), ca.ChildQualifiedName()) + } + if mod == u.mod { + t.Errorf("cross-association %s.%s names an entity in its own module; it should be a plain association", u.mod, ca.Name()) + } + } + } +} + +// copyPedAppFixture copies the Studio Pro-authored PedApp fixture into a temp +// dir; its elements have GUID != $ID, so a lost storage GUID is observable. +func copyPedAppFixture(t *testing.T) string { + t.Helper() + dst := t.TempDir() + if err := os.CopyFS(dst, os.DirFS("../../../testdata/pedapp")); err != nil { + t.Fatalf("copy PedApp fixture: %v", err) + } + return filepath.Join(dst, "PedApp.mpr") +} diff --git a/mdl/backend/modelsdk/microflow_list_error_handling_test.go b/mdl/backend/modelsdk/microflow_list_error_handling_test.go new file mode 100644 index 0000000000..b16c47514b --- /dev/null +++ b/mdl/backend/modelsdk/microflow_list_error_handling_test.go @@ -0,0 +1,134 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "fmt" + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// mendixlabs/mxcli#591. The writer stamped a literal "Rollback" on the list +// activities and cast, which is CE6035 in a nanoflow, whose activities store +// "Abort". The builder now supplies the flow flavour's default; this pins that +// the writer emits it and the reader returns it, and that an action with none +// keeps the historical "Rollback". +func TestListActions_ErrorHandlingTypeRoundTrips(t *testing.T) { + actions := func(eh microflows.ErrorHandlingType) []microflows.MicroflowAction { + return []microflows.MicroflowAction{ + µflows.CreateListAction{OutputVariable: "L", EntityQualifiedName: "M.E", ErrorHandlingType: eh}, + µflows.ChangeListAction{ChangeVariable: "L", Type: microflows.ChangeListTypeAdd, Value: "$P", ErrorHandlingType: eh}, + µflows.ListOperationAction{OutputVariable: "H", Operation: µflows.HeadOperation{ListVariable: "L"}, ErrorHandlingType: eh}, + µflows.AggregateListAction{InputVariable: "L", OutputVariable: "N", Function: microflows.AggregateFunctionCount, ErrorHandlingType: eh}, + µflows.CastAction{OutputVariable: "S", ErrorHandlingType: eh}, + } + } + for _, tc := range []struct { + set, want microflows.ErrorHandlingType + }{ + {microflows.ErrorHandlingTypeAbort, microflows.ErrorHandlingTypeAbort}, + {"", microflows.ErrorHandlingTypeRollback}, + } { + oc := µflows.MicroflowObjectCollection{} + for i, a := range actions(tc.set) { + setActionID(a, model.ID(fmt.Sprintf("a-%d", i))) + act := µflows.ActionActivity{Action: a} + act.ID = model.ID(fmt.Sprintf("act-%d", i)) + act.Position = model.Point{X: 100 * i, Y: 100} + oc.Objects = append(oc.Objects, act) + } + mf := µflows.Microflow{Name: "MF", ObjectCollection: oc} + mf.ID = "mf-1" + got := roundTripMicroflow(t, mf) + n := 0 + for _, obj := range got.ObjectCollection.Objects { + aa, ok := obj.(*microflows.ActionActivity) + if !ok || aa.Action == nil { + continue + } + var eh microflows.ErrorHandlingType + switch a := aa.Action.(type) { + case *microflows.CreateListAction: + eh = a.ErrorHandlingType + case *microflows.ChangeListAction: + eh = a.ErrorHandlingType + case *microflows.ListOperationAction: + eh = a.ErrorHandlingType + case *microflows.AggregateListAction: + eh = a.ErrorHandlingType + case *microflows.CastAction: + eh = a.ErrorHandlingType + default: + continue + } + n++ + if eh != tc.want { + t.Errorf("set %q: %T read back %q, want %q", tc.set, aa.Action, eh, tc.want) + } + } + if n != 5 { + t.Fatalf("set %q: %d actions survived the round trip, want 5", tc.set, n) + } + } +} + +func setActionID(a microflows.MicroflowAction, id model.ID) { + switch x := a.(type) { + case *microflows.CreateListAction: + x.ID = id + case *microflows.ChangeListAction: + x.ID = id + case *microflows.ListOperationAction: + x.ID = id + case *microflows.AggregateListAction: + x.ID = id + case *microflows.CastAction: + x.ID = id + } +} + +// mendixlabs/mxcli#870: the writer omitted the workflow selection whenever the +// "Pause instances" / "Unpause instances" flag was set, reading the flag as +// "all workflows". Studio Pro stores the flag WITH the selection +// (WorkflowCommons.ACT_WorkflowDefinition_Lock in ako/TestApp), and without +// one the activity is CE1825. +func TestLockWorkflow_FlagKeepsTheSelection(t *testing.T) { + lock := µflows.LockWorkflowAction{PauseAllWorkflows: true, WorkflowVariable: "WorkflowDefinition"} + lock.ID = "l-1" + unlock := µflows.UnlockWorkflowAction{ResumeAllPausedWorkflows: true, Workflow: "M.Approve"} + unlock.ID = "u-1" + oc := µflows.MicroflowObjectCollection{} + for i, a := range []microflows.MicroflowAction{lock, unlock} { + act := µflows.ActionActivity{Action: a} + act.ID = model.ID(fmt.Sprintf("act-%d", i)) + act.Position = model.Point{X: 100 * i, Y: 100} + oc.Objects = append(oc.Objects, act) + } + mf := µflows.Microflow{Name: "MF", ObjectCollection: oc} + mf.ID = "mf-1" + got := roundTripMicroflow(t, mf) + n := 0 + for _, obj := range got.ObjectCollection.Objects { + aa, _ := obj.(*microflows.ActionActivity) + if aa == nil { + continue + } + switch a := aa.Action.(type) { + case *microflows.LockWorkflowAction: + n++ + if !a.PauseAllWorkflows || a.WorkflowVariable != "WorkflowDefinition" { + t.Errorf("lock read back flag=%v var=%q, want true/WorkflowDefinition", a.PauseAllWorkflows, a.WorkflowVariable) + } + case *microflows.UnlockWorkflowAction: + n++ + if !a.ResumeAllPausedWorkflows || a.Workflow != "M.Approve" { + t.Errorf("unlock read back flag=%v workflow=%q, want true/M.Approve", a.ResumeAllPausedWorkflows, a.Workflow) + } + } + } + if n != 2 { + t.Fatalf("%d actions survived, want 2", n) + } +} diff --git a/mdl/backend/modelsdk/microflow_read_actions.go b/mdl/backend/modelsdk/microflow_read_actions.go index 8e49cd756f..2f1bf36f2d 100644 --- a/mdl/backend/modelsdk/microflow_read_actions.go +++ b/mdl/backend/modelsdk/microflow_read_actions.go @@ -197,15 +197,17 @@ func actionFromGen(el element.Element) microflows.MicroflowAction { out := µflows.CreateListAction{ EntityQualifiedName: a.EntityQualifiedName(), OutputVariable: a.OutputVariableName(), + ErrorHandlingType: microflows.ErrorHandlingType(a.ErrorHandlingType()), } out.ID = model.ID(a.ID()) return out case *genMf.ChangeListAction: out := µflows.ChangeListAction{ - ChangeVariable: a.ChangeVariableName(), - Type: microflows.ChangeListType(a.Type()), - Value: a.Value(), + ChangeVariable: a.ChangeVariableName(), + Type: microflows.ChangeListType(a.Type()), + Value: a.Value(), + ErrorHandlingType: microflows.ErrorHandlingType(a.ErrorHandlingType()), } out.ID = model.ID(a.ID()) return out @@ -223,6 +225,7 @@ func actionFromGen(el element.Element) microflows.MicroflowAction { // them back silently deleted the fold (#1004). ReduceInitialValue: a.ReduceInitialValueExpression(), ReduceReturnType: dataTypeFromGen(a.ReduceReturnDataType()), + ErrorHandlingType: microflows.ErrorHandlingType(a.ErrorHandlingType()), } out.ID = model.ID(a.ID()) return out @@ -230,7 +233,10 @@ func actionFromGen(el element.Element) microflows.MicroflowAction { case *genMf.CastAction: // ObjectVariable (the cast input) is not stored via a gen setter, so it is // not reconstructable here; OutputVariable is. - out := µflows.CastAction{OutputVariable: a.OutputVariableName()} + out := µflows.CastAction{ + OutputVariable: a.OutputVariableName(), + ErrorHandlingType: microflows.ErrorHandlingType(a.ErrorHandlingType()), + } out.ID = model.ID(a.ID()) return out @@ -369,7 +375,10 @@ func actionFromGen(el element.Element) microflows.MicroflowAction { // keys), so read both from the raw BSON — the inverse of the write's // listOperationToGen. raw := a.Raw() - out := µflows.ListOperationAction{OutputVariable: rawStr(raw, "ResultVariableName")} + out := µflows.ListOperationAction{ + OutputVariable: rawStr(raw, "ResultVariableName"), + ErrorHandlingType: microflows.ErrorHandlingType(rawStr(raw, "ErrorHandlingType")), + } out.ID = model.ID(a.ID()) if opDoc, ok := raw.Lookup("NewOperation").DocumentOK(); ok { out.Operation = listOperationFromRaw(opDoc) diff --git a/mdl/backend/modelsdk/microflow_workflow_write.go b/mdl/backend/modelsdk/microflow_workflow_write.go index 74b32f37c2..a23bc9d54c 100644 --- a/mdl/backend/modelsdk/microflow_workflow_write.go +++ b/mdl/backend/modelsdk/microflow_workflow_write.go @@ -86,7 +86,11 @@ func workflowMicroflowActionToGen(act microflows.MicroflowAction) element.Elemen g := newElem("Microflows$LockWorkflowAction", string(a.ID)) addStr(g, "ErrorHandlingType", orDefault(string(a.ErrorHandlingType), "Rollback")) addBool(g, "PauseAllWorkflows", a.PauseAllWorkflows) - if !a.PauseAllWorkflows { + // The flag is "Pause instances" on the selected workflow, not "all + // workflows": Studio Pro stores it WITH the selection, and leaving the + // selection out for it was CE1825 (mendixlabs/mxcli#870). Only a lock + // naming nothing — refused by check and exec — has none to write. + if a.Workflow != "" || a.WorkflowVariable != "" { addPart(g, "WorkflowSelection", workflowSelectionToGen(a.Workflow, a.WorkflowVariable)) } return g @@ -94,7 +98,7 @@ func workflowMicroflowActionToGen(act microflows.MicroflowAction) element.Elemen g := newElem("Microflows$UnlockWorkflowAction", string(a.ID)) addStr(g, "ErrorHandlingType", orDefault(string(a.ErrorHandlingType), "Rollback")) addBool(g, "ResumeAllPausedWorkflows", a.ResumeAllPausedWorkflows) - if !a.ResumeAllPausedWorkflows { + if a.Workflow != "" || a.WorkflowVariable != "" { addPart(g, "WorkflowSelection", workflowSelectionToGen(a.Workflow, a.WorkflowVariable)) } return g diff --git a/mdl/backend/modelsdk/microflow_write.go b/mdl/backend/modelsdk/microflow_write.go index a81a4df8c1..a7983791ef 100644 --- a/mdl/backend/modelsdk/microflow_write.go +++ b/mdl/backend/modelsdk/microflow_write.go @@ -702,7 +702,7 @@ func microflowActionToGen(action microflows.MicroflowAction) element.Element { // Storage $Type Microflows$CastAction; output bound to "VariableName". g := genMf.NewCastAction() g.SetID(element.ID(a.ID)) - g.SetErrorHandlingType("Rollback") + g.SetErrorHandlingType(orDefault(string(a.ErrorHandlingType), "Rollback")) g.SetOutputVariableName(a.OutputVariable) return g case *microflows.AggregateListAction: @@ -711,7 +711,7 @@ func microflowActionToGen(action microflows.MicroflowAction) element.Element { // by-name ref. Expression mode is mutually exclusive with Attribute. g := genMf.NewAggregateListAction() g.SetID(element.ID(a.ID)) - g.SetErrorHandlingType("Rollback") + g.SetErrorHandlingType(orDefault(string(a.ErrorHandlingType), "Rollback")) g.SetAggregateFunction(string(a.Function)) g.SetInputListVariableName(a.InputVariable) if a.UseExpression { @@ -735,7 +735,10 @@ func microflowActionToGen(action microflows.MicroflowAction) element.Element { // Storage $Type Microflows$CreateListAction; output bound to "VariableName". g := genMf.NewCreateListAction() g.SetID(element.ID(a.ID)) - g.SetErrorHandlingType("Rollback") + // The flow flavour's default, which the builder supplies: Rollback in a + // microflow, Abort in a nanoflow — "Rollback" there is CE6035 + // (mendixlabs/mxcli#591). Same for the other list activities and cast. + g.SetErrorHandlingType(orDefault(string(a.ErrorHandlingType), "Rollback")) if a.EntityQualifiedName != "" { g.SetEntityQualifiedName(a.EntityQualifiedName) } @@ -747,7 +750,7 @@ func microflowActionToGen(action microflows.MicroflowAction) element.Element { // when called, so guard it the same way. g := genMf.NewChangeListAction() g.SetID(element.ID(a.ID)) - g.SetErrorHandlingType("Rollback") + g.SetErrorHandlingType(orDefault(string(a.ErrorHandlingType), "Rollback")) g.SetChangeVariableName(a.ChangeVariable) g.SetType(string(a.Type)) if a.Value != "" { @@ -780,7 +783,7 @@ func microflowActionToGen(action microflows.MicroflowAction) element.Element { // "VariableName"), so this action and its operation sub-elements are // built directly with the verified legacy BSON keys. e := newElem("Microflows$ListOperationsAction", string(a.ID)) - addStr(e, "ErrorHandlingType", "Rollback") + addStr(e, "ErrorHandlingType", orDefault(string(a.ErrorHandlingType), "Rollback")) if a.Operation != nil { addPart(e, "NewOperation", listOperationToGen(a.Operation)) } diff --git a/mdl/backend/modelsdk/published_rest_write.go b/mdl/backend/modelsdk/published_rest_write.go index 8370e4e4d6..919db33b7e 100644 --- a/mdl/backend/modelsdk/published_rest_write.go +++ b/mdl/backend/modelsdk/published_rest_write.go @@ -13,25 +13,27 @@ import ( "github.com/mendixlabs/mxcli/modelsdk/element" mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" "github.com/mendixlabs/mxcli/modelsdk/property" + "github.com/mendixlabs/mxcli/sdk/microflows" ) func init() { - // Resources / Operations / operation Parameters serialize with the typed-array - // marker 2 (populated keyed by child $Type; empty via MandatoryListMarkers). The - // service's AllowedRoles is a marker-1 reference-string list, AuthenticationTypes - // and Parameters are empty marker-2 lists, and CorsConfiguration is BSON null. - codec.RegisterListMarker("Rest$PublishedRestServiceResource", 2) + // List markers as Studio Pro 11 writes them (measured on ako/TestApp's + // Services.OrdersRestApi): Resources and operation Parameters use 3, + // Operations uses 2, and the service's empty Parameters list is [3]. The + // service's AllowedRoles and AuthenticationTypes are marker-1 string lists, + // and CorsConfiguration is BSON null. + codec.RegisterListMarker("Rest$PublishedRestServiceResource", 3) codec.RegisterListMarker("Rest$PublishedRestServiceOperation", 2) - codec.RegisterListMarker("Rest$RestOperationParameter", 2) + codec.RegisterListMarker("Rest$RestOperationParameter", 3) codec.RegisterTypeDefaults("Rest$PublishedRestService", codec.TypeDefaults{ - MandatoryListMarkers: map[string]int32{"AllowedRoles": 1, "AuthenticationTypes": 2, "Parameters": 2}, + MandatoryListMarkers: map[string]int32{"AllowedRoles": 1, "AuthenticationTypes": 1, "Parameters": 3}, NullFields: []string{"CorsConfiguration"}, }) codec.RegisterTypeDefaults("Rest$PublishedRestServiceResource", codec.TypeDefaults{ MandatoryListMarkers: map[string]int32{"Operations": 2}, }) codec.RegisterTypeDefaults("Rest$PublishedRestServiceOperation", codec.TypeDefaults{ - MandatoryListMarkers: map[string]int32{"Parameters": 2}, + MandatoryListMarkers: map[string]int32{"Parameters": 3}, }) } @@ -73,6 +75,12 @@ func (b *Backend) UpdatePublishedRestService(svc *model.PublishedRestService) er if err != nil { return fmt.Errorf("UpdatePublishedRestService: %w", err) } + // ... and the service-level properties MDL has no spelling for as + // constants too: carry the stored ones (ako/mxcli#571). + contents, err = b.keepStoredTopLevel(string(svc.ID), contents, publishedRestServiceUnauthored) + if err != nil { + return fmt.Errorf("UpdatePublishedRestService: %w", err) + } return b.writer.UpdateRawUnit(string(svc.ID), contents) } @@ -121,6 +129,20 @@ func (b *Backend) UpdatePublishedRestServiceRoles(unitID model.ID, roles []strin return b.writer.UpdateRawUnit(string(unitID), out) } +// publishedRestServiceUnauthored are the Rest$PublishedRestService keys a +// create or modify / alter cannot state: the writer emits a constant for each, +// so a rewrite carries the stored value instead. Without the carry, executing +// the describe output of a Studio Pro service turned its Basic and Session +// authentication off (ako/mxcli#571). +var publishedRestServiceUnauthored = []string{ + "AuthenticationMicroflow", + "AuthenticationTypes", + "CorsConfiguration", + "Documentation", + "Parameters", + "PublicDocumentation", +} + func publishedRestServiceToGen(svc *model.PublishedRestService) element.Element { g := newElem("Rest$PublishedRestService", string(svc.ID)) addStr(g, "Name", svc.Name) @@ -164,26 +186,36 @@ func publishedRestOperationToGen(op *model.PublishedRestOperation) element.Eleme addStr(g, "Microflow", op.Microflow) addStr(g, "Summary", op.Summary) addBool(g, "Deprecated", op.Deprecated) - addStr(g, "Commit", "Yes") - addStr(g, "Documentation", "") - addStr(g, "ExportMapping", "") - addStr(g, "ImportMapping", "") - addStr(g, "ObjectHandlingBackup", "Create") - // Path parameters are auto-extracted from {name} placeholders and wired to the - // matching microflow parameter (Module.Microflow.name) — without that wiring - // mx check raises CE6538 / CE0350. - params := make([]element.Element, 0) - for _, name := range extractPathParams(op.Path) { - p := newElem("Rest$RestOperationParameter", "") - addStr(p, "Name", name) - addPart(p, "Type", newElem("DataTypes$StringType", "")) - addStr(p, "ParameterType", "Path") - mfParam := "" - if op.Microflow != "" { - mfParam = op.Microflow + "." + name + addStr(g, "Commit", orDefault(op.Commit, "Yes")) + addStr(g, "Documentation", op.Documentation) + addStr(g, "ExportMapping", op.ExportMapping) + addStr(g, "ImportMapping", op.ImportMapping) + addStr(g, "ObjectHandlingBackup", orDefault(op.ObjectHandlingBackup, "Create")) + // The executor derives the parameters from the microflow (path, query, body), + // as Studio Pro does. When it could not read the microflow only the path's + // {name} placeholders are known: those are written as String path + // parameters wired to the microflow parameter of that name, since without + // that wiring mx check raises CE6538 / CE0350. + opParams := op.OperationParameters + if len(opParams) == 0 { + for _, name := range op.PathParameterNames() { + mfParam := "" + if op.Microflow != "" { + mfParam = op.Microflow + "." + name + } + opParams = append(opParams, &model.PublishedRestOperationParameter{ + Name: name, ParameterType: "Path", MicroflowParameter: mfParam, DataType: "String", + }) } - addStr(p, "MicroflowParameter", mfParam) - addStr(p, "Description", "") + } + params := make([]element.Element, 0, len(opParams)) + for _, param := range opParams { + p := newElem("Rest$RestOperationParameter", "") + addStr(p, "Name", param.Name) + addPart(p, "Type", publishedRestParameterTypeToGen(param)) + addStr(p, "ParameterType", param.ParameterType) + addStr(p, "MicroflowParameter", param.MicroflowParameter) + addStr(p, "Description", param.Description) params = append(params, p) } if len(params) > 0 { @@ -192,6 +224,49 @@ func publishedRestOperationToGen(op *model.PublishedRestOperation) element.Eleme return g } +// publishedRestParameterTypeToGen builds the DataTypes$* element of an +// operation parameter. Long is not among an operation parameter's types +// (Studio Pro 11.14's schema for Rest$RestOperationParameter.type): it is +// written as Integer, as microflowDataTypeToGen writes it. +func publishedRestParameterTypeToGen(p *model.PublishedRestOperationParameter) element.Element { + switch p.DataType { + case "Float": + return newElem("DataTypes$FloatType", "") + case "Empty": + return newElem("DataTypes$EmptyType", "") + case "Unknown": + return newElem("DataTypes$UnknownType", "") + } + return microflowDataTypeToGen(publishedRestParameterDataType(p)) +} + +// publishedRestParameterDataType is the microflow data type an operation +// parameter carries. +func publishedRestParameterDataType(p *model.PublishedRestOperationParameter) microflows.DataType { + switch p.DataType { + case "Boolean": + return µflows.BooleanType{} + case "Integer", "Long": + return µflows.IntegerType{} + case "Decimal": + return µflows.DecimalType{} + case "DateTime", "Date": + return µflows.DateTimeType{} + case "Binary": + return µflows.BinaryType{} + case "Enumeration": + return µflows.EnumerationType{EnumerationQualifiedName: p.QualifiedName} + case "Object": + return µflows.ObjectType{EntityQualifiedName: p.QualifiedName} + case "List": + return µflows.ListType{EntityQualifiedName: p.QualifiedName} + case "Void": + return nil + default: + return µflows.StringType{} + } +} + // addByNameRefList adds a marker-1 reference-string list property (qualified // names), the form Mendix uses for AllowedRoles / AllowedModuleRoles. func addByNameRefList(b *element.Base, name, targetType string, qnames []string) { @@ -202,24 +277,6 @@ func addByNameRefList(b *element.Base, name, targetType string, qnames []string) } } -// extractPathParams returns parameter names from {param} placeholders in a path. -func extractPathParams(path string) []string { - var names []string - for { - start := strings.Index(path, "{") - if start < 0 { - break - } - end := strings.Index(path[start:], "}") - if end < 0 { - break - } - names = append(names, path[start+1:start+end]) - path = path[start+end+1:] - } - return names -} - // httpMethodToMendix converts an HTTP method name to Mendix casing. func httpMethodToMendix(method string) string { switch strings.ToUpper(method) { diff --git a/mdl/backend/modelsdk/published_rest_write_test.go b/mdl/backend/modelsdk/published_rest_write_test.go index 3a0781d1d1..6c4766bc67 100644 --- a/mdl/backend/modelsdk/published_rest_write_test.go +++ b/mdl/backend/modelsdk/published_rest_write_test.go @@ -3,8 +3,11 @@ package modelsdkbackend import ( + "strings" "testing" + "go.mongodb.org/mongo-driver/bson" + "github.com/mendixlabs/mxcli/model" ) @@ -72,3 +75,125 @@ func TestCreatePublishedRestService_RoundTrip(t *testing.T) { t.Errorf("operations = %d, want 2", len(got.Resources[0].Operations)) } } + +// TestCreatePublishedRestService_WritesParametersAndBindings is ako/mxcli#571 / +// mendixlabs/mxcli#1206 at the storage layer: the operation's query and body +// parameters, each with its own type, its mapping bindings and its commit +// option are written and read back. Before, only String path parameters were +// written, and ExportMapping / ImportMapping / Commit were constants. +func TestCreatePublishedRestService_WritesParametersAndBindings(t *testing.T) { + proj := copyFixture(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + mod, err := b.GetModuleByName("MyFirstModule") + if err != nil || mod == nil { + t.Fatalf("GetModuleByName: %v", err) + } + params := []*model.PublishedRestOperationParameter{ + {Name: "id", ParameterType: "Path", MicroflowParameter: "MyFirstModule.ACT_Put.id", DataType: "Integer"}, + {Name: "verbose", ParameterType: "Query", MicroflowParameter: "MyFirstModule.ACT_Put.verbose", DataType: "Boolean"}, + {Name: "X-Mode", ParameterType: "Header", MicroflowParameter: "MyFirstModule.ACT_Put.mode", DataType: "Enumeration", QualifiedName: "MyFirstModule.Mode", Description: "mode"}, + {Name: "body", ParameterType: "Body", MicroflowParameter: "MyFirstModule.ACT_Put.body", DataType: "Object", QualifiedName: "MyFirstModule.Thing"}, + } + svc := &model.PublishedRestService{ + ContainerID: mod.ID, Name: "ZzParams", Path: "rest/zzp/v1", + Resources: []*model.PublishedRestResource{{ + Name: "things", + Operations: []*model.PublishedRestOperation{ + { + Path: "{id}", HTTPMethod: "PUT", Microflow: "MyFirstModule.ACT_Put", + ImportMapping: "MyFirstModule.IMM_Thing", ExportMapping: "MyFirstModule.EMM_Thing", + Commit: "YesWithoutEvents", ObjectHandlingBackup: "Error", + OperationParameters: params, + }, + // No derived parameters: the path's placeholder as a String (the + // writer's fallback when the microflow could not be read). + {Path: "{key}", HTTPMethod: "GET", Microflow: "MyFirstModule.ACT_Get"}, + }, + }}, + } + if err := b.CreatePublishedRestService(svc); err != nil { + t.Fatalf("CreatePublishedRestService: %v", err) + } + + b2 := New() + if err := b2.Connect(proj); err != nil { + t.Fatalf("reconnect: %v", err) + } + t.Cleanup(func() { _ = b2.Disconnect() }) + all, err := b2.ListPublishedRestServices() + if err != nil { + t.Fatalf("ListPublishedRestServices: %v", err) + } + var got *model.PublishedRestService + for _, s := range all { + if s.Name == "ZzParams" { + got = s + } + } + if got == nil { + t.Fatal("ZzParams not found") + } + op := got.Resources[0].Operations[0] + if op.ImportMapping != "MyFirstModule.IMM_Thing" || op.ExportMapping != "MyFirstModule.EMM_Thing" || + op.Commit != "YesWithoutEvents" || op.ObjectHandlingBackup != "Error" { + t.Errorf("bindings read back as import %q export %q commit %q handling %q", + op.ImportMapping, op.ExportMapping, op.Commit, op.ObjectHandlingBackup) + } + if len(op.OperationParameters) != len(params) { + t.Fatalf("parameters = %d, want %d: %+v", len(op.OperationParameters), len(params), op.OperationParameters) + } + for i, want := range params { + if g := *op.OperationParameters[i]; g != *want { + t.Errorf("parameter %d = %+v, want %+v", i, g, *want) + } + } + fallback := got.Resources[0].Operations[1] + if len(fallback.OperationParameters) != 1 || *fallback.OperationParameters[0] != (model.PublishedRestOperationParameter{ + Name: "key", ParameterType: "Path", MicroflowParameter: "MyFirstModule.ACT_Get.key", DataType: "String", + }) { + t.Errorf("fallback parameters = %+v", fallback.OperationParameters) + } + if fallback.Commit != "Yes" || fallback.ObjectHandlingBackup != "Create" { + t.Errorf("defaults = commit %q handling %q, want Yes / Create", fallback.Commit, fallback.ObjectHandlingBackup) + } +} + +// TestWithStoredTopLevel carries the stored values of the listed keys, +// inserts a stored-only key in sorted position, and leaves every other key as +// written. The control: an unlisted key keeps the written value. +func TestWithStoredTopLevel(t *testing.T) { + written, _ := bson.Marshal(bson.D{ + {Key: "$ID", Value: "x"}, {Key: "AuthenticationTypes", Value: bson.A{int32(1)}}, + {Key: "Name", Value: "new"}, {Key: "Version", Value: "2"}, + }) + stored, _ := bson.Marshal(bson.D{ + {Key: "$ID", Value: "x"}, {Key: "AuthenticationTypes", Value: bson.A{int32(1), "Basic"}}, + {Key: "Name", Value: "old"}, {Key: "PublicDocumentation", Value: ""}, {Key: "Version", Value: "1"}, + }) + out, err := withStoredTopLevel(written, stored, []string{"AuthenticationTypes", "PublicDocumentation", "Missing"}) + if err != nil { + t.Fatal(err) + } + var d bson.D + if err := bson.Unmarshal(out, &d); err != nil { + t.Fatal(err) + } + var keys []string + for _, e := range d { + keys = append(keys, e.Key) + } + if got := strings.Join(keys, ","); got != "$ID,AuthenticationTypes,Name,PublicDocumentation,Version" { + t.Errorf("keys = %s", got) + } + m := d.Map() + if a, ok := m["AuthenticationTypes"].(bson.A); !ok || len(a) != 2 || a[1] != "Basic" { + t.Errorf("AuthenticationTypes = %v, want the stored [1 Basic]", m["AuthenticationTypes"]) + } + if m["Name"] != "new" || m["Version"] != "2" { + t.Errorf("unlisted keys changed: Name %v Version %v", m["Name"], m["Version"]) + } +} diff --git a/mdl/backend/modelsdk/widget_write.go b/mdl/backend/modelsdk/widget_write.go index a39d7e18d2..ca37e6755e 100644 --- a/mdl/backend/modelsdk/widget_write.go +++ b/mdl/backend/modelsdk/widget_write.go @@ -490,7 +490,7 @@ func widgetToGen(w pages.Widget) (element.Element, error) { if x.Label != "" { g.SetLabelTemplate(textAsClientTemplate(textFromString(x.Label))) } - g.SetMaxLengthCode(-1) + g.SetMaxLengthCode(maxLengthCodeFor(x.SourceVariable)) onChange, err := clientActionToGen(x.OnChangeAction) if err != nil { return nil, err @@ -584,7 +584,7 @@ func widgetToGen(w pages.Widget) (element.Element, error) { if x.Label != "" { g.SetLabelTemplate(textAsClientTemplate(textFromString(x.Label))) } - g.SetMaxLengthCode(-1) + g.SetMaxLengthCode(maxLengthCodeFor(x.SourceVariable)) lines := int32(x.Rows) if lines == 0 { lines = 5 @@ -1291,6 +1291,19 @@ func pageVariableToGen(widget, name, kind string) element.Element { return pv } +// maxLengthCodeFor is a text input's "Maximum length": -1, the attribute's +// length, for an attribute binding; 0, unlimited, for one bound to a page +// variable, which has no attribute length to take — Studio Pro's default for +// variables, and the only valid one: mx check reports -1 there as CE6553 "Text +// length cannot be the attribute length when configured with a variable" +// (mendixlabs/mxcli#1235). +func maxLengthCodeFor(sv *pages.WidgetVariable) int32 { + if sv != nil && sv.Kind == "local" && sv.Widget == "" && sv.Variable != "" { + return 0 + } + return -1 +} + // inputSourceVariableToGen writes an input widget's widget-scoped // SourceVariable, or nil — the null Studio Pro stores on an input bound to its // enclosing data context. diff --git a/mdl/backend/modelsdk/widget_write_maxlength_test.go b/mdl/backend/modelsdk/widget_write_maxlength_test.go new file mode 100644 index 0000000000..885812f582 --- /dev/null +++ b/mdl/backend/modelsdk/widget_write_maxlength_test.go @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// mendixlabs/mxcli#1235: a text box or text area bound to a page variable has +// no attribute length; -1 there is mx check CE6553. +func TestMaxLengthCodeFor(t *testing.T) { + cases := []struct { + name string + sv *pages.WidgetVariable + want int32 + }{ + {"attribute binding", nil, -1}, + {"through a data view", &pages.WidgetVariable{Widget: "dv", Variable: "Account"}, -1}, + {"page variable", &pages.WidgetVariable{Variable: "Filter", Kind: "local"}, 0}, + } + for _, c := range cases { + if got := maxLengthCodeFor(c.sv); got != c.want { + t.Errorf("%s: MaxLengthCode = %d, want %d", c.name, got, c.want) + } + } +} diff --git a/mdl/backend/pagemutator/alter_target.go b/mdl/backend/pagemutator/alter_target.go index 07cd5cb06b..0a150d3636 100644 --- a/mdl/backend/pagemutator/alter_target.go +++ b/mdl/backend/pagemutator/alter_target.go @@ -28,6 +28,9 @@ func (m *Mutator) ResolveAlterTarget(t backend.AlterTarget) (backend.AlterTarget if kind, ok, err := m.resolveScrollRegion(container, member); ok { return backend.AlterTargetMatch{Kind: kind, Name: name}, err } + if _, ok := m.findDataViewFooter(container, member); ok { + return backend.AlterTargetMatch{Kind: "region", Name: name}, nil + } if _, err := findBsonColumn(m.rawData, container, member, m.widgetFinder); err != nil { return backend.AlterTargetMatch{}, err } diff --git a/mdl/backend/pagemutator/editable_test.go b/mdl/backend/pagemutator/editable_test.go new file mode 100644 index 0000000000..b09e9d297e --- /dev/null +++ b/mdl/backend/pagemutator/editable_test.go @@ -0,0 +1,107 @@ +// SPDX-License-Identifier: Apache-2.0 + +package pagemutator + +import ( + "testing" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/backend/bsonnav" +) + +// mendixlabs/mxcli#1214: `alter page … set Editable = true on ` +// reported "Altered page" and stored nothing. A list view's (and data view's) +// Editable is a BOOLEAN; the setter only wrote string values (the input-widget +// enum), so a boolean fell through and returned nil. +func TestSetWidgetProperty_Editable_ListViewBoolean(t *testing.T) { + for _, typ := range []string{"Forms$ListView", "Forms$DataView"} { + t.Run(typ, func(t *testing.T) { + w := bson.D{ + {Key: "$Type", Value: typ}, + {Key: "Name", Value: "lv"}, + {Key: "Editable", Value: false}, + } + rawData := makeRawPage(w) + m := &Mutator{rawData: rawData, widgetFinder: findBsonWidget} + if err := m.SetWidgetProperty("lv", "Editable", true); err != nil { + t.Fatalf("set Editable = true: %v", err) + } + if got := bsonnav.DGet(findBsonWidget(rawData, "lv").widget, "Editable"); got != true { + t.Fatalf("Editable = %#v after set Editable = true, want true", got) + } + // Back to false, so the test is not satisfied by a setter that only ever writes true. + if err := m.SetWidgetProperty("lv", "editable", false); err != nil { + t.Fatalf("set Editable = false: %v", err) + } + if got := bsonnav.DGet(findBsonWidget(rawData, "lv").widget, "Editable"); got != false { + t.Fatalf("Editable = %#v after set Editable = false, want false", got) + } + // The input-widget enum is not a list view's vocabulary: refuse, never no-op. + if err := m.SetWidgetProperty("lv", "Editable", "Never"); err == nil { + t.Fatal("set Editable = 'Never' on a list view succeeded; want an error naming true/false") + } + }) + } +} + +// An input widget's Editable is the Always/Never/Conditional enum. A boolean +// used to be dropped silently; a lower-case enum was stored verbatim, which is +// not a value Studio Pro reads. +func TestSetWidgetProperty_Editable_InputEnum(t *testing.T) { + w := bson.D{ + {Key: "$Type", Value: "Forms$TextBox"}, + {Key: "Name", Value: "tb"}, + {Key: "Editable", Value: "Always"}, + {Key: "ConditionalEditabilitySettings", Value: nil}, + } + rawData := makeRawPage(w) + m := &Mutator{rawData: rawData, widgetFinder: findBsonWidget} + + if err := m.SetWidgetProperty("tb", "Editable", "never"); err != nil { + t.Fatalf("set Editable = 'never': %v", err) + } + if got := bsonnav.DGet(findBsonWidget(rawData, "tb").widget, "Editable"); got != "Never" { + t.Fatalf("Editable = %#v, want the canonical \"Never\"", got) + } + if err := m.SetWidgetProperty("tb", "Editable", true); err == nil { + t.Fatal("set Editable = true on a text box succeeded; want an error naming Always/Never") + } + if err := m.SetWidgetProperty("tb", "Editable", "Conditional"); err == nil { + t.Fatal("set Editable = 'Conditional' succeeded with no expression; want an error pointing at [expr]") + } + + // EDITABLE IF writes the settings element AND the enum that says it applies, + // as CREATE does (pages.WidgetEditability). Setting a plain value afterwards + // clears the element again, so the two never contradict each other. + if err := m.SetWidgetProperty("tb", "EditableIf", "$currentObject/Active"); err != nil { + t.Fatalf("set EditableIf: %v", err) + } + got := findBsonWidget(rawData, "tb").widget + if e := bsonnav.DGet(got, "Editable"); e != "Conditional" { + t.Fatalf("Editable = %#v after set Editable = [expr], want \"Conditional\"", e) + } + if err := m.SetWidgetProperty("tb", "Editable", "Always"); err != nil { + t.Fatalf("set Editable = 'Always': %v", err) + } + got = findBsonWidget(rawData, "tb").widget + if e := bsonnav.DGet(got, "Editable"); e != "Always" { + t.Fatalf("Editable = %#v, want \"Always\"", e) + } + if s := bsonnav.DGet(got, "ConditionalEditabilitySettings"); s != nil { + t.Fatalf("ConditionalEditabilitySettings = %#v after set Editable = 'Always', want null", s) + } +} + +// A widget with no Editable property at all (a container) refuses the set. +func TestSetWidgetProperty_Editable_NoSuchProperty(t *testing.T) { + w := bson.D{ + {Key: "$Type", Value: "Forms$DivContainer"}, + {Key: "Name", Value: "ctn"}, + } + rawData := makeRawPage(w) + m := &Mutator{rawData: rawData, widgetFinder: findBsonWidget} + if err := m.SetWidgetProperty("ctn", "Editable", true); err == nil { + t.Fatal("set Editable on a container succeeded; it has no Editable property") + } +} diff --git a/mdl/backend/pagemutator/footer_region_test.go b/mdl/backend/pagemutator/footer_region_test.go new file mode 100644 index 0000000000..bb8db58e7e --- /dev/null +++ b/mdl/backend/pagemutator/footer_region_test.go @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: Apache-2.0 + +package pagemutator + +import ( + "strings" + "testing" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/bsonnav" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// A data view's footer is a region, not a widget: Studio Pro stores its +// widgets in the data view's FooterWidgets list and the footer itself has no +// Name. ako/mxcli#528 / mendixlabs/mxcli#293: neither the name the script +// wrote nor describe's invented `footer1` resolved, so a footer could not be +// edited at all. It is addressed as `.footer`. +func footerPage(footer ...bson.D) bson.D { + fw := bson.A{int32(2)} + for _, w := range footer { + fw = append(fw, w) + } + return makeRawPage( + bson.D{ + {Key: "$Type", Value: "Forms$DataView"}, + {Key: "Name", Value: "dvMain"}, + {Key: "ShowFooter", Value: len(footer) > 0}, + {Key: "Widgets", Value: bson.A{int32(2), bson.D{{Key: "$Type", Value: "Forms$TextBox"}, {Key: "Name", Value: "txtReason"}}}}, + {Key: "FooterWidgets", Value: fw}, + }, + bson.D{{Key: "$Type", Value: "Forms$DivContainer"}, {Key: "Name", Value: "ctn"}}, + ) +} + +func button(name string) bson.D { + return bson.D{{Key: "$Type", Value: "Forms$ActionButton"}, {Key: "Name", Value: name}} +} + +func newFooterMutator(raw bson.D) *Mutator { + return New(raw, model.ID("page-1"), &stubWidgetDeps{}) +} + +func footerNames(t *testing.T, m *Mutator) []string { + t.Helper() + dv := findBsonWidget(m.rawData, "dvMain") + if dv == nil { + t.Fatal("dvMain not found") + } + var names []string + for _, el := range bsonnav.DGetArrayElements(bsonnav.DGet(dv.widget, "FooterWidgets")) { + if d, ok := el.(bson.D); ok { + names = append(names, bsonnav.DGetString(d, "Name")) + } + } + return names +} + +func textBoxes(names ...string) []pages.Widget { + var ws []pages.Widget + for _, n := range names { + ws = append(ws, &pages.TextBox{BaseWidget: pages.BaseWidget{Name: n}}) + } + return ws +} + +func TestDataViewFooter_Resolves(t *testing.T) { + m := newFooterMutator(footerPage(button("btnSave"))) + got, err := m.ResolveAlterTarget(backend.AlterTarget{Path: []string{"dvMain", "footer"}}) + if err != nil { + t.Fatalf("dvMain.footer did not resolve: %v", err) + } + if got.Kind != "region" { + t.Errorf("kind = %q, want region", got.Kind) + } + // Control: a member a data view does not have is still refused. + if _, err := m.ResolveAlterTarget(backend.AlterTarget{Path: []string{"dvMain", "header"}}); err == nil { + t.Error("dvMain.header resolved; a data view has no header region") + } + if !m.ResolvesTarget("dvMain", "footer") { + t.Error("ResolvesTarget(dvMain, footer) = false; check would refuse what exec does") + } +} + +func TestDataViewFooter_InsertInto(t *testing.T) { + m := newFooterMutator(footerPage(button("btnSave"))) + if err := m.InsertWidget("dvMain", "footer", backend.InsertPosition("into"), textBoxes("added")); err != nil { + t.Fatalf("insert into dvMain.footer: %v", err) + } + if got := strings.Join(footerNames(t, m), ","); got != "btnSave,added" { + t.Errorf("FooterWidgets = %s, want btnSave,added", got) + } + // BEFORE/AFTER a region means nothing: refuse rather than guess. + if err := m.InsertWidget("dvMain", "footer", backend.InsertPosition("after"), textBoxes("x")); err == nil { + t.Error("insert after dvMain.footer succeeded; only INTO addresses a region") + } +} + +// Inserting into an EMPTY footer turns it on, as a `footer { … }` block does in +// CREATE — otherwise the widgets are stored where nothing renders them. +func TestDataViewFooter_InsertIntoEmptyShowsFooter(t *testing.T) { + m := newFooterMutator(footerPage()) + if err := m.InsertWidget("dvMain", "footer", backend.InsertPosition("into"), textBoxes("added")); err != nil { + t.Fatalf("insert into empty dvMain.footer: %v", err) + } + dv := findBsonWidget(m.rawData, "dvMain").widget + if bsonnav.DGet(dv, "ShowFooter") != true { + t.Errorf("ShowFooter = %v after filling an empty footer, want true", bsonnav.DGet(dv, "ShowFooter")) + } + if got := bsonnav.ToBsonA(bsonnav.DGet(dv, "FooterWidgets")); len(got) == 0 || got[0] != int32(2) { + t.Errorf("FooterWidgets lost its list marker: %v", got) + } +} + +func TestDataViewFooter_Replace(t *testing.T) { + m := newFooterMutator(footerPage(button("btnSave"), button("btnCancel"))) + if err := m.ReplaceWidget("dvMain", "footer", textBoxes("btnS2")); err != nil { + t.Fatalf("replace dvMain.footer: %v", err) + } + if got := strings.Join(footerNames(t, m), ","); got != "btnS2" { + t.Errorf("FooterWidgets = %s, want btnS2 (the whole footer replaced)", got) + } + // The data view's body is untouched. + if findBsonWidget(m.rawData, "txtReason") == nil { + t.Error("replacing the footer removed the data view's body") + } +} + +func TestDataViewFooter_Drop(t *testing.T) { + m := newFooterMutator(footerPage(button("btnSave"))) + if err := m.DropWidget([]backend.WidgetRef{{Widget: "dvMain", Column: "footer"}}); err != nil { + t.Fatalf("drop dvMain.footer: %v", err) + } + if got := footerNames(t, m); len(got) != 0 { + t.Errorf("FooterWidgets = %v after drop, want empty", got) + } + dv := findBsonWidget(m.rawData, "dvMain").widget + if got := bsonnav.ToBsonA(bsonnav.DGet(dv, "FooterWidgets")); len(got) != 1 || got[0] != int32(2) { + t.Errorf("FooterWidgets = %v after drop, want the bare list marker", got) + } + if findBsonWidget(m.rawData, "dvMain") == nil { + t.Error("dropping the footer dropped the data view") + } +} + +// A miss on a name that is not stored names the address that works, when the +// page has a data view footer to point at. +func TestDataViewFooter_NotFoundNamesTheRegion(t *testing.T) { + m := newFooterMutator(footerPage(button("btnSave"))) + err := m.ReplaceWidget("footer1", "", textBoxes("x")) + if err == nil || !strings.Contains(err.Error(), "dvMain.footer") { + t.Errorf("replace footer1: error %v should name dvMain.footer", err) + } +} diff --git a/mdl/backend/pagemutator/footerregion.go b/mdl/backend/pagemutator/footerregion.go new file mode 100644 index 0000000000..46d1428682 --- /dev/null +++ b/mdl/backend/pagemutator/footerregion.go @@ -0,0 +1,206 @@ +// SPDX-License-Identifier: Apache-2.0 + +package pagemutator + +import ( + "fmt" + "strings" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/bsonnav" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// A data view's footer is a REGION, not a widget (ako/mxcli#528, +// mendixlabs/mxcli#293). Studio Pro keeps its widgets in the data view's own +// FooterWidgets list and stores no name for it, so `footer footerButtons { … }` +// had nowhere to keep `footerButtons`, and describe's `footer1` was invented. +// Neither resolved, and a footer could not be edited at all. +// +// It is addressed the way a scroll container's region is — positionally, by the +// owner and the slot: `dvMain.footer`. INSERT INTO appends to it, REPLACE swaps +// its whole content, DROP empties it; BEFORE/AFTER a region mean nothing and are +// refused, as they are for a scroll-container region. + +const dataViewFooterSlot = "footer" + +// isDataViewFooterRef reports whether a dotted member names the footer slot. +func isDataViewFooterRef(member string) bool { + return strings.EqualFold(member, dataViewFooterSlot) +} + +// findDataViewFooter resolves `.footer`. ok is false when the owner is +// not a data view, so the caller falls through to the other dotted forms (a +// DataGrid 2 column called "footer" stays addressable). +func (m *Mutator) findDataViewFooter(owner, member string) (dv *bsonWidgetResult, ok bool) { + if !isDataViewFooterRef(member) { + return nil, false + } + result := m.widgetFinder(m.rawData, owner) + if result == nil { + return nil, false + } + if t := bsonnav.DGetString(result.widget, "$Type"); t != "Forms$DataView" && t != "Pages$DataView" { + return nil, false + } + return result, true +} + +// setDataViewFooter stores the footer's new content, keeping the list marker +// (2, as CREATE writes it — widget_write.go MandatoryListMarkers), and writes +// the data view back into its parent slot: bson.D is a slice, and a grown field +// list that is not written back is a silent no-op reported as success. +func setDataViewFooter(dv *bsonWidgetResult, widgets []any) { + out := bson.A{int32(2)} + if raw := bsonnav.ToBsonA(bsonnav.DGet(dv.widget, "FooterWidgets")); len(raw) > 0 && isListMarker(raw[0]) { + out = bson.A{raw[0]} + } + out = append(out, widgets...) + doc := dv.widget + if !bsonnav.DSet(doc, "FooterWidgets", out) { + doc = append(doc, bson.E{Key: "FooterWidgets", Value: out}) + } + dv.widget = doc + dv.parentArr[dv.index] = doc + bsonnav.DSetArray(dv.parentDoc, dv.parentKey, dv.parentArr) +} + +// dataViewFooterWidgets is the footer's current content, without the marker. +func dataViewFooterWidgets(dv bson.D) []any { + return bsonnav.DGetArrayElements(bsonnav.DGet(dv, "FooterWidgets")) +} + +// insertIntoDataViewFooter handles `insert into .footer { … }`. +func (m *Mutator) insertIntoDataViewFooter(owner, member string, position backend.InsertPosition, widgets []pages.Widget) (bool, error) { + dv, ok := m.findDataViewFooter(owner, member) + if !ok { + return false, nil + } + if !strings.EqualFold(string(position), "into") { + return true, fmt.Errorf("a data view footer can only be an INSERT INTO target; "+ + "to place a widget relative to another, name that widget: `insert %s { … }`", + strings.ToLower(string(position))) + } + newBson, err := m.serializeWidgets(widgets) + if err != nil { + return true, fmt.Errorf("serialize widgets: %w", err) + } + existing := dataViewFooterWidgets(dv.widget) + if len(existing) == 0 && len(newBson) > 0 { + // Filling an empty footer turns it on, as a `footer { … }` block does in + // CREATE; otherwise the widgets are stored where nothing renders them. + bsonnav.DSet(dv.widget, "ShowFooter", true) + } + setDataViewFooter(dv, append(append([]any{}, existing...), newBson...)) + return true, nil +} + +// replaceDataViewFooter handles `replace .footer with { … }`: the +// footer's whole content becomes the new widgets. +func (m *Mutator) replaceDataViewFooter(owner, member string, widgets []pages.Widget) (bool, error) { + dv, ok := m.findDataViewFooter(owner, member) + if !ok { + return false, nil + } + newBson, err := m.serializeWidgets(widgets) + if err != nil { + return true, fmt.Errorf("serialize widgets: %w", err) + } + if len(dataViewFooterWidgets(dv.widget)) == 0 && len(newBson) > 0 { + bsonnav.DSet(dv.widget, "ShowFooter", true) + } + setDataViewFooter(dv, newBson) + return true, nil +} + +// dropDataViewFooter handles `drop .footer`: the footer is emptied. +// The data view's ShowFooter is its own property and is left as stored. +func (m *Mutator) dropDataViewFooter(owner, member string) bool { + dv, ok := m.findDataViewFooter(owner, member) + if !ok { + return false + } + setDataViewFooter(dv, nil) + return true +} + +// dataViewFooterHint names the footer addresses a page has, for a not-found +// message: the name a script wrote on a data view footer, and describe's old +// `footer1`, were never stored, so a miss on one should say what does resolve. +func (m *Mutator) dataViewFooterHint() string { + var owners []string + var walk func(v any) + walk = func(v any) { + switch x := v.(type) { + case bson.D: + if t := bsonnav.DGetString(x, "$Type"); (t == "Forms$DataView" || t == "Pages$DataView") && + len(dataViewFooterWidgets(x)) > 0 { + if n := bsonnav.DGetString(x, "Name"); n != "" { + owners = append(owners, n+"."+dataViewFooterSlot) + } + } + for _, e := range x { + walk(e.Value) + } + case bson.A: + for _, e := range x { + walk(e) + } + } + } + walk(m.rawData) + if len(owners) == 0 { + return "" + } + return ". A data view footer stores no name — address it by its data view: " + strings.Join(owners, ", ") +} + +// ContainedWidgetNames lists the names stored INSIDE what a reference addresses +// — a widget's descendants, or a data view footer's widgets and theirs. A +// REPLACE removes them with its target, so the replacement may reuse them; the +// duplicate-name scope used to count them as taken (mendixlabs/mxcli#293). +func (m *Mutator) ContainedWidgetNames(widgetRef, columnRef string) []string { + var roots []any + switch { + case columnRef != "": + dv, ok := m.findDataViewFooter(widgetRef, columnRef) + if !ok { + return nil + } + roots = dataViewFooterWidgets(dv.widget) + default: + result := m.widgetFinder(m.rawData, widgetRef) + if result == nil { + return nil + } + // The widget's own fields, not the widget: its own name is the target. + for _, e := range result.widget { + roots = append(roots, e.Value) + } + } + var names []string + var walk func(v any) + walk = func(v any) { + switch x := v.(type) { + case bson.D: + if strings.HasPrefix(bsonnav.DGetString(x, "$Type"), "Forms$") || strings.HasPrefix(bsonnav.DGetString(x, "$Type"), "CustomWidgets$CustomWidget") { + if n := bsonnav.DGetString(x, "Name"); n != "" { + names = append(names, n) + } + } + for _, e := range x { + walk(e.Value) + } + case bson.A: + for _, e := range x { + walk(e) + } + } + } + for _, r := range roots { + walk(r) + } + return names +} diff --git a/mdl/backend/pagemutator/mutator.go b/mdl/backend/pagemutator/mutator.go index 499883a9e6..27066d5573 100644 --- a/mdl/backend/pagemutator/mutator.go +++ b/mdl/backend/pagemutator/mutator.go @@ -429,6 +429,9 @@ func (m *Mutator) InsertWidget(widgetRef string, columnRef string, position back if handled, err := m.insertIntoScrollRegion(widgetRef, columnRef, position, widgets); handled { return err } + if handled, err := m.insertIntoDataViewFooter(widgetRef, columnRef, position, widgets); handled { + return err + } // Resolve first, so a mistyped column still reports "not found" (with the // available names) rather than the refusal below. if _, err := findBsonColumn(m.rawData, widgetRef, columnRef, m.widgetFinder); err != nil { @@ -558,6 +561,9 @@ func (m *Mutator) DropWidget(refs []backend.WidgetRef) error { for _, ref := range refs { // Re-find widget each iteration because previous drops mutate the tree. var result *bsonWidgetResult + if ref.IsColumn() && m.dropDataViewFooter(ref.Widget, ref.Column) { + continue + } if ref.IsColumn() { r, err := findBsonColumn(m.rawData, ref.Widget, ref.Column, m.widgetFinder) if err != nil { @@ -583,6 +589,9 @@ func (m *Mutator) DropWidget(refs []backend.WidgetRef) error { func (m *Mutator) ReplaceWidget(widgetRef string, columnRef string, widgets []pages.Widget) error { if columnRef != "" { + if handled, err := m.replaceDataViewFooter(widgetRef, columnRef, widgets); handled { + return err + } // Resolve first, so a mistyped column still reports "not found" (with the // available names) rather than the refusal below. if _, err := findBsonColumn(m.rawData, widgetRef, columnRef, m.widgetFinder); err != nil { @@ -1902,7 +1911,7 @@ func (m *Mutator) widgetNotFoundError(name string) error { "available columns: %s (run DESCRIBE PAGE to confirm)", name, formatColumnNameList(cols)) } - return fmt.Errorf("widget %q not found", name) + return fmt.Errorf("widget %q not found%s", name, m.dataViewFooterHint()) } // formatColumnNameList renders derived column names for an error message: each @@ -2818,6 +2827,76 @@ func setWidgetConditionalSettingMut(widget bson.D, field, typeName, expression s return bsonnav.DSet(widget, field, doc) } +// setWidgetEditableMut writes `set Editable = …`. Two widget families store two +// different things under the same key, so the stored value's type decides what +// the statement may say: +// +// - a list view or data view stores a BOOLEAN — `true` / `false`; +// - an input widget stores the Always / Never / Conditional enum. +// +// Anything else is refused. Before mendixlabs/mxcli#1214 only a string was +// written, so `set Editable = true on lvRows` — the boolean every list view +// takes — returned nil and the command reported "Altered page" with nothing +// stored, leaving the list's inputs read-only at runtime. +func setWidgetEditableMut(widget bson.D, value any) error { + typ := bsonnav.DGetString(widget, "$Type") + switch bsonnav.DGet(widget, "Editable").(type) { + case bool: + b, ok := editableBool(value) + if !ok { + return fmt.Errorf("Editable on a %s is true or false, not %v", widgetTypeLabel(typ), value) + } + bsonnav.DSet(widget, "Editable", b) + return nil + case string: + s, isString := value.(string) + canon, ok := pages.CanonicalEditability(s) + if !isString || !ok { + return fmt.Errorf("Editable on a %s is Always or Never (or `Editable = [expression]` for a condition), not %v", + widgetTypeLabel(typ), value) + } + if canon == "Conditional" { + // The enum without its settings element is a condition with no + // expression; the bracketed form writes both. + return fmt.Errorf("`set Editable = Conditional` needs the condition itself: write `set Editable = [expression]`") + } + bsonnav.DSet(widget, "Editable", canon) + // A plain value replaces a stored condition rather than contradicting it. + bsonnav.DSet(widget, "ConditionalEditabilitySettings", nil) + return nil + default: + return fmt.Errorf("a %s has no Editable property", widgetTypeLabel(typ)) + } +} + +// editableBool reads a list/data view's boolean Editable from MDL, which hands +// it over as a bool (or, quoted, as the string "true"/"false"). +func editableBool(value any) (bool, bool) { + switch v := value.(type) { + case bool: + return v, true + case string: + switch strings.ToLower(strings.TrimSpace(v)) { + case "true": + return true, true + case "false": + return false, true + } + } + return false, false +} + +// widgetTypeLabel is a stored $Type as an author reads it: "Forms$ListView" → "ListView". +func widgetTypeLabel(typ string) string { + if i := strings.LastIndex(typ, "$"); i >= 0 { + typ = typ[i+1:] + } + if typ == "" { + return "widget" + } + return typ +} + func setRawWidgetPropertyMut(widget bson.D, propName string, value any) error { // Property names arrive verbatim from MDL (any case) — `set class on …` is as // valid as `set Class on …`, and `create page` reads them case-insensitively @@ -2864,10 +2943,7 @@ func setRawWidgetPropertyMut(widget bson.D, propName string, value any) error { } return nil case "editable": - if s, ok := value.(string); ok { - bsonnav.DSet(widget, "Editable", s) - } - return nil + return setWidgetEditableMut(widget, value) case "visible": // A page widget has no plain boolean "Visible" field — visibility is modeled // via ConditionalVisibilitySettings. Route static booleans and expression @@ -2900,6 +2976,12 @@ func setRawWidgetPropertyMut(widget bson.D, propName string, value any) error { "Forms$ConditionalEditabilitySettings", expr, false) { return fmt.Errorf("widget does not support conditional editability (only input widgets are editable)") } + // The element only applies under the enum that says so — CREATE writes + // both (pages.WidgetEditability); writing the element alone left the + // stored enum saying Always or Never beside it. + if _, isEnum := bsonnav.DGet(widget, "Editable").(string); isEnum { + bsonnav.DSet(widget, "Editable", "Conditional") + } return nil case "name": if s, ok := value.(string); ok { @@ -3432,3 +3514,17 @@ func errExpressionNotAString(propName string, _ any) error { "set %s = if $currentObject/Featured then 'a' else 'b'", propName, propName) } + +// PageVariableNames lists the stored document's page variables (its +// Forms$LocalVariable entries), by name. +func (m *Mutator) PageVariableNames() []string { + var names []string + for _, el := range bsonnav.DGetArrayElements(bsonnav.DGet(m.rawData, "Variables")) { + if doc, ok := el.(bson.D); ok { + if n := bsonnav.DGetString(doc, "Name"); n != "" { + names = append(names, n) + } + } + } + return names +} diff --git a/mdl/backend/pagemutator/pluggable_replace.go b/mdl/backend/pagemutator/pluggable_replace.go new file mode 100644 index 0000000000..8f3b858f99 --- /dev/null +++ b/mdl/backend/pagemutator/pluggable_replace.go @@ -0,0 +1,427 @@ +// SPDX-License-Identifier: Apache-2.0 + +package pagemutator + +import ( + "bytes" + "strconv" + + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" + + "github.com/mendixlabs/mxcli/mdl/backend/bsonnav" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// REPLACE of a pluggable widget by one of the same widget type keeps what the +// statement does not state (mendixlabs/mxcli#1247) — the passthrough principle +// of ako/mxcli#830, at property granularity. +// +// A pluggable widget is rebuilt from its package TEMPLATE plus what MDL maps +// onto it, so a REPLACE wrote every property MDL has no word for at the +// template's value: measured on TestApp's Studio Pro-authored +// Rules.BusinessRule_NewEdit, adding `sort by Name asc` to comboBox1's data +// source also turned its stored Editable "Never" into "Always" and an +// expression property's PrimitiveValue "" into "false"; the reporter's +// translated placeholder (emptyOptionText) and readOnlyStyle were reset the +// same way. exec reported success, mx check stayed green. +// +// "What the statement states" is measured, not listed: the executor builds the +// stored widget as describe prints it (the BASELINE) next to the replacement. +// A property that comes out the same from both is one the statement did not +// change — the template filled it either way — so the stored value stays. A +// property that differs is the statement's, and its new value is grafted into +// the stored widget, its type pointers re-aimed at the stored Type by property +// key. The stored Type is kept whole, as #830 keeps it. +// +// Anything that cannot be matched up — a different widget package, a stated +// property the stored schema does not declare, a type pointer with no +// counterpart — leaves the plain replacement, which is what REPLACE did before. + +// ReplacePluggableKeepingUnstated replaces the pluggable widget widgetRef names +// with replacement, keeping every stored property on which replacement and +// baseline agree. handled is false when the merge does not apply; the caller +// then replaces as usual. +func (m *Mutator) ReplacePluggableKeepingUnstated(widgetRef string, replacement, baseline pages.Widget) (handled bool, err error) { + if replacement == nil || baseline == nil { + return false, nil + } + result := m.widgetFinder(m.rawData, widgetRef) + if result == nil || len(result.colPropKeys) > 0 { + return false, nil + } + if bsonnav.DGetString(result.widget, "$Type") != "CustomWidgets$CustomWidget" { + return false, nil + } + newDoc := m.deps.SerializeWidget(replacement) + baseDoc := m.deps.SerializeWidget(baseline) + merged, ok := mergeUnstatedPluggable(result.widget, newDoc, baseDoc) + if !ok { + return false, nil + } + result.parentArr[result.index] = merged + bsonnav.DSetArray(result.parentDoc, result.parentKey, result.parentArr) + return true, nil +} + +// mergeUnstatedPluggable returns stored with the replacement's stated changes +// applied, or false when the three cannot be lined up. +func mergeUnstatedPluggable(stored, replacement, baseline bson.D) (bson.D, bool) { + if stored == nil || replacement == nil || baseline == nil { + return nil, false + } + id := pluggableWidgetID(stored) + if id == "" || pluggableWidgetID(replacement) != id || pluggableWidgetID(baseline) != id { + return nil, false + } + out := cloneDoc(stored) + + // The widget's own fields (name, label, editability, appearance, …). + for _, e := range replacement { + switch e.Key { + case "$ID", "$Type", "Type", "Object": + continue + } + if sameIgnoringIdentity(e.Value, bsonnav.DGet(baseline, e.Key)) { + if _, has := docField(out, e.Key); has { + continue + } + } + if !bsonnav.DSet(out, e.Key, e.Value) { + out = append(out, bson.E{Key: e.Key, Value: e.Value}) + } + } + + // The widget's properties, by the path of their property type — which + // names a nested object's properties (a DataGrid 2 column's) as well. + m := pluggableMerge{ + newPaths: typeIDPaths(bsonnav.DGetDoc(replacement, "Type")), + basePaths: typeIDPaths(bsonnav.DGetDoc(baseline, "Type")), + storedPaths: typeIDPaths(bsonnav.DGetDoc(out, "Type")), + } + m.storedIDs = invertPaths(m.storedPaths) + storedObj := bsonnav.DGetDoc(out, "Object") + merged, ok := m.mergeObject(storedObj, bsonnav.DGetDoc(replacement, "Object"), bsonnav.DGetDoc(baseline, "Object")) + if !ok { + return nil, false + } + bsonnav.DSet(out, "Object", merged) + return out, true +} + +// pluggableMerge carries the three Types' ID→path maps through a merge. +type pluggableMerge struct { + newPaths, basePaths, storedPaths map[string]string + storedIDs map[string]string +} + +// propertiesByPath indexes a WidgetObject's properties by their TypePointer's +// path in the Type paths names. The documents are the ones inside obj. +func propertiesByPath(obj bson.D, paths map[string]string) map[string]bson.D { + out := map[string]bson.D{} + for _, p := range bsonnav.DGetArrayElements(bsonnav.DGet(obj, "Properties")) { + d, ok := p.(bson.D) + if !ok { + continue + } + if path := paths[idKey(bsonnav.DGet(d, "TypePointer"))]; path != "" { + out[path] = d + } + } + return out +} + +// mergeObject returns stored (a WidgetObject) with every property on which +// replacement and baseline differ taken from replacement, and every other +// property kept as stored. +func (m pluggableMerge) mergeObject(stored, replacement, baseline bson.D) (bson.D, bool) { + if stored == nil || replacement == nil { + return nil, false + } + out := cloneDoc(stored) + newProps := propertiesByPath(replacement, m.newPaths) + baseProps := propertiesByPath(baseline, m.basePaths) + storedProps := propertiesByPath(out, m.storedPaths) + if len(newProps) == 0 || len(storedProps) == 0 { + return nil, false + } + for path, np := range newProps { + newValue := bsonnav.DGetDoc(np, "Value") + var baseValue bson.D + if bp, ok := baseProps[path]; ok { + baseValue = bsonnav.DGetDoc(bp, "Value") + if sameIgnoringIdentity(newValue, baseValue) { + continue // not the statement's: the stored value stays + } + } + sp, ok := storedProps[path] + if !ok { + return nil, false + } + merged, ok := m.mergeValue(bsonnav.DGetDoc(sp, "Value"), newValue, baseValue) + if !ok { + return nil, false + } + bsonnav.DSet(sp, "Value", merged) + } + return out, true +} + +// mergeValue is one property's new value. When the property is an object +// list (a DataGrid 2's columns) whose objects line up one for one with the +// baseline's, and nothing outside the list changed, each object is merged in +// turn, so changing one column's caption keeps every other column — and the +// changed column's unstated properties — as stored. Otherwise the +// statement's value is grafted whole, re-aimed at the stored Type. +func (m pluggableMerge) mergeValue(stored, replacement, baseline bson.D) (any, bool) { + newObjs := listElements(replacement, "Objects") + baseObjs := listElements(baseline, "Objects") + storedObjs := listElements(stored, "Objects") + if len(newObjs) > 0 && len(newObjs) == len(baseObjs) && len(newObjs) == len(storedObjs) && + sameIgnoringIdentity(withoutField(replacement, "Objects"), withoutField(baseline, "Objects")) { + out := cloneDoc(stored) + objs := bson.A{} + if raw := bsonnav.ToBsonA(bsonnav.DGet(out, "Objects")); len(raw) > 0 && isListMarker(raw[0]) { + objs = append(objs, raw[0]) + } + for i := range newObjs { + nd, ok1 := newObjs[i].(bson.D) + bd, ok2 := baseObjs[i].(bson.D) + sd, ok3 := storedObjs[i].(bson.D) + if !ok1 || !ok2 || !ok3 { + return nil, false + } + merged, ok := m.mergeObject(sd, nd, bd) + if !ok { + return nil, false + } + objs = append(objs, merged) + } + bsonnav.DSet(out, "Objects", objs) + return out, true + } + return remapTypePointers(replacement, m.newPaths, m.storedIDs) +} + +// listElements is a list field's elements without the list marker. +func listElements(d bson.D, key string) []any { + if d == nil { + return nil + } + return bsonnav.DGetArrayElements(bsonnav.DGet(d, key)) +} + +// withoutField is d without one field, for comparing the rest of it. +func withoutField(d bson.D, key string) bson.D { + out := make(bson.D, 0, len(d)) + for _, e := range d { + if e.Key != key { + out = append(out, e) + } + } + return out +} + +func pluggableWidgetID(w bson.D) string { + return bsonnav.DGetString(bsonnav.DGetDoc(w, "Type"), "WidgetId") +} + +func docField(d bson.D, key string) (any, bool) { + for _, e := range d { + if e.Key == key { + return e.Value, true + } + } + return nil, false +} + +// propertiesByKey indexes a pluggable widget's Object properties by their +// PropertyKey, read through the widget's own Type. The returned documents are +// the ones inside w, so setting a field on one changes w. +func propertiesByKey(w bson.D) map[string]bson.D { + keyByID := map[string]string{} + for _, pt := range bsonnav.DGetArrayElements(bsonnav.DGet(bsonnav.DGetDoc(bsonnav.DGetDoc(w, "Type"), "ObjectType"), "PropertyTypes")) { + if d, ok := pt.(bson.D); ok { + keyByID[idKey(bsonnav.DGet(d, "$ID"))] = bsonnav.DGetString(d, "PropertyKey") + } + } + obj := bsonnav.DGetDoc(w, "Object") + if obj == nil || len(keyByID) == 0 { + return nil + } + out := map[string]bson.D{} + for _, p := range bsonnav.DGetArrayElements(bsonnav.DGet(obj, "Properties")) { + d, ok := p.(bson.D) + if !ok { + continue + } + if key := keyByID[idKey(bsonnav.DGet(d, "TypePointer"))]; key != "" { + out[key] = d + } + } + return out +} + +// typeIDPaths maps every element ID in a widget Type to a path that names it +// independently of IDs: the chain of field names, with list elements named by +// their PropertyKey (or Key, or position). +func typeIDPaths(typ bson.D) map[string]string { + out := map[string]string{} + var walk func(d bson.D, path string) + walk = func(d bson.D, path string) { + if id := idKey(bsonnav.DGet(d, "$ID")); id != "" { + out[id] = path + } + for _, e := range d { + switch v := e.Value.(type) { + case bson.D: + walk(v, path+"/"+e.Key) + case bson.A: + for i, el := range v { + ed, ok := el.(bson.D) + if !ok { + continue + } + name := bsonnav.DGetString(ed, "PropertyKey") + if name == "" { + name = bsonnav.DGetString(ed, "Key") + } + if name == "" { + name = "#" + strconv.Itoa(i) + } + walk(ed, path+"/"+e.Key+"["+name+"]") + } + } + } + } + walk(typ, "") + return out +} + +// invertPaths turns ID→path into path→stored ID value. +func invertPaths(byID map[string]string) map[string]string { + out := make(map[string]string, len(byID)) + for id, p := range byID { + out[p] = id + } + return out +} + +// remapTypePointers deep-copies v with every TypePointer re-aimed from the +// replacement's Type to the stored one, by path. false when one has no +// counterpart. +func remapTypePointers(v any, newPaths, storedIDs map[string]string) (any, bool) { + switch x := v.(type) { + case bson.D: + // A pluggable widget nested in the value (a column's filter) carries + // its own Type, and its pointers aim into that; they stay as built. + if bsonnav.DGetString(x, "$Type") == "CustomWidgets$CustomWidget" { + return cloneDoc(x), true + } + out := make(bson.D, 0, len(x)) + for _, e := range x { + if e.Key == "TypePointer" { + if id := idKey(e.Value); id != "" { + path, ok := newPaths[id] + if !ok { + return nil, false + } + stored, ok := storedIDs[path] + if !ok { + return nil, false + } + out = append(out, bson.E{Key: e.Key, Value: idValue(stored, e.Value)}) + continue + } + } + nv, ok := remapTypePointers(e.Value, newPaths, storedIDs) + if !ok { + return nil, false + } + out = append(out, bson.E{Key: e.Key, Value: nv}) + } + return out, true + case bson.A: + out := make(bson.A, 0, len(x)) + for _, el := range x { + nv, ok := remapTypePointers(el, newPaths, storedIDs) + if !ok { + return nil, false + } + out = append(out, nv) + } + return out, true + } + return v, true +} + +// idKey renders an element ID — binary as stored, or a string — as a map key. +func idKey(v any) string { + switch x := v.(type) { + case primitive.Binary: + return "b:" + string(x.Data) + case []byte: + return "b:" + string(x) + case string: + return "s:" + x + } + return "" +} + +// idValue turns a key from idKey back into the stored representation, in the +// shape like (the pointer it replaces). +func idValue(key string, like any) any { + switch key[:2] { + case "b:": + data := []byte(key[2:]) + if b, ok := like.(primitive.Binary); ok { + return primitive.Binary{Subtype: b.Subtype, Data: data} + } + return primitive.Binary{Data: data} + default: + return key[2:] + } +} + +// sameIgnoringIdentity compares two values with every $ID and TypePointer +// left out: two builds of the same widget mint different IDs for the same +// content. +func sameIgnoringIdentity(a, b any) bool { + ea, errA := bson.Marshal(bson.D{{Key: "v", Value: stripIdentity(a)}}) + eb, errB := bson.Marshal(bson.D{{Key: "v", Value: stripIdentity(b)}}) + return errA == nil && errB == nil && bytes.Equal(ea, eb) +} + +func stripIdentity(v any) any { + switch x := v.(type) { + case bson.D: + out := make(bson.D, 0, len(x)) + for _, e := range x { + if e.Key == "$ID" || e.Key == "TypePointer" { + continue + } + out = append(out, bson.E{Key: e.Key, Value: stripIdentity(e.Value)}) + } + return out + case bson.A: + out := make(bson.A, 0, len(x)) + for _, el := range x { + out = append(out, stripIdentity(el)) + } + return out + } + return v +} + +// cloneDoc deep-copies a document through its encoding. +func cloneDoc(d bson.D) bson.D { + raw, err := bson.Marshal(d) + if err != nil { + return d + } + var out bson.D + if err := bson.Unmarshal(raw, &out); err != nil { + return d + } + return out +} diff --git a/mdl/backend/pagemutator/pluggable_replace_test.go b/mdl/backend/pagemutator/pluggable_replace_test.go new file mode 100644 index 0000000000..6700f51f2a --- /dev/null +++ b/mdl/backend/pagemutator/pluggable_replace_test.go @@ -0,0 +1,219 @@ +// SPDX-License-Identifier: Apache-2.0 + +package pagemutator + +import ( + "testing" + + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" + + "github.com/mendixlabs/mxcli/mdl/backend/bsonnav" +) + +// mendixlabs/mxcli#1247: REPLACE of a pluggable widget by one of the same +// package keeps every stored property the statement does not change. + +func bid(s string) primitive.Binary { return primitive.Binary{Data: []byte(s)} } + +// comboDoc builds a pluggable widget the way the three sources differ: each has +// its own element IDs (prefix), and the given property values. +func comboDoc(prefix, editable string, props map[string]string) bson.D { + keys := []string{"optionsSourceAssociationDataSource", "emptyOptionText", "readOnlyStyle"} + var pts, ps bson.A + pts = append(pts, int32(2)) + ps = append(ps, int32(2)) + for _, k := range keys { + pts = append(pts, bson.D{ + {Key: "$ID", Value: bid(prefix + "pt-" + k)}, + {Key: "PropertyKey", Value: k}, + {Key: "ValueType", Value: bson.D{{Key: "$ID", Value: bid(prefix + "vt-" + k)}}}, + }) + ps = append(ps, bson.D{ + {Key: "$ID", Value: bid(prefix + "p-" + k)}, + {Key: "TypePointer", Value: bid(prefix + "pt-" + k)}, + {Key: "Value", Value: bson.D{ + {Key: "$ID", Value: bid(prefix + "v-" + k)}, + {Key: "PrimitiveValue", Value: props[k]}, + {Key: "TypePointer", Value: bid(prefix + "vt-" + k)}, + }}, + }) + } + return bson.D{ + {Key: "$ID", Value: bid(prefix + "w")}, + {Key: "$Type", Value: "CustomWidgets$CustomWidget"}, + {Key: "Editable", Value: editable}, + {Key: "Name", Value: "comboBox1"}, + {Key: "Object", Value: bson.D{{Key: "$ID", Value: bid(prefix + "o")}, {Key: "Properties", Value: ps}}}, + {Key: "Type", Value: bson.D{ + {Key: "$ID", Value: bid(prefix + "t")}, + {Key: "ObjectType", Value: bson.D{{Key: "$ID", Value: bid(prefix + "ot")}, {Key: "PropertyTypes", Value: pts}}}, + {Key: "WidgetId", Value: "com.mendix.widget.web.combobox.Combobox"}, + }}, + } +} + +func propValue(t *testing.T, w bson.D, key string) (string, bson.D) { + t.Helper() + p := propertiesByKey(w)[key] + if p == nil { + t.Fatalf("no property %s", key) + } + v := bsonnav.DGetDoc(p, "Value") + return bsonnav.DGetString(v, "PrimitiveValue"), v +} + +func TestMergeUnstatedPluggable(t *testing.T) { + stored := comboDoc("S", "Never", map[string]string{ + "optionsSourceAssociationDataSource": "unsorted", "emptyOptionText": "Country", "readOnlyStyle": "text"}) + // Both builds come from the template, so MDL-unmapped properties carry the + // template's values; only the data source differs — the statement's change. + baseline := comboDoc("B", "Always", map[string]string{ + "optionsSourceAssociationDataSource": "unsorted", "emptyOptionText": "", "readOnlyStyle": "bordered"}) + replacement := comboDoc("N", "Always", map[string]string{ + "optionsSourceAssociationDataSource": "sorted", "emptyOptionText": "", "readOnlyStyle": "bordered"}) + + got, ok := mergeUnstatedPluggable(stored, replacement, baseline) + if !ok { + t.Fatal("merge did not apply") + } + if v, _ := propValue(t, got, "emptyOptionText"); v != "Country" { + t.Errorf("emptyOptionText = %q, want the stored \"Country\"", v) + } + if v, _ := propValue(t, got, "readOnlyStyle"); v != "text" { + t.Errorf("readOnlyStyle = %q, want the stored \"text\"", v) + } + if e := bsonnav.DGetString(got, "Editable"); e != "Never" { + t.Errorf("Editable = %q, want the stored \"Never\"", e) + } + v, doc := propValue(t, got, "optionsSourceAssociationDataSource") + if v != "sorted" { + t.Errorf("data source = %q, want the statement's \"sorted\"", v) + } + // The grafted value points into the STORED Type, which is the one kept. + if tp := idKey(bsonnav.DGet(doc, "TypePointer")); tp != idKey(bid("Svt-optionsSourceAssociationDataSource")) { + t.Errorf("grafted value's TypePointer = %q, want the stored value type", tp) + } + if pluggableWidgetID(got) == "" || idKey(bsonnav.DGet(bsonnav.DGetDoc(got, "Type"), "$ID")) != idKey(bid("St")) { + t.Error("the stored Type was not kept") + } + // The stored document itself is not modified in place. + if v, _ := propValue(t, stored, "optionsSourceAssociationDataSource"); v != "unsorted" { + t.Error("merge modified the stored document it was given") + } + + // A stated change of a widget-level field is applied. + renamed := comboDoc("N", "Always", map[string]string{ + "optionsSourceAssociationDataSource": "unsorted", "emptyOptionText": "", "readOnlyStyle": "bordered"}) + bsonnav.DSet(renamed, "Name", "cbCountry") + got, ok = mergeUnstatedPluggable(stored, renamed, baseline) + if !ok || bsonnav.DGetString(got, "Name") != "cbCountry" { + t.Errorf("a renaming replace was not applied: ok=%v name=%q", ok, bsonnav.DGetString(got, "Name")) + } + + // Control: a different widget package is not merged. + other := comboDoc("N", "Always", nil) + bsonnav.DSet(bsonnav.DGetDoc(other, "Type"), "WidgetId", "com.mendix.widget.web.datagrid.Datagrid") + if _, ok := mergeUnstatedPluggable(stored, other, baseline); ok { + t.Error("merged a widget of another package") + } +} + +// A value that carries a pluggable widget of its own — a DataGrid 2 column's +// filter — is grafted with that widget's pointers left as built: they aim into +// the nested widget's own Type, not the outer one. Re-aiming them failed, and +// the whole REPLACE fell back to the template rebuild. +func TestRemapTypePointersLeavesANestedPluggableWidget(t *testing.T) { + nested := bson.D{ + {Key: "$ID", Value: bid("f")}, + {Key: "$Type", Value: "CustomWidgets$CustomWidget"}, + {Key: "Object", Value: bson.D{{Key: "Properties", Value: bson.A{int32(2), + bson.D{{Key: "TypePointer", Value: bid("f-pt")}}}}}}, + } + value := bson.D{ + {Key: "TypePointer", Value: bid("Nvt")}, + {Key: "Widgets", Value: bson.A{int32(2), nested}}, + } + newPaths := map[string]string{idKey(bid("Nvt")): "/vt"} + storedIDs := map[string]string{"/vt": idKey(bid("Svt"))} + got, ok := remapTypePointers(value, newPaths, storedIDs) + if !ok { + t.Fatal("a nested pluggable widget's own pointers made the graft fail") + } + d := got.(bson.D) + if idKey(bsonnav.DGet(d, "TypePointer")) != idKey(bid("Svt")) { + t.Error("the value's own pointer was not re-aimed at the stored Type") + } + w := bsonnav.DGetArrayElements(bsonnav.DGet(d, "Widgets"))[0].(bson.D) + p := bsonnav.DGetArrayElements(bsonnav.DGet(bsonnav.DGetDoc(w, "Object"), "Properties"))[0].(bson.D) + if idKey(bsonnav.DGet(p, "TypePointer")) != idKey(bid("f-pt")) { + t.Error("the nested widget's pointer into its own Type was changed") + } +} + +// gridDoc is a pluggable widget with one object-list property, "columns", +// whose objects have "header" and "hidden" properties. +func gridDoc(prefix string, headers, hidden []string) bson.D { + colType := bson.D{{Key: "$ID", Value: bid(prefix + "cot")}, {Key: "PropertyTypes", Value: bson.A{int32(2), + bson.D{{Key: "$ID", Value: bid(prefix + "pt-header")}, {Key: "PropertyKey", Value: "header"}}, + bson.D{{Key: "$ID", Value: bid(prefix + "pt-hidden")}, {Key: "PropertyKey", Value: "hidden"}}, + }}} + objs := bson.A{int32(2)} + for i := range headers { + objs = append(objs, bson.D{ + {Key: "$ID", Value: bid(prefix + "col" + headers[i])}, + {Key: "Properties", Value: bson.A{int32(2), + bson.D{{Key: "TypePointer", Value: bid(prefix + "pt-header")}, + {Key: "Value", Value: bson.D{{Key: "PrimitiveValue", Value: headers[i]}}}}, + bson.D{{Key: "TypePointer", Value: bid(prefix + "pt-hidden")}, + {Key: "Value", Value: bson.D{{Key: "PrimitiveValue", Value: hidden[i]}}}}, + }}, + }) + } + return bson.D{ + {Key: "$ID", Value: bid(prefix + "w")}, + {Key: "$Type", Value: "CustomWidgets$CustomWidget"}, + {Key: "Object", Value: bson.D{{Key: "Properties", Value: bson.A{int32(2), + bson.D{{Key: "TypePointer", Value: bid(prefix + "pt-columns")}, + {Key: "Value", Value: bson.D{{Key: "Objects", Value: objs}, {Key: "PrimitiveValue", Value: ""}}}}, + }}}}, + {Key: "Type", Value: bson.D{ + {Key: "ObjectType", Value: bson.D{{Key: "PropertyTypes", Value: bson.A{int32(2), + bson.D{{Key: "$ID", Value: bid(prefix + "pt-columns")}, {Key: "PropertyKey", Value: "columns"}, + {Key: "ValueType", Value: bson.D{{Key: "ObjectType", Value: colType}}}}, + }}}}, + {Key: "WidgetId", Value: "com.mendix.widget.web.datagrid.Datagrid"}, + }}, + } +} + +// Changing one column's header keeps every column's unstated properties: the +// object list is merged object by object, not grafted whole. +func TestMergeUnstatedPluggableMergesAnObjectList(t *testing.T) { + stored := gridDoc("S", []string{"Name", "Age"}, []string{"yes", "yes"}) + baseline := gridDoc("B", []string{"Name", "Age"}, []string{"no", "no"}) + replacement := gridDoc("N", []string{"Rule name", "Age"}, []string{"no", "no"}) + got, ok := mergeUnstatedPluggable(stored, replacement, baseline) + if !ok { + t.Fatal("merge did not apply") + } + col := bsonnav.DGetArrayElements(bsonnav.DGet(bsonnav.DGetDoc(bsonnav.DGetArrayElements( + bsonnav.DGet(bsonnav.DGetDoc(got, "Object"), "Properties"))[0].(bson.D), "Value"), "Objects")) + if len(col) != 2 { + t.Fatalf("%d columns, want 2", len(col)) + } + for i, want := range []string{"Rule name", "Age"} { + props := bsonnav.DGetArrayElements(bsonnav.DGet(col[i].(bson.D), "Properties")) + h := bsonnav.DGetString(bsonnav.DGetDoc(props[0].(bson.D), "Value"), "PrimitiveValue") + hid := bsonnav.DGetString(bsonnav.DGetDoc(props[1].(bson.D), "Value"), "PrimitiveValue") + if h != want { + t.Errorf("column %d header = %q, want %q", i, h, want) + } + if hid != "yes" { + t.Errorf("column %d hidden = %q, want the stored \"yes\"", i, hid) + } + if idKey(bsonnav.DGet(col[i].(bson.D), "$ID")) != idKey(bid("Scol"+[]string{"Name", "Age"}[i])) { + t.Errorf("column %d is not the stored column", i) + } + } +} diff --git a/mdl/backend/pagemutator/probe.go b/mdl/backend/pagemutator/probe.go index 08c62d95b1..9575ffc9e6 100644 --- a/mdl/backend/pagemutator/probe.go +++ b/mdl/backend/pagemutator/probe.go @@ -59,6 +59,9 @@ func (m *Mutator) ResolvesTarget(widgetRef, columnRef string) bool { return true // page-level SET addresses the document itself } if columnRef != "" { + if _, ok := m.findDataViewFooter(widgetRef, columnRef); ok { + return true + } _, err := findBsonColumn(m.rawData, widgetRef, columnRef, m.widgetFinder) return err == nil } diff --git a/mdl/deprecation/deprecation.go b/mdl/deprecation/deprecation.go index 69cf53731d..418115a004 100644 --- a/mdl/deprecation/deprecation.go +++ b/mdl/deprecation/deprecation.go @@ -97,7 +97,8 @@ const ( AggregateFunctionForm = "MDL-DEPR004" // UnstoredWidgetName is a name written on a page element Mendix stores no // name for: a layout grid's rows and columns, a data grid's columns and - // control bar, a gallery's template and filter (R12, ako/mxcli#749). + // control bar, a gallery's template and filter (R12, ako/mxcli#749), a + // data view's footer (ako/mxcli#528). UnstoredWidgetName = "MDL-DEPR005" // R10: document types named as Studio Pro names them (ako/mxcli#755). The @@ -419,8 +420,8 @@ var entries = []Entry{ Rewrite: Rewrite{Structural: "name out of the element: `row row1 {` becomes `row {`"}, RemovedIn: 2, Note: "Mendix stores no name on a layout grid's row, a row's column, a data grid's column or control bar, " + - "or a gallery's template or filter, so the name was never written and describe no longer invents one. " + - "A data grid column is addressed as `grid column(Attr)` or `grid column('Caption')`.", + "a gallery's template or filter, or a data view's footer, so the name was never written and describe no longer invents one. " + + "A data grid column is addressed as `grid column(Attr)` or `grid column('Caption')`, a data view's footer as `dv.footer`.", Example: "create page M.P (Title: 'P', Layout: Atlas_Core.Atlas_Default) { datagrid dg (DataSource: database from M.E) { column Name (Attribute: Name) } };", CanonicalExample: "create page M.P (Title: 'P', Layout: Atlas_Core.Atlas_Default) { datagrid dg (DataSource: database from M.E) { column (Attribute: Name) } };", }, diff --git a/mdl/executor/check_exec_agreement_pedapp_test.go b/mdl/executor/check_exec_agreement_pedapp_test.go new file mode 100644 index 0000000000..30121e894e --- /dev/null +++ b/mdl/executor/check_exec_agreement_pedapp_test.go @@ -0,0 +1,275 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// check -p and exec must not disagree (ako/mxcli#906, mendixlabs/mxcli#1210 – +// #1213, ako/mxcli#558, #563). Each case below passed `check -p` and was then +// refused by exec — after the statements before it had been written — or was +// accepted by both and broke the project. Every test pairs the case with a +// control that both accept, on the Studio Pro-authored PedApp fixture. + +// agreeCheck is what `mxcli check -p` reports as an error: the no-project rules, +// the reference pass, and the project conflicts (cmd/mxcli/cmd_check.go). +func agreeCheck(t *testing.T, exec *Executor, dir, src string) []string { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse: %v\n%s", errs[0], src) + } + var out []string + for _, v := range ValidateProgram(prog, filepath.Join(dir, "PedApp.mpr")) { + if v.Severity == linter.SeverityError { + out = append(out, v.RuleID+": "+v.Message) + } + } + refErrs, _ := exec.ValidateProgramWithWarnings(prog) + refErrs = append(refErrs, exec.CheckProjectConflicts(prog)...) + for _, e := range refErrs { + out = append(out, e.Error()) + } + return out +} + +// agreeExec runs the script's statements as `exec --no-check` does, so what is +// asserted is exec's own verdict, not the pre-flight it shares with check. +func agreeExec(t *testing.T, exec *Executor, src string) error { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse: %v\n%s", errs[0], src) + } + return exec.ExecuteProgram(prog) +} + +// assertAgree runs check, then exec, and asserts both refuse (want != "", +// matched in both outputs) or both accept. +func assertAgree(t *testing.T, exec *Executor, dir, src, want string) { + t.Helper() + checkErrs := agreeCheck(t, exec, dir, src) + execErr := agreeExec(t, exec, src) + checkSays := strings.Join(checkErrs, "\n") + if want == "" { + if len(checkErrs) != 0 || execErr != nil { + t.Errorf("control: check and exec must both accept\ncheck: %s\nexec: %v", checkSays, execErr) + } + return + } + if !strings.Contains(checkSays, want) { + t.Errorf("check -p must predict the refusal %q, reported:\n%s", want, checkSays) + } + if execErr == nil || !strings.Contains(execErr.Error(), want) { + t.Errorf("exec must refuse with %q, got: %v", want, execErr) + } +} + +// ako/mxcli#906: an existing demo user. The control is `create or modify`. +func TestCheckExecAgree_DemoUserExists(t *testing.T) { + exec, out, dir := openPedAppCopy(t) + const create = "mdl 1;\ncreate demo user 'agree_demo' ( Password: 'Agree!23456789', UserRoles: (User) );" + if err := agreeExec(t, exec, create); err != nil { + t.Fatalf("setup: %v\n%s", err, out.String()) + } + assertAgree(t, exec, dir, create, "demo user already exists") + assertAgree(t, exec, dir, + "mdl 1;\ncreate or modify demo user 'agree_demo' ( Password: 'Agree!23456789', UserRoles: (User) );", "") +} + +// ako/mxcli#906: an existing jar dependency. The control changes its version, +// which exec does through the same applyJarDepAction. +func TestCheckExecAgree_JarDependencyExists(t *testing.T) { + exec, out, dir := openPedAppCopy(t) + const add = `mdl 1; +alter module MyFirstModule + add jar dependency ( group = 'org.example', artifact = 'agree-lib', version = '1.0.0', included = true );` + if err := agreeExec(t, exec, add); err != nil { + t.Fatalf("setup: %v\n%s", err, out.String()) + } + assertAgree(t, exec, dir, add, "jar dependency 'org.example:agree-lib' already exists") + assertAgree(t, exec, dir, + "mdl 1;\nalter module MyFirstModule set jar dependency 'org.example:agree-lib' version '1.0.1';", "") + // The replay follows the script: an add after a drop of the same coordinate + // is fine, and check must not flag it from the stored state. + assertAgree(t, exec, dir, `mdl 1; +alter module MyFirstModule drop jar dependency 'org.example:agree-lib'; +alter module MyFirstModule + add jar dependency ( group = 'org.example', artifact = 'agree-lib', version = '1.0.2', included = true );`, "") +} + +// ako/mxcli#906: translations into the project's source language (PedApp's is +// en_US). The control is another language. +func TestCheckExecAgree_SourceLanguageTranslations(t *testing.T) { + exec, _, dir := openPedAppCopy(t) + assertAgree(t, exec, dir, "mdl 1;\ncreate or modify translations in MyFirstModule for en_US (\n 'Hello' as 'Hi'\n);", + "en_US is the project's source language") + assertAgree(t, exec, dir, "mdl 1;\ncreate or modify translations in MyFirstModule for nl_NL (\n 'Hello' as 'Hallo'\n);", "") +} + +// mendixlabs/mxcli#1211 / ako/mxcli#558: a task queue the script creates +// before the call that names it was "not found" by check and by exec's +// pre-flight. The other half: the same queue created AFTER the call (as +// `create or modify`, which MDL-ORDER01 does not judge) is refused by both. +func TestCheckExecAgree_TaskQueueCreatedInScript(t *testing.T) { + exec, _, dir := openPedAppCopy(t) + // The reporter's script, headerless (mdl 0) as filed. + assertAgree(t, exec, dir, `create module "ModQ"; +create or modify queue "ModQ"."TQ_Work" (Parallelism: 1); +create microflow "ModQ"."SUB_Work" () +begin + log info node 'Repro' 'work'; +end; +/ +create microflow "ModQ"."ACT_Enqueue" () +begin + call microflow "ModQ"."SUB_Work"() in queue "ModQ"."TQ_Work"; +end; +/`, "") + + exec2, _, dir2 := openPedAppCopy(t) + assertAgree(t, exec2, dir2, `mdl 1; +create module ModQ2; +create microflow ModQ2.SUB_Work () +begin + log info node 'Repro' 'work'; +end; +create microflow ModQ2.ACT_Enqueue () +begin + call microflow ModQ2.SUB_Work() in queue ModQ2.TQ_Work; +end; +create or modify task queue ModQ2.TQ_Work (Parallelism: 1);`, "ModQ2.TQ_Work") +} + +// mendixlabs/mxcli#1212: a page whose button calls a microflow the script +// creates further down. Control: the microflow first — its `show page` of the +// later page resolves lazily, so that order works. +func TestCheckExecAgree_PageCallsLaterMicroflow(t *testing.T) { + const page = `create page ModO.ItemPage ( + Title: 'Item', + Layout: Atlas_Core.PopupLayout, + Params: ( $Item: ModO.Item ) +) { + dataview dvItem (DataSource: $Item) { + textbox txtName (Attribute: Name) + actionbutton btnReopen (Caption: 'Reopen', Action: call microflow ModO.ACT_OpenItem(Item = $Item)) + } +}; +` + const flow = `create microflow ModO.ACT_OpenItem ($Item: ModO.Item) +begin + show page ModO.ItemPage(Item = $Item); +end; +` + const head = "mdl 1;\ncreate module ModO;\ncreate persistent entity ModO.Item (Name: String(100));\n" + + exec, _, dir := openPedAppCopy(t) + assertAgree(t, exec, dir, head+page+flow, "ModO.ACT_OpenItem") + + exec2, _, dir2 := openPedAppCopy(t) + assertAgree(t, exec2, dir2, head+flow+page, "") + + // The later definition as `create or modify`: no project-free rule can + // judge it, the project tier must. + exec3, _, dir3 := openPedAppCopy(t) + assertAgree(t, exec3, dir3, head+page+strings.Replace(flow, "create microflow", "create or modify microflow", 1), + "ModO.ACT_OpenItem") +} + +// mendixlabs/mxcli#1210: a variable passed to a Microflow-typed Java action +// parameter was written as the reference's text and the project no longer +// loaded. Control: the literal name. +func TestCheckExecAgree_MicroflowTypedJavaActionParameter(t *testing.T) { + const head = `mdl 1; +create module ModA; +create java action ModA.RunNamed (Target: Microflow not null) returns Boolean as $$ +return true; +$$; +create microflow ModA.SUB_Target () returns Boolean as $Ok +begin + declare $Ok Boolean = true; + return $Ok; +end; +` + exec, _, dir := openPedAppCopy(t) + assertAgree(t, exec, dir, head+`create microflow ModA.SUB_Variable ($Name: String) returns Boolean as $R +begin + $R = call java action ModA.RunNamed(Target = $Name); + return $R; +end;`, "parameter Target is of type Microflow") + + exec2, _, dir2 := openPedAppCopy(t) + assertAgree(t, exec2, dir2, head+`create microflow ModA.SUB_Literal () returns Boolean as $R +begin + $R = call java action ModA.RunNamed(Target = 'ModA.SUB_Target'); + return $R; +end;`, "") +} + +// ako/mxcli#563: `referenceselector` parses, so check passed it, and the page +// builder has no writer for it. Control: the same page with a text box. +func TestCheckExecAgree_ReferenceSelector(t *testing.T) { + const head = `mdl 1; +create module ModR; +create persistent entity ModR.Sys (Name: String(100)); +create persistent entity ModR.Req (Title: String(100)); +create association ModR.Req_Sys from ModR.Req to ModR.Sys type Reference; +create page ModR.ReqPage ( + Title: 'Req', + Layout: Atlas_Core.PopupLayout, + Params: ( $Req: ModR.Req ) +) { + dataview dvReq (DataSource: $Req) { + %s + } +};` + exec, _, dir := openPedAppCopy(t) + assertAgree(t, exec, dir, strings.Replace(head, "%s", + "referenceselector rsSystem (Label: 'System', Association: ModR.Req_Sys)", 1), + "built-in Forms widget Forms$ReferenceSelector") + + exec2, _, dir2 := openPedAppCopy(t) + assertAgree(t, exec2, dir2, strings.Replace(head, "%s", "textbox txtTitle (Attribute: Title)", 1), "") +} + +// mendixlabs/mxcli#1213: retrieve constraints mxbuild rejects with CE0161 — +// measured on 11.13.0, each against the control beside it, which builds clean: +// +// startsWith(Title, 'X') starts-with(Title, 'X') +// ModX.Order_Customer != empty ModX.Order_Customer/ModX.Customer +// [NoSuchAttr = 'x'] [Title = 'x'] +// [CreatedDate > …] [createdDate > …] +// +// MDL047/MDL091 are exec-enforced rules, so exec refuses those two itself; the +// members are resolved by the reference pass exec runs as its pre-flight. +func TestCheckExecAgree_RetrieveConstraint(t *testing.T) { + const head = `mdl 1; +create module ModX; +create persistent entity ModX.Order (Title: String(100), CreatedDate: AutoCreatedDate); +create persistent entity ModX.Customer (Name: String(100)); +create association ModX.Order_Customer from ModX.Order to ModX.Customer type Reference; +create microflow ModX.SUB_Find () +begin + retrieve $A from ModX.Order where %s; +end;` + cases := []struct{ bad, want, good string }{ + {"startsWith(Title, 'X')", "MDL091", "starts-with(Title, 'X')"}, + {"ModX.Order_Customer != empty", "MDL047", "ModX.Order_Customer/ModX.Customer"}, + {"[NoSuchAttr = 'x']", `names "NoSuchAttr"`, "[Title = 'x']"}, + {"[CreatedDate > '[%CurrentDateTime%]']", "spelled `createdDate`", "[createdDate > '[%CurrentDateTime%]']"}, + } + for _, c := range cases { + exec, _, dir := openPedAppCopy(t) + bad := strings.Replace(head, "%s", c.bad, 1) + if got := strings.Join(agreeCheck(t, exec, dir, bad), "\n"); !strings.Contains(got, c.want) { + t.Errorf("%s: check -p must report %q, reported:\n%s", c.bad, c.want, got) + } + assertAgree(t, exec, dir, strings.Replace(head, "%s", c.good, 1), "") + } +} diff --git a/mdl/executor/cmd_alter_page.go b/mdl/executor/cmd_alter_page.go index 642242a6ed..9f6ae2e9cd 100644 --- a/mdl/executor/cmd_alter_page.go +++ b/mdl/executor/cmd_alter_page.go @@ -56,6 +56,18 @@ func execAlterPage(ctx *ExecContext, s *ast.AlterPageStmt) error { // Resolve module name for building new widgets modName := h.GetModuleName(containerID) + // The stored document as describe prints it, parsed back — read once, and + // only when a REPLACE needs a pluggable widget's baseline (#1247). + var described map[string]*ast.WidgetV3 + describedOnce := false + storedWidgets := func() map[string]*ast.WidgetV3 { + if !describedOnce { + describedOnce = true + described = describedStoredWidgets(ctx, unitID, containerType, s.PageName) + } + return described + } + for _, op := range s.Operations { // Every target resolves through the document type's resolver before the // operation runs (ADR-0012): what an address means is answered once, @@ -82,7 +94,7 @@ func execAlterPage(ctx *ExecContext, s *ast.AlterPageStmt) error { return mdlerrors.NewBackend("drop TEMPLATE", err) } case *ast.ReplaceWidgetOp: - if err := applyReplaceWidgetMutator(ctx, mutator, o, modName, containerID); err != nil { + if err := applyReplaceWidgetMutator(ctx, mutator, o, modName, containerID, storedWidgets); err != nil { return mdlerrors.NewBackend("replace", err) } case *ast.AddVariableOp: @@ -472,18 +484,24 @@ func applyDropWidgetMutator(mutator backend.PageMutator, op *ast.DropWidgetOp) e // REPLACE widget via mutator // ============================================================================ -func applyReplaceWidgetMutator(ctx *ExecContext, mutator backend.PageMutator, op *ast.ReplaceWidgetOp, moduleName string, moduleID model.ID) error { +func applyReplaceWidgetMutator(ctx *ExecContext, mutator backend.PageMutator, op *ast.ReplaceWidgetOp, moduleName string, moduleID model.ID, storedWidgets func() map[string]*ast.WidgetV3) error { newWidgets, err := expandAlterFragments(ctx, op.NewWidgets, moduleName, moduleID) if err != nil { return err } expanded := *op - expanded.NewWidgets = newWidgets + expanded.NewWidgets = unwrapFooterReplacement(op.Target, newWidgets) op = &expanded + footerRegion := isFooterRegionTarget(op.Target) + + // The widgets the replace removes are not duplicates of what replaces them: + // restating a footer's or a container's own children under their names is + // the ordinary edit, and was refused as "duplicate widget name" (#293). + replaced := replacedWidgetNames(mutator, op.Target) // Check for duplicate widget names (skip the widget being replaced) for _, w := range op.NewWidgets { - if w.Name != "" && w.Name != op.Target.Widget && w.Name != columnRefOf(op.Target) && mutator.FindWidget(w.Name) { + if w.Name != "" && w.Name != op.Target.Widget && w.Name != columnRefOf(op.Target) && !replaced[w.Name] && mutator.FindWidget(w.Name) { return mdlerrors.NewAlreadyExistsMsg("widget", w.Name, fmt.Sprintf("duplicate widget name '%s': a widget with this name already exists on the page", w.Name)) } } @@ -500,19 +518,139 @@ func applyReplaceWidgetMutator(ctx *ExecContext, mutator backend.PageMutator, op return mutator.ReplaceColumn(op.Target.Widget, columnRefOf(op.Target), specs) } - // Find entity context from enclosing DataView/DataGrid/ListView for regular widget replace. - entityCtx, _ := alterEntityContext(ctx, mutator, op.Target.Widget, false, moduleName, moduleID) + // Find entity context from enclosing DataView/DataGrid/ListView for regular + // widget replace. A data view footer's widgets sit INSIDE the data view, so + // they take its own context, as INSERT INTO does. + entityCtx, _ := alterEntityContext(ctx, mutator, op.Target.Widget, footerRegion, moduleName, moduleID) - // Build new widgets from AST, excluding the target widget/column from the - // duplicate-name scope so a same-name replacement is allowed. - widgets, err := buildWidgetsFromAST(ctx, op.NewWidgets, moduleName, moduleID, entityCtx, mutator, op.Target.Widget, columnRefOf(op.Target)) + // Build new widgets from AST, excluding the target widget/column — and what + // it contains — from the duplicate-name scope so a same-name replacement is + // allowed. + exclude := []string{op.Target.Widget, columnRefOf(op.Target)} + for n := range replaced { + exclude = append(exclude, n) + } + widgets, err := buildWidgetsFromAST(ctx, op.NewWidgets, moduleName, moduleID, entityCtx, mutator, exclude...) if err != nil { return mdlerrors.NewBackend("build replacement widgets", err) } + // One pluggable widget replaced by one of the same package keeps what the + // statement does not state (mendixlabs/mxcli#1247): the stored widget, as + // describe prints it, is built beside the replacement, and the mutator keeps + // every stored property the two builds agree on. + if handled, err := replacePluggableKeepingUnstated(ctx, mutator, op, widgets, storedWidgets, moduleName, moduleID, entityCtx, exclude); handled || err != nil { + return err + } + return mutator.ReplaceWidget(op.Target.Widget, columnRefOf(op.Target), widgets) } +// pluggableKeepingReplacer is the PageMutator half of +// replacePluggableKeepingUnstated; mutators without it replace as before. +type pluggableKeepingReplacer interface { + ReplacePluggableKeepingUnstated(widgetRef string, replacement, baseline pages.Widget) (bool, error) +} + +// replacePluggableKeepingUnstated handles `replace with { }`. handled is false when it does not apply — not exactly one +// widget for one, no description of the stored one, or a baseline that does +// not build — and the caller replaces as before. +func replacePluggableKeepingUnstated(ctx *ExecContext, mutator backend.PageMutator, op *ast.ReplaceWidgetOp, + widgets []pages.Widget, storedWidgets func() map[string]*ast.WidgetV3, + moduleName string, moduleID model.ID, entityCtx string, exclude []string) (bool, error) { + keeper, ok := mutator.(pluggableKeepingReplacer) + if !ok || storedWidgets == nil || op.Target.Column != "" || op.Target.IsColumnAddress() || + len(widgets) != 1 || len(op.NewWidgets) != 1 { + return false, nil + } + if _, isPluggable := widgets[0].(*pages.CustomWidget); !isPluggable { + return false, nil + } + stored := storedWidgets()[op.Target.Widget] + if stored == nil || !strings.EqualFold(stored.Type, op.NewWidgets[0].Type) { + return false, nil + } + baseline, err := buildWidgetsFromAST(ctx, cloneWidgets([]*ast.WidgetV3{stored}), moduleName, moduleID, entityCtx, mutator, exclude...) + if err != nil || len(baseline) != 1 { + return false, nil + } + return keeper.ReplacePluggableKeepingUnstated(op.Target.Widget, widgets[0], baseline[0]) +} + +// describedStoredWidgets describes the document an ALTER edits, in the +// script's language, and indexes its widgets by name; nil when it cannot. +func describedStoredWidgets(ctx *ExecContext, unitID model.ID, containerType string, name ast.QualifiedName) map[string]*ast.WidgetV3 { + var describe func() error + switch containerType { + case "page": + describe = func() error { return describePage(ctx, name) } + case "snippet": + describe = func() error { return describeSnippet(ctx, name) } + default: + return nil + } + out, err := describedWidgets(ctx, func() error { + prev := ctx.describeID + ctx.describeID = unitID + defer func() { ctx.describeID = prev }() + return describe() + }) + if err != nil { + return nil + } + return out +} + +// storedPageVariables is the stored document's page variables, so a widget an +// ALTER adds can bind to one (`Attribute: $ShowAll`, mendixlabs/mxcli#1235) and +// a `$name` in a template resolves as it does in CREATE. +func storedPageVariables(mutator backend.PageMutator) map[string]bool { + vars := map[string]bool{} + if lister, ok := mutator.(interface{ PageVariableNames() []string }); ok { + for _, n := range lister.PageVariableNames() { + vars[n] = true + } + } + return vars +} + +// isFooterRegionTarget reports whether an ALTER target is `.footer` — +// a data view's footer region (ako/mxcli#528). Whether the owner IS a data view +// is the mutator's call; on anything else the dotted form keeps its other +// meanings (a scroll-container region, a grid column). +func isFooterRegionTarget(t ast.WidgetRef) bool { + return t.Widget != "" && !t.IsColumnAddress() && strings.EqualFold(t.Column, "footer") +} + +// unwrapFooterReplacement lets `replace dv.footer with { footer { … } }` — the +// footer block as describe prints it — mean its content: a footer region holds +// widgets, and a `footer` built outside a data view would be a container nested +// inside the region rather than the region's content. +func unwrapFooterReplacement(target ast.WidgetRef, widgets []*ast.WidgetV3) []*ast.WidgetV3 { + if !isFooterRegionTarget(target) || len(widgets) != 1 || !strings.EqualFold(widgets[0].Type, "footer") { + return widgets + } + return widgets[0].Children +} + +// replacedWidgetNames lists the names a REPLACE removes along with its target: +// everything inside it. Only mutators that can walk the stored tree answer; the +// others keep the old, stricter scope. +func replacedWidgetNames(mutator backend.PageMutator, target ast.WidgetRef) map[string]bool { + walker, ok := mutator.(interface { + ContainedWidgetNames(widgetRef, columnRef string) []string + }) + if !ok { + return nil + } + names := map[string]bool{} + for _, n := range walker.ContainedWidgetNames(target.Widget, columnRefOf(target)) { + names[n] = true + } + return names +} + // allColumns returns true if all widgets in the slice have type "column". // Used to dispatch ALTER PAGE INSERT/REPLACE into a DataGrid2 column to the // column-specific mutator path. @@ -620,6 +758,7 @@ func buildColumnSpecsFromAST(ctx *ExecContext, widgets []*ast.WidgetV3, moduleNa fragments: ctx.Fragments, themeRegistry: ctx.GetThemeRegistry(), widgetBackend: ctx.Backend, + localVariables: storedPageVariables(mutator), } var result []*backend.DataGridColumnSpec @@ -713,6 +852,7 @@ func buildWidgetsFromAST(ctx *ExecContext, widgets []*ast.WidgetV3, moduleName s fragments: ctx.Fragments, themeRegistry: ctx.GetThemeRegistry(), widgetBackend: ctx.Backend, + localVariables: storedPageVariables(mutator), } var result []pages.Widget diff --git a/mdl/executor/cmd_alter_page_mock_test.go b/mdl/executor/cmd_alter_page_mock_test.go index 497ea98dee..a85aaeb91b 100644 --- a/mdl/executor/cmd_alter_page_mock_test.go +++ b/mdl/executor/cmd_alter_page_mock_test.go @@ -436,7 +436,7 @@ func TestApplyReplaceWidgetMutator_SameNameAllowed(t *testing.T) { } ctx, _ := newMockCtx(t, withBackend(mb)) - err := applyReplaceWidgetMutator(ctx, mutator, op, "MyModule", model.ID("mod-id")) + err := applyReplaceWidgetMutator(ctx, mutator, op, "MyModule", model.ID("mod-id"), nil) if err != nil { t.Errorf("same-name replacement should be allowed, got: %v", err) } diff --git a/mdl/executor/cmd_associations_storage_test.go b/mdl/executor/cmd_associations_storage_test.go index 0b8824a093..86fb559d0b 100644 --- a/mdl/executor/cmd_associations_storage_test.go +++ b/mdl/executor/cmd_associations_storage_test.go @@ -62,83 +62,3 @@ func TestDescribeAssociationStorageSurvivesReplay(t *testing.T) { }) } } - -// `mxcli diff` must show a storage change, and must not show one for a statement -// that leaves storage unstated (exec carries the stored value in that case). -func TestDiffAssociationComparesStorage(t *testing.T) { - base := func(s ast.StorageType) *ast.CreateAssociationStmt { - return &ast.CreateAssociationStmt{ - Name: ast.QualifiedName{Module: "M", Name: "Child_Parent"}, - Parent: ast.QualifiedName{Module: "M", Name: "Child"}, - Child: ast.QualifiedName{Module: "M", Name: "Parent"}, - Type: ast.AssocReference, - DeleteBehavior: ast.DeleteCascade, - Storage: s, - } - } - - ctx, assoc := assocFixture(t) - assoc.StorageFormat = domainmodel.StorageFormatTable - - r, err := diffAssociation(ctx, base(ast.StorageColumn)) - assertNoError(t, err) - if r.Proposed == r.Current { - t.Errorf("Table → Column reported as no change\n--- current ---\n%s\n--- proposed ---\n%s", r.Current, r.Proposed) - } - - for _, s := range []ast.StorageType{ast.StorageDefault, ast.StorageTable} { - r, err = diffAssociation(ctx, base(s)) - assertNoError(t, err) - if r.Proposed != r.Current { - t.Errorf("storage clause %d against stored Table reported as a change\n--- current ---\n%s\n--- proposed ---\n%s", - s, r.Current, r.Proposed) - } - } -} - -// A cross-module association lives in dm.CrossAssociations. The differ looked -// only in dm.Associations, so it reported every existing cross-module -// association as new — a storage flip on one was shown as a creation, never as -// the Table → Column change exec would make. -func TestDiffCrossModuleAssociationComparesStorage(t *testing.T) { - ctx, assoc := assocFixture(t) - dm, err := ctx.Backend.GetDomainModel("") - assertNoError(t, err) - dm.Associations = nil - dm.CrossAssociations = []*domainmodel.CrossModuleAssociation{{ - Name: "Child_Remote", - ParentID: assoc.ParentID, - ChildRef: "Other.Remote", - Type: domainmodel.AssociationTypeReference, - Owner: domainmodel.AssociationOwnerDefault, - StorageFormat: domainmodel.StorageFormatTable, - ChildDeleteBehavior: &domainmodel.DeleteBehavior{Type: domainmodel.DeleteBehaviorTypeDeleteMeAndReferences}, - }} - stmt := func(s ast.StorageType) *ast.CreateAssociationStmt { - return &ast.CreateAssociationStmt{ - Name: ast.QualifiedName{Module: "M", Name: "Child_Remote"}, - Parent: ast.QualifiedName{Module: "M", Name: "Child"}, - Child: ast.QualifiedName{Module: "Other", Name: "Remote"}, - Type: ast.AssocReference, - DeleteBehavior: ast.DeleteCascade, - Storage: s, - } - } - - r, err := diffAssociation(ctx, stmt(ast.StorageColumn)) - assertNoError(t, err) - if r.IsNew { - t.Fatalf("existing cross-module association reported as new\n%s", r.Proposed) - } - if r.Proposed == r.Current { - t.Errorf("cross-module Table → Column reported as no change\n%s", r.Current) - } - for _, s := range []ast.StorageType{ast.StorageDefault, ast.StorageTable} { - r, err = diffAssociation(ctx, stmt(s)) - assertNoError(t, err) - if r.Proposed != r.Current { - t.Errorf("cross-module storage %d against stored Table reported as a change\n--- current ---\n%s\n--- proposed ---\n%s", - s, r.Current, r.Proposed) - } - } -} diff --git a/mdl/executor/cmd_diff.go b/mdl/executor/cmd_diff.go index 8e04a48ba1..9a9b22de1f 100644 --- a/mdl/executor/cmd_diff.go +++ b/mdl/executor/cmd_diff.go @@ -1,18 +1,17 @@ // SPDX-License-Identifier: Apache-2.0 -// Package executor - Diff command implementation for comparing MDL scripts against project state +// Package executor - diff results and their rendering, shared by `mxcli diff` +// (mdl/scriptdiff, which runs the script against a scratch copy) and +// `mxcli diff-local`. package executor import ( - "context" - "errors" "fmt" - "sort" - "strings" - "unicode" + "io" + + "github.com/pmezard/go-difflib/difflib" "github.com/mendixlabs/mxcli/mdl/ast" - mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" ) // DiffFormat represents the output format for diff results @@ -53,16 +52,13 @@ type DiffResult struct { ObjectType string ObjectName ast.QualifiedName Current string // MDL from MPR (empty if new) - Proposed string // MDL from script + Proposed string // MDL after the change (empty if deleted) IsNew bool IsDeleted bool Changes []StructuralChange - // Refused is why exec would refuse the statement, writing nothing, when - // it would (ako/mxcli#839); "" otherwise. - Refused string - // Writes is what exec would write when the two renderings are the same - // but exec still writes (a patch the rendering does not show, a move to - // another folder); "" otherwise. It makes the statement modified. + // Writes is what is written that the two renderings do not show (a move + // to another folder, a property describe does not print), or why they + // could not be made; "" otherwise. Writes string } @@ -75,667 +71,74 @@ const ( colorYellow = "\033[33m" ) -// DiffProgram compares an MDL program against the current project state -func diffProgram(ctx *ExecContext, prog *ast.Program, opts DiffOptions) error { - if !ctx.Connected() { - return mdlerrors.NewNotConnected() - } - - // Set defaults +// WriteDiffResults prints results in the chosen format. A result whose change +// the renderings do not show is printed as one line saying what is written. +func WriteDiffResults(w io.Writer, results []DiffResult, opts DiffOptions) { if opts.Format == "" { opts.Format = DiffFormatUnified } if opts.Width == 0 { opts.Width = 120 } - - var results []DiffResult - var newCount, modifiedCount, unchangedCount, refusedCount int - - // Track processed objects to avoid duplicates (script may have multiple statements for same object) - processed := make(map[string]bool) - - // Process each statement - // Statements diff cannot compare, and statements whose comparison failed. - // Both are reported rather than dropped — see unsupportedDiffError. - skipped := map[string]int{} - var failures []string - - for _, stmt := range prog.Statements { - result, err := diffStatement(ctx, stmt) - if err != nil { - var unsupported *unsupportedDiffError - if errors.As(err, &unsupported) { - skipped[unsupported.kind]++ - } else { - failures = append(failures, err.Error()) - } - continue - } - if result != nil { - // CREATE … IF NOT EXISTS on an element that is already there is - // skipped by exec and leaves the element untouched (#731), so what - // the script would leave behind is what is stored now. - if g, ok := stmt.(ast.IfNotExistsCreate); ok && g.CreateIfNotExists() && !result.IsNew { - result.Proposed = result.Current - result.Changes = nil - } - // Create unique key for deduplication - key := result.ObjectType + ":" + result.ObjectName.String() - if processed[key] { - // Skip duplicate - already processed this object - continue - } - processed[key] = true - - results = append(results, *result) - if result.Refused != "" { - refusedCount++ - } else if result.IsNew { - newCount++ - } else if result.Current != result.Proposed || result.Writes != "" { - modifiedCount++ - } else { - unchangedCount++ - } - } - } - - // Output results based on format + ctx := &ExecContext{Output: w} for _, result := range results { - if result.Refused != "" { - fmt.Fprintf(ctx.Output, "Refused: %s %s: exec would refuse this statement and write nothing: %s\n", - result.ObjectType, result.ObjectName, result.Refused) + verdict := "Modified" + switch { + case result.IsNew: + verdict = "New" + case result.IsDeleted: + verdict = "Removed" + } + shows := result.Current != result.Proposed + if !shows { + fmt.Fprintf(w, "%s: %s %s: %s\n", verdict, result.ObjectType, result.ObjectName, result.Writes) continue } - if result.Writes != "" && result.Current == result.Proposed { - fmt.Fprintf(ctx.Output, "Modified: %s %s: its MDL renders as stored, but exec would write it: %s\n", - result.ObjectType, result.ObjectName, result.Writes) - continue - } - if result.Current == result.Proposed && !result.IsNew { - // Skip unchanged objects unless showing structural - if opts.Format != DiffFormatStructural { - continue - } - } - switch opts.Format { - case DiffFormatUnified: - outputUnifiedDiff(ctx, result, opts.UseColor) case DiffFormatSideBySide: outputSideBySideDiff(ctx, result, opts.Width, opts.UseColor) case DiffFormatStructural: outputStructuralDiff(ctx, result, opts.UseColor) + default: + outputUnifiedDiff(ctx, result, opts.UseColor) } - } - - // Output summary - summary := fmt.Sprintf("\nSummary: %d new, %d modified, %d unchanged", newCount, modifiedCount, unchangedCount) - if refusedCount > 0 { - summary += fmt.Sprintf(", %d refused", refusedCount) - } - fmt.Fprintln(ctx.Output, summary) - reportUndiffed(ctx, skipped, failures) - - return nil -} - -// reportUndiffed prints what the summary above does NOT account for. -// -// The counts only ever describe statements diff understands, so a script made -// entirely of statements it does not understand summarises as all zeros. That -// reads as "nothing would change" for a script that may add documents, which -// is exactly the wrong answer from a pre-apply safety gate. -func reportUndiffed(ctx *ExecContext, skipped map[string]int, failures []string) { - if len(skipped) > 0 { - kinds := make([]string, 0, len(skipped)) - for k := range skipped { - kinds = append(kinds, k) - } - sort.Strings(kinds) - total := 0 - for _, n := range skipped { - total += n - } - fmt.Fprintf(ctx.Output, "\nNot compared (%d statement(s)) — diff has no comparison for these,\n"+ - "so they are absent from the summary above, not unchanged:\n", total) - for _, k := range kinds { - fmt.Fprintf(ctx.Output, " %s x%d\n", k, skipped[k]) - } - } - for _, f := range failures { - fmt.Fprintf(ctx.Output, "\nCould not diff: %s\n", f) - } -} - -// DiffProgram is a method wrapper for external callers. -// -// The program runs under its own language header, as exec runs it: what a -// statement means, and whether exec would refuse it, depends on it. -func (e *Executor) DiffProgram(prog *ast.Program, opts DiffOptions) error { - defer e.enterLanguage(prog.LanguageVersion)() - return diffProgram(e.newExecContext(context.Background()), prog, opts) -} - -// diffStatement generates a diff result for a single statement -func diffStatement(ctx *ExecContext, stmt ast.Statement) (*DiffResult, error) { - switch s := stmt.(type) { - case *ast.CreateEntityStmt: - return diffEntity(ctx, s) - case *ast.CreateViewEntityStmt: - return diffViewEntity(ctx, s) - case *ast.CreateEnumerationStmt: - return diffEnumeration(ctx, s) - case *ast.CreateAssociationStmt: - return diffAssociation(ctx, s) - case *ast.CreateMicroflowStmt: - return diffMicroflow(ctx, s) - case *ast.CreateNanoflowStmt: - return diffNanoflow(ctx, s) - default: - return nil, &unsupportedDiffError{kind: statementKindName(stmt)} - } -} - -// unsupportedDiffError marks a statement diff has no comparison for. It is an -// error rather than a nil result so that diffProgram can SAY so: skipping -// silently made `diff` report "0 new, 0 modified, 0 unchanged" for a script -// that would genuinely add documents, which is worse than a wrong count -// because there is nothing on screen to disbelieve (#997). -type unsupportedDiffError struct{ kind string } - -func (e *unsupportedDiffError) Error() string { - return "diff does not compare " + e.kind + " statements" -} - -// statementKindName turns an AST statement type into something an MDL author -// recognises: *ast.GrantMicroflowAccessStmt → "grant microflow access". -func statementKindName(stmt ast.Statement) string { - name := strings.TrimPrefix(fmt.Sprintf("%T", stmt), "*ast.") - name = strings.TrimSuffix(name, "Stmt") - var out []rune - for i, r := range name { - if i > 0 && r >= 'A' && r <= 'Z' { - out = append(out, ' ') - } - out = append(out, unicode.ToLower(r)) - } - return string(out) -} - -// diffEntity compares a CREATE ENTITY statement against the project -func diffEntity(ctx *ExecContext, s *ast.CreateEntityStmt) (*DiffResult, error) { - result := &DiffResult{ - ObjectType: "Entity", - ObjectName: s.Name, - Proposed: entityStmtToMDL(ctx, s), - } - - // Try to find existing entity - module, err := findModule(ctx, s.Name.Module) - if err != nil { - result.IsNew = true - return result, nil - } - - dm, err := ctx.Backend.GetDomainModel(module.ID) - if err != nil { - result.IsNew = true - return result, nil - } - - for _, entity := range dm.Entities { - if entity.Name == s.Name.Name { - // Found existing entity - get its MDL representation - result.Current = entityToMDL(ctx, module.Name, entity, dm) - result.Changes = compareEntities(ctx, result.Current, result.Proposed) - return result, nil - } - } - - result.IsNew = true - return result, nil -} - -// diffViewEntity compares a CREATE VIEW ENTITY statement against the project -func diffViewEntity(ctx *ExecContext, s *ast.CreateViewEntityStmt) (*DiffResult, error) { - result := &DiffResult{ - ObjectType: "View Entity", - ObjectName: s.Name, - Proposed: viewEntityStmtToMDL(ctx, s), - } - - module, err := findModule(ctx, s.Name.Module) - if err != nil { - result.IsNew = true - return result, nil - } - - dm, err := ctx.Backend.GetDomainModel(module.ID) - if err != nil { - result.IsNew = true - return result, nil - } - - for _, entity := range dm.Entities { - if entity.Name == s.Name.Name { - result.Current = viewEntityFromProjectToMDL(ctx, module.Name, entity, dm) - return result, nil - } - } - - result.IsNew = true - return result, nil -} - -// diffEnumeration compares a CREATE ENUMERATION statement against the project -func diffEnumeration(ctx *ExecContext, s *ast.CreateEnumerationStmt) (*DiffResult, error) { - result := &DiffResult{ - ObjectType: "Enumeration", - ObjectName: s.Name, - Proposed: enumerationStmtToMDL(ctx, s), - } - - // Try to find existing enumeration - existingEnum := findEnumeration(ctx, s.Name.Module, s.Name.Name) - if existingEnum == nil { - result.IsNew = true - return result, nil - } - - // ContainerID is a folder when the enumeration is filed in one, so walk up - // to the module before asking for its name; asking directly rendered the - // stored side as `create enumeration .Name` and made an untouched - // enumeration diff as modified (ako/mxcli#794). findEnumeration matched the - // statement's module through the same walk, so that is the module name. - h, _ := getHierarchy(ctx) - modName := h.GetModuleName(h.FindModuleID(existingEnum.ContainerID)) - result.Current = enumerationToMDL(ctx, modName, existingEnum) - result.Changes = compareEnumerations(ctx, result.Current, result.Proposed) - - return result, nil -} - -// diffAssociation compares a CREATE ASSOCIATION statement against the project -func diffAssociation(ctx *ExecContext, s *ast.CreateAssociationStmt) (*DiffResult, error) { - result := &DiffResult{ - ObjectType: "Association", - ObjectName: s.Name, - } - - module, err := findModule(ctx, s.Name.Module) - if err != nil { - result.IsNew = true - result.Proposed = associationStmtToMDL(ctx, s, "") - return result, nil - } - - dm, err := ctx.Backend.GetDomainModel(module.ID) - if err != nil { - result.IsNew = true - result.Proposed = associationStmtToMDL(ctx, s, "") - return result, nil - } - - for _, assoc := range dm.Associations { - if assoc.Name == s.Name.Name { - result.Current = associationToMDL(ctx, module.Name, assoc, dm) - result.Proposed = associationStmtToMDL(ctx, s, assoc.StorageFormat) - return result, nil - } - } - // A cross-module association is stored apart, in CrossAssociations; without - // this lookup every existing one diffed as new. - for _, ca := range dm.CrossAssociations { - if ca.Name == s.Name.Name { - result.Current = crossAssociationToMDL(module.Name, ca, dm) - result.Proposed = associationStmtToMDL(ctx, s, ca.StorageFormat) - return result, nil - } - } - - result.IsNew = true - result.Proposed = associationStmtToMDL(ctx, s, "") - return result, nil -} - -// diffMicroflow compares a CREATE MICROFLOW statement against the project -func diffMicroflow(ctx *ExecContext, s *ast.CreateMicroflowStmt) (*DiffResult, error) { - result := &DiffResult{ObjectType: "Microflow", ObjectName: s.Name} - - // Build the flow the script describes, without writing anything, then - // render it through the SAME describer the stored side goes through. The - // second AST-to-MDL renderer this replaces dropped every activity type it - // did not know, which the diff then showed as a deletion (#997). - built, err := buildMicroflowFromStmt(ctx, s, buildFlowOpts{}) - if err != nil { - return nil, err - } - proposed, err := renderFlowFromModel(ctx, "microflow", built.Microflow, s.Name) - if err != nil { - return nil, err - } - result.Proposed = proposed - - if built.ExistingID == "" { - result.IsNew = true - return result, nil - } - - stored, err := ctx.Backend.GetMicroflow(built.ExistingID) - if err != nil || stored == nil { - result.IsNew = true - return result, nil - } - current, err := renderFlowFromModel(ctx, "microflow", stored, s.Name) - if err != nil { - return nil, err - } - result.Current = current - result.Changes = compareMicroflows(ctx, result.Current, result.Proposed) - if s.CreateOrModify { - spliceVerdict(ctx, microflowDecl(s), result) - } - return result, nil -} - -// diffNanoflow compares a CREATE NANOFLOW statement against the project -func diffNanoflow(ctx *ExecContext, s *ast.CreateNanoflowStmt) (*DiffResult, error) { - result := &DiffResult{ObjectType: "Nanoflow", ObjectName: s.Name} - - built, err := buildNanoflowFromStmt(ctx, s, buildFlowOpts{}) - if err != nil { - return nil, err - } - proposed, err := renderFlowFromModel(ctx, "nanoflow", nanoflowAsMicroflow(built.Nanoflow), s.Name) - if err != nil { - return nil, err - } - result.Proposed = proposed - - if built.ExistingID == "" { - result.IsNew = true - return result, nil - } - - stored, err := ctx.Backend.GetNanoflow(built.ExistingID) - if err != nil || stored == nil { - result.IsNew = true - return result, nil - } - current, err := renderFlowFromModel(ctx, "nanoflow", nanoflowAsMicroflow(stored), s.Name) - if err != nil { - return nil, err - } - result.Current = current - result.Changes = compareMicroflows(ctx, result.Current, result.Proposed) - if s.CreateOrModify { - spliceVerdict(ctx, nanoflowDecl(s), result) - } - return result, nil -} - -// spliceVerdict brings the diff of a `create or modify` of a stored flow to the -// verdict exec reaches, which is not the rendered comparison above: exec -// patches the stored flow (diff-then-patch, planFlowModify), and it is the -// patch that decides whether anything is written (ako/mxcli#839, where diff -// said unchanged for a statement exec refused). -// -// - A change the splice cannot make is refused under mdl 1, and diff says -// so; under mdl 0 exec rebuilds the flow, which the rendered comparison -// already shows. -// - An empty patch in the same folder writes nothing, whatever the two -// renderings say, so it is unchanged. -// - A patch to make, or a move to another folder, is a write whatever the -// two renderings say: when they are the same (the rendering leaves the -// folder out), what exec would write is stated instead. -// -// An error exec would report itself leaves the rendered comparison as it is. -func spliceVerdict(ctx *ExecContext, d *flowDecl, result *DiffResult) { - v := decideFlowModify(ctx, d) - p := v.plan - switch { - case v.why != nil: - if v.refused { - result.Refused = v.why.reason - } - case v.err != nil || p == nil: - case p.mut == nil && !movesFolder(d.folder, p.storedFolder): - result.Proposed = result.Current - result.Changes = nil - case result.Current == result.Proposed: - var what []string - if s := patchSummary(p.ops, p.moves, p.set); s != "" { - what = append(what, s) - } - if movesFolder(d.folder, p.storedFolder) { - what = append(what, fmt.Sprintf("moved to folder '%s'", d.folder)) - } - if len(what) == 0 { - what = append(what, "patched") - } - result.Writes = strings.Join(what, "; ") - } -} - -// ============================================================================ -// Structural Comparison Functions -// ============================================================================ - -// compareEntities extracts structural changes between two entity MDL representations -func compareEntities(ctx *ExecContext, current, proposed string) []StructuralChange { - var changes []StructuralChange - - // Simple line-based comparison for now - currentLines := strings.Split(current, "\n") - proposedLines := strings.Split(proposed, "\n") - - // Extract attributes from both - currentAttrs := extractAttributes(ctx, currentLines) - proposedAttrs := extractAttributes(ctx, proposedLines) - - // Find added attributes - for name, proposed := range proposedAttrs { - if _, exists := currentAttrs[name]; !exists { - changes = append(changes, StructuralChange{ - ChangeType: ChangeAdded, - ElementType: "Attribute", - ElementName: name, - Details: proposed, - }) - } - } - - // Find removed attributes - for name := range currentAttrs { - if _, exists := proposedAttrs[name]; !exists { - changes = append(changes, StructuralChange{ - ChangeType: ChangeRemoved, - ElementType: "Attribute", - ElementName: name, - }) - } - } - - // Find modified attributes - for name, proposed := range proposedAttrs { - if current, exists := currentAttrs[name]; exists && current != proposed { - changes = append(changes, StructuralChange{ - ChangeType: ChangeModified, - ElementType: "Attribute", - ElementName: name, - Details: "changed", - }) - } - } - - return changes -} - -// compareEnumerations extracts structural changes between two enumeration MDL representations -func compareEnumerations(ctx *ExecContext, current, proposed string) []StructuralChange { - var changes []StructuralChange - - currentValues := extractEnumValues(ctx, strings.Split(current, "\n")) - proposedValues := extractEnumValues(ctx, strings.Split(proposed, "\n")) - - for name := range proposedValues { - if _, exists := currentValues[name]; !exists { - changes = append(changes, StructuralChange{ - ChangeType: ChangeAdded, - ElementType: "Value", - ElementName: name, - }) - } - } - - for name := range currentValues { - if _, exists := proposedValues[name]; !exists { - changes = append(changes, StructuralChange{ - ChangeType: ChangeRemoved, - ElementType: "Value", - ElementName: name, - }) + if result.Writes != "" { + fmt.Fprintf(w, "%s: %s %s: also %s\n\n", verdict, result.ObjectType, result.ObjectName, result.Writes) } } - - return changes } -// compareMicroflows extracts structural changes between two microflow MDL representations -func compareMicroflows(ctx *ExecContext, current, proposed string) []StructuralChange { - var changes []StructuralChange - - currentParams := extractParameters(ctx, strings.Split(current, "\n")) - proposedParams := extractParameters(ctx, strings.Split(proposed, "\n")) - - for name := range proposedParams { - if _, exists := currentParams[name]; !exists { - changes = append(changes, StructuralChange{ - ChangeType: ChangeAdded, - ElementType: "Parameter", - ElementName: name, - }) - } - } - - for name := range currentParams { - if _, exists := proposedParams[name]; !exists { - changes = append(changes, StructuralChange{ - ChangeType: ChangeRemoved, - ElementType: "Parameter", - ElementName: name, - }) - } +// LineChanges summarises the difference between two renderings as lines added +// and removed, for the structural format. +func LineChanges(current, proposed string) []StructuralChange { + if current == proposed { + return nil } - - // Count body statements - currentStmts := countBodyStatements(ctx, current) - proposedStmts := countBodyStatements(ctx, proposed) - if currentStmts != proposedStmts { - diff := proposedStmts - currentStmts - if diff > 0 { - changes = append(changes, StructuralChange{ - ChangeType: ChangeAdded, - ElementType: "Body", - ElementName: "statements", - Details: fmt.Sprintf("%d statements added", diff), - }) - } else { - changes = append(changes, StructuralChange{ - ChangeType: ChangeRemoved, - ElementType: "Body", - ElementName: "statements", - Details: fmt.Sprintf("%d statements removed", -diff), - }) - } - } - - return changes -} - -// extractAttributes extracts attribute definitions from MDL lines -func extractAttributes(_ *ExecContext, lines []string) map[string]string { - attrs := make(map[string]string) - for _, line := range lines { - line = strings.TrimSpace(line) - if strings.Contains(line, ":") && !strings.HasPrefix(line, "create") && !strings.HasPrefix(line, "/**") && !strings.HasPrefix(line, "*") { - parts := strings.SplitN(line, ":", 2) - if len(parts) == 2 { - name := strings.TrimSpace(parts[0]) - if !strings.HasPrefix(name, "$") && !strings.HasPrefix(name, "@") { - attrs[name] = strings.TrimSuffix(strings.TrimSpace(parts[1]), ",") - } - } + lines := func(s string) []string { + if s == "" { + return nil } + return difflib.SplitLines(s) } - return attrs -} - -// extractEnumValues extracts enumeration values from MDL lines -func extractEnumValues(_ *ExecContext, lines []string) map[string]bool { - values := make(map[string]bool) - for _, line := range lines { - line = strings.TrimSpace(line) - if strings.Contains(line, "'") && !strings.HasPrefix(line, "create") { - parts := strings.Fields(line) - if len(parts) >= 1 { - name := strings.TrimSuffix(parts[0], ",") - if name != "" && !strings.HasPrefix(name, "/") && !strings.HasPrefix(name, "*") { - values[name] = true - } - } + m := difflib.NewMatcher(lines(current), lines(proposed)) + added, removed := 0, 0 + for _, op := range m.GetOpCodes() { + switch op.Tag { + case 'r': + removed += op.I2 - op.I1 + added += op.J2 - op.J1 + case 'd': + removed += op.I2 - op.I1 + case 'i': + added += op.J2 - op.J1 } } - return values -} - -// extractParameters extracts parameter names from MDL lines -func extractParameters(_ *ExecContext, lines []string) map[string]bool { - params := make(map[string]bool) - inParams := false - for _, line := range lines { - line = strings.TrimSpace(line) - if strings.HasPrefix(line, "create microflow") || strings.HasPrefix(line, "create nanoflow") || - strings.HasPrefix(line, "create or modify microflow") || strings.HasPrefix(line, "create or modify nanoflow") { - inParams = true - continue - } - if inParams { - if strings.HasPrefix(line, ")") { - inParams = false - continue - } - if strings.HasPrefix(line, "$") { - parts := strings.SplitN(line, ":", 2) - if len(parts) >= 1 { - name := strings.TrimPrefix(parts[0], "$") - name = strings.TrimSuffix(name, ",") - params[strings.TrimSpace(name)] = true - } - } - } + var out []StructuralChange + if added > 0 { + out = append(out, StructuralChange{ChangeType: ChangeAdded, ElementType: "Lines", Details: fmt.Sprintf("%d line(s) added", added)}) } - return params -} - -// countBodyStatements counts statements in a microflow body -func countBodyStatements(_ *ExecContext, mdl string) int { - count := 0 - inBody := false - for line := range strings.SplitSeq(mdl, "\n") { - line = strings.TrimSpace(line) - if line == "begin" { - inBody = true - continue - } - if line == "end;" { - break - } - if inBody && line != "" && !strings.HasPrefix(line, "--") { - count++ - } + if removed > 0 { + out = append(out, StructuralChange{ChangeType: ChangeRemoved, ElementType: "Lines", Details: fmt.Sprintf("%d line(s) removed", removed)}) } - return count + return out } diff --git a/mdl/executor/cmd_diff_if_not_exists_test.go b/mdl/executor/cmd_diff_if_not_exists_test.go deleted file mode 100644 index af81992aa9..0000000000 --- a/mdl/executor/cmd_diff_if_not_exists_test.go +++ /dev/null @@ -1,62 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 - -package executor - -import ( - "strings" - "testing" - - "github.com/mendixlabs/mxcli/mdl/backend/mock" - "github.com/mendixlabs/mxcli/mdl/visitor" - "github.com/mendixlabs/mxcli/model" -) - -// diffEnumScript diffs src against a project holding enumeration M.E (Red). -func diffEnumScript(t *testing.T, src string) string { - t.Helper() - mod := mkModule("M") - enum := mkEnumeration(mod.ID, "E", "Red") - mb := &mock.MockBackend{ - IsConnectedFunc: func() bool { return true }, - ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, - ListEnumerationsFunc: func() ([]*model.Enumeration, error) { return []*model.Enumeration{enum}, nil }, - } - ctx, out := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) - prog, errs := visitor.Build(src) - if len(errs) > 0 { - t.Fatalf("parse %q: %v", src, errs) - } - if err := diffProgram(ctx, prog, DiffOptions{}); err != nil { - t.Fatalf("diff: %v", err) - } - return out.String() -} - -// diff is the pre-apply view of exec, so a guarded create of an element that -// already exists — which exec skips and leaves untouched (ako/mxcli#731) — is -// unchanged, not modified. -func TestDiff_CreateIfNotExistsOnExistingElementIsUnchanged(t *testing.T) { - out := diffEnumScript(t, "create enumeration if not exists M.E (Blue 'Blue');") - if !strings.Contains(out, "Summary: 0 new, 0 modified, 1 unchanged") { - t.Errorf("a guarded create of an existing element must diff as unchanged; got:\n%s", out) - } - if strings.Contains(out, "Blue") { - t.Errorf("diff shows the script's definition, which exec will not apply:\n%s", out) - } -} - -// CONTROL: without the guard the same statement is a modification. -func TestDiff_UnguardedCreateOnExistingElementIsModified(t *testing.T) { - out := diffEnumScript(t, "create or modify enumeration M.E (Blue 'Blue');") - if !strings.Contains(out, "Summary: 0 new, 1 modified, 0 unchanged") { - t.Errorf("control: an unguarded create of a changed element must diff as modified; got:\n%s", out) - } -} - -// CONTROL: a guarded create of an absent element is new. -func TestDiff_CreateIfNotExistsOnAbsentElementIsNew(t *testing.T) { - out := diffEnumScript(t, "create enumeration if not exists M.F (Blue 'Blue');") - if !strings.Contains(out, "Summary: 1 new, 0 modified, 0 unchanged") { - t.Errorf("a guarded create of an absent element must diff as new; got:\n%s", out) - } -} diff --git a/mdl/executor/cmd_diff_mdl.go b/mdl/executor/cmd_diff_mdl.go deleted file mode 100644 index cd6ba7f1a3..0000000000 --- a/mdl/executor/cmd_diff_mdl.go +++ /dev/null @@ -1,510 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 - -// Package executor - MDL generation functions for diff (statement→text and project→text converters) -package executor - -import ( - "fmt" - "strings" - - "github.com/mendixlabs/mxcli/mdl/ast" - "github.com/mendixlabs/mxcli/model" - "github.com/mendixlabs/mxcli/sdk/domainmodel" -) - -// ============================================================================ -// Statement to MDL Converters -// ============================================================================ - -// entityStmtToMDL converts a CreateEntityStmt to MDL text -func entityStmtToMDL(ctx *ExecContext, s *ast.CreateEntityStmt) string { - var lines []string - - // Documentation - if s.Documentation != "" { - lines = append(lines, "/**") - lines = append(lines, " * "+s.Documentation) - lines = append(lines, " */") - } - - // Position annotation - if s.Position != nil { - lines = append(lines, fmt.Sprintf("@Position(%d, %d)", s.Position.X, s.Position.Y)) - } - - // Entity type. EntityKind.String() is upper case for error messages; the - // project side of the diff writes the keyword in lower case, and an - // unmodified describe dump was reported as modified purely on that casing - // (#997, the same two-renderer split as the flow body). - entityType := strings.ToLower(s.Kind.String()) - lines = append(lines, fmt.Sprintf("create %s entity %s (", entityType, s.Name)) - - // Attributes - for i, attr := range s.Attributes { - // Attribute documentation - if attr.Documentation != "" { - lines = append(lines, fmt.Sprintf(" /** %s */", attr.Documentation)) - } - - typeStr := dataTypeToString(ctx, attr.Type) - constraints := "" - - if attr.NotNull { - constraints += " not null" - if attr.NotNullError != "" { - constraints += fmt.Sprintf(" error message '%s'", attr.NotNullError) - } - } - if attr.Unique { - constraints += " unique" - if attr.UniqueError != "" { - constraints += fmt.Sprintf(" error message '%s'", attr.UniqueError) - } - } - if attr.HasDefault { - defaultVal := fmt.Sprintf("%v", attr.DefaultValue) - if attr.Type.Kind == ast.TypeString { - defaultVal = fmt.Sprintf("'%s'", attr.DefaultValue) - } - constraints += fmt.Sprintf(" default %s", defaultVal) - } - - comma := "," - if i == len(s.Attributes)-1 { - comma = "" - } - lines = append(lines, fmt.Sprintf(" %s: %s%s%s", attr.Name, typeStr, constraints, comma)) - } - - lines = append(lines, ")") - - // Indexes - for _, idx := range s.Indexes { - var cols []string - for _, col := range idx.Columns { - colStr := col.Name - if col.Descending { - colStr += " desc" - } - cols = append(cols, colStr) - } - lines = append(lines, fmt.Sprintf("index (%s)", strings.Join(cols, ", "))) - } - - lines = append(lines, ";") - - return strings.Join(lines, "\n") -} - -// viewEntityStmtToMDL converts a CreateViewEntityStmt to MDL text -func viewEntityStmtToMDL(ctx *ExecContext, s *ast.CreateViewEntityStmt) string { - var lines []string - - if s.Documentation != "" { - lines = append(lines, "/**") - lines = append(lines, " * "+s.Documentation) - lines = append(lines, " */") - } - - if s.Position != nil { - lines = append(lines, fmt.Sprintf("@Position(%d, %d)", s.Position.X, s.Position.Y)) - } - - lines = append(lines, fmt.Sprintf("create view entity %s (", s.Name)) - - for i, attr := range s.Attributes { - typeStr := dataTypeToString(ctx, attr.Type) - comma := "," - if i == len(s.Attributes)-1 { - comma = "" - } - lines = append(lines, fmt.Sprintf(" %s: %s%s", attr.Name, typeStr, comma)) - } - - lines = append(lines, ") as (") - // Indent OQL query - for line := range strings.SplitSeq(s.Query.RawQuery, "\n") { - lines = append(lines, " "+line) - } - lines = append(lines, ");") - - return strings.Join(lines, "\n") -} - -// enumerationStmtToMDL converts a CreateEnumerationStmt to MDL text -func enumerationStmtToMDL(ctx *ExecContext, s *ast.CreateEnumerationStmt) string { - var lines []string - - if s.Documentation != "" { - lines = append(lines, "/**") - lines = append(lines, " * "+s.Documentation) - lines = append(lines, " */") - } - - lines = append(lines, fmt.Sprintf("create enumeration %s (", s.Name)) - - for i, v := range s.Values { - comma := "," - if i == len(s.Values)-1 { - comma = "" - } - lines = append(lines, fmt.Sprintf(" %s '%s'%s", v.Name, v.Caption, comma)) - } - - lines = append(lines, ");") - - return strings.Join(lines, "\n") -} - -// associationStmtToMDL converts a CreateAssociationStmt to MDL text -// -// stored is the existing association's storage format ("" when there is none), -// so an unstated storage renders as what exec would leave in place rather than -// as a change. -func associationStmtToMDL(ctx *ExecContext, s *ast.CreateAssociationStmt, stored domainmodel.AssociationStorageFormat) string { - var lines []string - - if s.Documentation != "" { - lines = append(lines, "/**") - lines = append(lines, " * "+s.Documentation) - lines = append(lines, " */") - } - - lines = append(lines, fmt.Sprintf("create association %s", s.Name)) - lines = append(lines, fmt.Sprintf("from %s to %s", s.Parent, s.Child)) - - assocType := "Reference" - if s.Type == ast.AssocReferenceSet { - assocType = "ReferenceSet" - } - lines = append(lines, fmt.Sprintf("type %s", assocType)) - - owner := "Default" - if s.Owner == ast.OwnerBoth { - owner = "Both" - } - lines = append(lines, fmt.Sprintf("owner %s", owner)) - - storage, stated := statedStorageFormat(s.Storage) - if !stated { - storage = stored - if storage == "" { - storage = domainmodel.StorageFormatColumn - } - } - if line := storageClause(storage); line != "" { - lines = append(lines, line) - } - - deleteBehavior := "set null" - switch s.DeleteBehavior { - case ast.DeleteCascade: - deleteBehavior = "cascade" - case ast.DeleteIfNoReferences: - deleteBehavior = "restrict" - } - lines = append(lines, fmt.Sprintf("on delete %s;", deleteBehavior)) - - return strings.Join(lines, "\n") -} - -// ============================================================================ -// Project to MDL Converters -// ============================================================================ - -// entityToMDL converts a project entity to MDL text -func entityToMDL(ctx *ExecContext, moduleName string, entity *domainmodel.Entity, dm *domainmodel.DomainModel) string { - var lines []string - lang := describeDefaultLanguage(ctx) - - // Documentation - if entity.Documentation != "" { - lines = append(lines, "/**") - lines = append(lines, " * "+entity.Documentation) - lines = append(lines, " */") - } - - // Position - lines = append(lines, fmt.Sprintf("@Position(%d, %d)", entity.Location.X, entity.Location.Y)) - - // Entity type - entityType := "persistent" - if strings.Contains(entity.Source, "OqlView") { - entityType = "view" - } else if !entity.Persistable { - entityType = "non-persistent" - } - - lines = append(lines, fmt.Sprintf("create %s entity %s.%s (", entityType, moduleName, entity.Name)) - - // Build validation rules map - validationsByAttr := make(map[model.ID][]*domainmodel.ValidationRule) - validationsByName := make(map[string][]*domainmodel.ValidationRule) - for _, vr := range entity.ValidationRules { - validationsByAttr[vr.AttributeID] = append(validationsByAttr[vr.AttributeID], vr) - attrName := extractAttrNameFromQualified(string(vr.AttributeID)) - if attrName != "" { - validationsByName[attrName] = append(validationsByName[attrName], vr) - } - } - - // Attributes - for i, attr := range entity.Attributes { - // Documentation - if attr.Documentation != "" { - lines = append(lines, fmt.Sprintf(" /** %s */", attr.Documentation)) - } - - typeStr := formatAttributeType(attr.Type) - var constraints strings.Builder - - // Check for validation rules - attrValidations := validationsByAttr[attr.ID] - if len(attrValidations) == 0 { - attrValidations = validationsByName[attr.Name] - } - for _, vr := range attrValidations { - if vr.Type == "Required" { - constraints.WriteString(" not null") - if errMsg := pickTextTranslation(vr.ErrorMessage, lang); errMsg != "" { - constraints.WriteString(fmt.Sprintf(" error message '%s'", errMsg)) - } - } - if vr.Type == "Unique" { - constraints.WriteString(" unique") - if errMsg := pickTextTranslation(vr.ErrorMessage, lang); errMsg != "" { - constraints.WriteString(fmt.Sprintf(" error message '%s'", errMsg)) - } - } - } - - // Default value - if attr.Value != nil && attr.Value.DefaultValue != "" { - defaultVal := attr.Value.DefaultValue - if _, ok := attr.Type.(*domainmodel.StringAttributeType); ok { - defaultVal = fmt.Sprintf("'%s'", defaultVal) - } - constraints.WriteString(fmt.Sprintf(" default %s", defaultVal)) - } - - comma := "," - if i == len(entity.Attributes)-1 { - comma = "" - } - lines = append(lines, fmt.Sprintf(" %s: %s%s%s", attr.Name, typeStr, constraints.String(), comma)) - } - - lines = append(lines, ")") - - // Build attr name map for indexes - attrNames := make(map[model.ID]string) - for _, attr := range entity.Attributes { - attrNames[attr.ID] = attr.Name - } - - // Indexes - for _, idx := range entity.Indexes { - var cols []string - for _, ia := range idx.Attributes { - colName := attrNames[ia.AttributeID] - if !ia.Ascending { - colName += " desc" - } - cols = append(cols, colName) - } - if len(cols) > 0 { - lines = append(lines, fmt.Sprintf("index (%s)", strings.Join(cols, ", "))) - } - } - - lines = append(lines, ";") - - return strings.Join(lines, "\n") -} - -// viewEntityFromProjectToMDL converts a view entity from project to MDL -func viewEntityFromProjectToMDL(ctx *ExecContext, moduleName string, entity *domainmodel.Entity, dm *domainmodel.DomainModel) string { - var lines []string - - if entity.Documentation != "" { - lines = append(lines, "/**") - lines = append(lines, " * "+entity.Documentation) - lines = append(lines, " */") - } - - lines = append(lines, fmt.Sprintf("@Position(%d, %d)", entity.Location.X, entity.Location.Y)) - lines = append(lines, fmt.Sprintf("create view entity %s.%s (", moduleName, entity.Name)) - - for i, attr := range entity.Attributes { - typeStr := formatAttributeType(attr.Type) - comma := "," - if i == len(entity.Attributes)-1 { - comma = "" - } - lines = append(lines, fmt.Sprintf(" %s: %s%s", attr.Name, typeStr, comma)) - } - - lines = append(lines, ") as (") - if entity.OqlQuery != "" { - for line := range strings.SplitSeq(entity.OqlQuery, "\n") { - lines = append(lines, " "+line) - } - } - lines = append(lines, ");") - - return strings.Join(lines, "\n") -} - -// enumerationToMDL converts a project enumeration to MDL text -func enumerationToMDL(ctx *ExecContext, moduleName string, enum *model.Enumeration) string { - var lines []string - - if enum.Documentation != "" { - lines = append(lines, "/**") - lines = append(lines, " * "+enum.Documentation) - lines = append(lines, " */") - } - - lines = append(lines, fmt.Sprintf("create enumeration %s.%s (", moduleName, enum.Name)) - - // Same read as DESCRIBE: a hardcoded "en_US" renders every caption of a - // non-en_US project as '' and makes the diff claim the script changes them - // (mendixlabs/mxcli#1113). - lang := describeDefaultLanguage(ctx) - - for i, v := range enum.Values { - comma := "," - if i == len(enum.Values)-1 { - comma = "" - } - caption := pickTextTranslation(v.Caption, lang) - lines = append(lines, fmt.Sprintf(" %s '%s'%s", v.Name, caption, comma)) - } - - lines = append(lines, ");") - - return strings.Join(lines, "\n") -} - -// associationToMDL converts a project association to MDL text -func associationToMDL(ctx *ExecContext, moduleName string, assoc *domainmodel.Association, dm *domainmodel.DomainModel) string { - // Build entity name map - entityNames := make(map[model.ID]string) - for _, entity := range dm.Entities { - entityNames[entity.ID] = entity.Name - } - return storedAssociationToMDL(moduleName, assoc.Name, assoc.Documentation, - moduleName+"."+entityNames[assoc.ParentID], moduleName+"."+entityNames[assoc.ChildID], - assoc.Type, assoc.Owner, assoc.StorageFormat, assoc.ChildDeleteBehavior) -} - -// crossAssociationToMDL converts a project cross-module association (FROM an -// entity of this module, TO one named in another) to MDL text. -func crossAssociationToMDL(moduleName string, ca *domainmodel.CrossModuleAssociation, dm *domainmodel.DomainModel) string { - fromEntity := "" - for _, entity := range dm.Entities { - if entity.ID == ca.ParentID { - fromEntity = entity.Name - break - } - } - return storedAssociationToMDL(moduleName, ca.Name, ca.Documentation, - moduleName+"."+fromEntity, ca.ChildRef, - ca.Type, ca.Owner, ca.StorageFormat, ca.ChildDeleteBehavior) -} - -// storedAssociationToMDL renders the parts both association kinds share, in the -// shape associationStmtToMDL renders the script side. -func storedAssociationToMDL(moduleName, name, documentation, from, to string, - typ domainmodel.AssociationType, own domainmodel.AssociationOwner, - storage domainmodel.AssociationStorageFormat, childDelete *domainmodel.DeleteBehavior) string { - var lines []string - - if documentation != "" { - lines = append(lines, "/**") - lines = append(lines, " * "+documentation) - lines = append(lines, " */") - } - - lines = append(lines, fmt.Sprintf("create association %s.%s", moduleName, name)) - lines = append(lines, fmt.Sprintf("from %s to %s", from, to)) - - assocType := "Reference" - if typ == domainmodel.AssociationTypeReferenceSet { - assocType = "ReferenceSet" - } - lines = append(lines, fmt.Sprintf("type %s", assocType)) - - owner := "Default" - if own == domainmodel.AssociationOwnerBoth { - owner = "Both" - } - lines = append(lines, fmt.Sprintf("owner %s", owner)) - - if line := storageClause(storage); line != "" { - lines = append(lines, line) - } - - deleteBehavior := "set null" - if childDelete != nil { - switch childDelete.Type { - case domainmodel.DeleteBehaviorTypeDeleteMeAndReferences: - deleteBehavior = "cascade" - case domainmodel.DeleteBehaviorTypeDeleteMeIfNoReferences: - deleteBehavior = "restrict" - } - } - lines = append(lines, fmt.Sprintf("on delete %s;", deleteBehavior)) - - return strings.Join(lines, "\n") -} - -// ============================================================================ -// Helper Functions -// ============================================================================ - -// dataTypeToString converts a DataType to its string representation -func dataTypeToString(_ *ExecContext, dt ast.DataType) string { - switch dt.Kind { - case ast.TypeString: - if dt.Length > 0 { - return fmt.Sprintf("String(%d)", dt.Length) - } - return "String" - case ast.TypeInteger: - return "Integer" - case ast.TypeLong: - return "Long" - case ast.TypeDecimal: - return "Decimal" - case ast.TypeBoolean: - return "Boolean" - case ast.TypeDateTime: - return "DateTime" - case ast.TypeDate: - return "Date" - case ast.TypeAutoNumber: - return "AutoNumber" - case ast.TypeBinary: - return "Binary" - case ast.TypeEnumeration: - if dt.EnumRef != nil { - return fmt.Sprintf("Enumeration(%s)", dt.EnumRef.String()) - } - return "Enumeration" - case ast.TypeEntity: - if dt.EntityRef != nil { - return dt.EntityRef.String() - } - return "Object" - case ast.TypeListOf: - if dt.EntityRef != nil { - return fmt.Sprintf("List of %s", dt.EntityRef.String()) - } - return "List" - case ast.TypeVoid: - return "Void" - default: - return "Unknown" - } -} diff --git a/mdl/executor/cmd_diff_module_name_test.go b/mdl/executor/cmd_diff_module_name_test.go deleted file mode 100644 index f18ea7f00f..0000000000 --- a/mdl/executor/cmd_diff_module_name_test.go +++ /dev/null @@ -1,170 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 - -package executor - -import ( - "regexp" - "strings" - "testing" - - "github.com/mendixlabs/mxcli/mdl/backend/mock" - "github.com/mendixlabs/mxcli/mdl/visitor" - "github.com/mendixlabs/mxcli/model" - "github.com/mendixlabs/mxcli/sdk/domainmodel" - "github.com/mendixlabs/mxcli/sdk/microflows" -) - -// ako/mxcli#794: `mxcli diff` rendered an existing enumeration as -// -// create enumeration .Enum_DistanceUnit ( -// -// — an empty module name. The enumeration sat in a FOLDER, so its ContainerID -// is the folder's ID, and diffEnumeration asked the hierarchy for the module -// NAME of that ID directly instead of walking up to the module first. The same -// defect as DROP ENUMERATION's #976, in a second resolver. The consequence is -// not cosmetic: the stored side's header never matches the script's, so an -// untouched enumeration diffs as "modified". -// -// The issue asked for every diff kind to be checked for the same empty -// qualifier, so this is one table over all of them: each document exists in the -// project, in a folder wherever the kind can live in one, and the stored side of -// the diff must name its module. - -// emptyQualifier matches a qualified name whose module part is missing: -// ".Name" after a space, a paren or the start of a line. -var emptyQualifier = regexp.MustCompile(`(^|[\s(])\.[A-Za-z_]`) - -// diffModuleNameFixture is module M with a folder F holding an enumeration, a -// microflow and a nanoflow; the domain model holds a persistent entity, a view -// entity and an association between two entities. `Root` is the control: the -// same enumeration at the module root, which rendered correctly before the fix. -func diffModuleNameFixture(t *testing.T) *ExecContext { - t.Helper() - mod := mkModule("M") - folderID := model.ID("folder-f") - - h := mkHierarchy(mod) - withContainer(h, folderID, mod.ID) - h.folderNames[folderID] = "F" - - filed := mkEnumeration(folderID, "Filed", "Red") - root := mkEnumeration(mod.ID, "Root", "Red") - - dm := mkDomainModel(mod.ID) - withContainer(h, dm.ID, mod.ID) - parent := mkEntity(dm.ID, "Parent") - child := mkEntity(dm.ID, "Child") - view := mkEntity(dm.ID, "View") - view.Persistable = false - view.Source = "DomainModels$OqlViewEntitySource" - view.OqlQuery = "select 1" - dm.Entities = []*domainmodel.Entity{parent, child, view} - dm.Associations = []*domainmodel.Association{mkAssociation(dm.ID, "Child_Parent", child.ID, parent.ID)} - - mf := mkMicroflow(folderID, "MF") - nf := mkNanoflow(folderID, "NF") - - mb := &mock.MockBackend{ - IsConnectedFunc: func() bool { return true }, - ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, - ListEnumerationsFunc: func() ([]*model.Enumeration, error) { - return []*model.Enumeration{filed, root}, nil - }, - ListDomainModelsFunc: func() ([]*domainmodel.DomainModel, error) { - return []*domainmodel.DomainModel{dm}, nil - }, - GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { return dm, nil }, - ListMicroflowsFunc: func() ([]*microflows.Microflow, error) { return []*microflows.Microflow{mf}, nil }, - ListNanoflowsFunc: func() ([]*microflows.Nanoflow, error) { return []*microflows.Nanoflow{nf}, nil }, - GetMicroflowFunc: func(model.ID) (*microflows.Microflow, error) { return mf, nil }, - GetNanoflowFunc: func(model.ID) (*microflows.Nanoflow, error) { return nf, nil }, - } - ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) - return ctx -} - -func TestDiff_StoredSideNamesItsModule(t *testing.T) { - cases := []struct { - kind, src, want string - }{ - {"enumeration in a folder (#794)", "create or modify enumeration M.Filed (Red 'Red');", "create enumeration M.Filed ("}, - {"enumeration at the module root (control)", "create or modify enumeration M.Root (Red 'Red');", "create enumeration M.Root ("}, - {"entity", "create or modify persistent entity M.Parent ();", "entity M.Parent ("}, - {"view entity", "create or modify view entity M.View () as (select 1);", "create view entity M.View ("}, - {"association", "create or modify association M.Child_Parent from M.Child to M.Parent;", "create association M.Child_Parent"}, - {"microflow in a folder", "create or modify microflow M.MF () begin end;", "M.MF"}, - {"nanoflow in a folder", "create or modify nanoflow M.NF () begin end;", "M.NF"}, - } - for _, tc := range cases { - t.Run(tc.kind, func(t *testing.T) { - ctx := diffModuleNameFixture(t) - prog, errs := visitor.Build(tc.src) - if len(errs) > 0 { - t.Fatalf("parse %q: %v", tc.src, errs) - } - res, err := diffStatement(ctx, prog.Statements[0]) - if err != nil { - t.Fatalf("diff: %v", err) - } - if res.IsNew { - t.Fatalf("the document exists in the project; diff reported it as new") - } - if !strings.Contains(res.Current, tc.want) { - t.Errorf("stored side does not name its module (want %q):\n%s", tc.want, res.Current) - } - t.Logf("stored side:\n%s", res.Current) - for line := range strings.SplitSeq(res.Current, "\n") { - if emptyQualifier.MatchString(line) { - t.Errorf("stored side has an empty module qualifier: %q", line) - } - } - }) - } -} - -// A flow's stored side is headed by the statement's own name, so the table above -// cannot catch an empty qualifier inside its body. Those references — called -// microflows, retrieved entities — are printed from flowNameMaps, and the flows -// here are filed in a folder: every name must still carry its module. -func TestDiff_FlowReferenceNamesCarryTheirModule(t *testing.T) { - ctx := diffModuleNameFixture(t) - entityNames, microflowNames, err := flowNameMaps(ctx) - if err != nil { - t.Fatalf("flowNameMaps: %v", err) - } - want := map[string]bool{"M.Parent": true, "M.Child": true, "M.View": true, "M.MF": true, "M.NF": true} - got := map[string]bool{} - for _, m := range []map[model.ID]string{entityNames, microflowNames} { - for _, name := range m { - got[name] = true - } - } - for name := range want { - if !got[name] { - t.Errorf("reference name %q missing; flowNameMaps produced %v", name, got) - } - } - for name := range got { - if strings.HasPrefix(name, ".") { - t.Errorf("reference name %q has an empty module qualifier", name) - } - } -} - -// The consequence the user sees: an enumeration in a folder, diffed against a -// script that defines it exactly as stored, must be unchanged. Before the fix the -// two headers differed (".Filed" vs "M.Filed") and it diffed as modified. -func TestDiff_UnchangedEnumerationInAFolderIsUnchanged(t *testing.T) { - ctx := diffModuleNameFixture(t) - prog, errs := visitor.Build("create or modify enumeration M.Filed (Red '');") - if len(errs) > 0 { - t.Fatalf("parse: %v", errs) - } - res, err := diffStatement(ctx, prog.Statements[0]) - if err != nil { - t.Fatalf("diff: %v", err) - } - if res.Current != res.Proposed { - t.Errorf("an enumeration identical to the stored one must diff as unchanged\ncurrent:\n%s\nproposed:\n%s", res.Current, res.Proposed) - } -} diff --git a/mdl/executor/cmd_diff_output.go b/mdl/executor/cmd_diff_output.go index f5974b7a0b..82302ed31d 100644 --- a/mdl/executor/cmd_diff_output.go +++ b/mdl/executor/cmd_diff_output.go @@ -41,11 +41,16 @@ func outputUnifiedDiff(ctx *ExecContext, result DiffResult, useColor bool) { } // Use difflib for unified diff + from := fmt.Sprintf("%s.%s (current)", result.ObjectType, result.ObjectName) + to := fmt.Sprintf("%s.%s (script)", result.ObjectType, result.ObjectName) + if result.IsDeleted { + to = "/dev/null" + } diff := difflib.UnifiedDiff{ A: difflib.SplitLines(result.Current), B: difflib.SplitLines(result.Proposed), - FromFile: fmt.Sprintf("%s.%s (current)", result.ObjectType, result.ObjectName), - ToFile: fmt.Sprintf("%s.%s (script)", result.ObjectType, result.ObjectName), + FromFile: from, + ToFile: to, Context: 3, } @@ -87,6 +92,9 @@ func outputSideBySideDiff(ctx *ExecContext, result DiffResult, width int, useCol if result.IsNew { leftHeader = "(new)" } + if result.IsDeleted { + rightHeader = "(removed)" + } fmt.Fprintf(ctx.Output, "%-*s │ %s\n", colWidth, leftHeader, rightHeader) fmt.Fprintln(ctx.Output, strings.Repeat("─", width)) @@ -148,6 +156,12 @@ func outputStructuralDiff(ctx *ExecContext, result DiffResult, useColor bool) { } else { fmt.Fprintln(ctx.Output, " + New") } + } else if result.IsDeleted { + if useColor { + fmt.Fprintf(ctx.Output, " %s- Removed%s\n", colorRed, colorReset) + } else { + fmt.Fprintln(ctx.Output, " - Removed") + } } else if result.Current == result.Proposed { fmt.Fprintln(ctx.Output, " (no changes)") } else if len(result.Changes) == 0 { diff --git a/mdl/executor/cmd_diff_render.go b/mdl/executor/cmd_diff_render.go deleted file mode 100644 index 94a21e4abf..0000000000 --- a/mdl/executor/cmd_diff_render.go +++ /dev/null @@ -1,84 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 - -// Package executor - rendering both sides of a diff through one describer. -package executor - -import ( - "github.com/mendixlabs/mxcli/mdl/ast" - mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" - "github.com/mendixlabs/mxcli/model" - "github.com/mendixlabs/mxcli/sdk/microflows" -) - -// flowNameMaps builds the ID → qualified-name maps renderMicroflowMDL needs to -// print entity and flow references. Shared so that the two sides of a diff -// resolve names identically; a map built for one side only would show a -// reference as a name on one side and a stub on the other. -func flowNameMaps(ctx *ExecContext) (entityNames, microflowNames map[model.ID]string, err error) { - h, err := getHierarchy(ctx) - if err != nil { - return nil, nil, mdlerrors.NewBackend("build hierarchy", err) - } - - entityNames = make(map[model.ID]string) - domainModels, _ := ctx.Backend.ListDomainModels() - for _, dm := range domainModels { - modName := h.GetModuleName(dm.ContainerID) - for _, entity := range dm.Entities { - entityNames[entity.ID] = modName + "." + entity.Name - } - } - - microflowNames = make(map[model.ID]string) - allMicroflows, err := ctx.Backend.ListMicroflows() - if err != nil { - return nil, nil, mdlerrors.NewBackend("list microflows", err) - } - for _, mf := range allMicroflows { - microflowNames[mf.ID] = h.GetQualifiedName(mf.ContainerID, mf.Name) - } - allNanoflows, _ := ctx.Backend.ListNanoflows() - for _, nf := range allNanoflows { - microflowNames[nf.ID] = h.GetQualifiedName(nf.ContainerID, nf.Name) - } - return entityNames, microflowNames, nil -} - -// renderFlowFromModel renders an in-memory flow as MDL through the same -// describer DESCRIBE and diff-local use. -// -// This is the whole point of the #997 fix. diff used to render its script side -// with a second AST-to-MDL renderer, which covered 18 of 43 activity types and -// silently emitted nothing for the rest — so a java-action call, a `download -// file` or a canvas annotation appeared in the diff as a deletion, and mxcli -// confidently reported that a script would gut a microflow that exec proved -// was a no-op. One renderer for both sides makes that class of false report -// unrepresentable rather than fixed case by case. -func renderFlowFromModel(ctx *ExecContext, flowType string, mf *microflows.Microflow, name ast.QualifiedName) (string, error) { - entityNames, microflowNames, err := flowNameMaps(ctx) - if err != nil { - return "", err - } - return renderMicroflowMDL(ctx, flowType, mf, name, entityNames, microflowNames, nil), nil -} - -// nanoflowAsMicroflow wraps a Nanoflow so renderMicroflowMDL can print it. -// ContainerID is carried so the `folder` line matches the microflow path; -// without it one side of a nanoflow diff would print a folder and the other -// would not. -func nanoflowAsMicroflow(nf *microflows.Nanoflow) *microflows.Microflow { - if nf == nil { - return nil - } - return µflows.Microflow{ - BaseElement: nf.BaseElement, - ContainerID: nf.ContainerID, - Name: nf.Name, - Documentation: nf.Documentation, - Excluded: nf.Excluded, - Parameters: nf.Parameters, - ReturnType: nf.ReturnType, - ObjectCollection: nf.ObjectCollection, - AllowedModuleRoles: nf.AllowedModuleRoles, - } -} diff --git a/mdl/executor/cmd_diff_render_test.go b/mdl/executor/cmd_diff_render_test.go deleted file mode 100644 index 1264101ffb..0000000000 --- a/mdl/executor/cmd_diff_render_test.go +++ /dev/null @@ -1,87 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 - -package executor - -import ( - "errors" - "os" - "strings" - "testing" - - "github.com/mendixlabs/mxcli/mdl/ast" - "github.com/mendixlabs/mxcli/sdk/microflows" -) - -// #997: diff rendered its script side with a second AST-to-MDL renderer whose -// statement switch had no default case, so an activity it did not know emitted -// zero lines and showed up in the diff as a deletion. mxcli reported that a -// script would gut a microflow that exec proved was a no-op. -// -// The fix is structural — one renderer for both sides — so the regression test -// that matters is that the dead renderer stays dead. A reviewer adding a case -// to a revived script-side renderer would re-create the drift; this fails -// first and says why. -func TestDiffHasNoSecondFlowRenderer(t *testing.T) { - for _, name := range []string{ - "microflowStmtToMDL", - "nanoflowStmtToMDL", - "microflowStatementToMDL", - "diffExpressionToString", - } { - if diffMDLSource(t, name) { - t.Errorf("%s is back in cmd_diff_mdl.go. diff must render its script side "+ - "through renderMicroflowMDL — the describer both `describe` and `diff-local` "+ - "use — so that an activity it cannot print is impossible rather than shown "+ - "as a deletion (#997).", name) - } - } -} - -// diffMDLSource reports whether cmd_diff_mdl.go still defines fn. -func diffMDLSource(t *testing.T, fn string) bool { - t.Helper() - b, err := os.ReadFile("cmd_diff_mdl.go") - if err != nil { - t.Fatalf("read cmd_diff_mdl.go: %v", err) - } - return strings.Contains(string(b), "func "+fn+"(") -} - -// nanoflowAsMicroflow must carry ContainerID: renderMicroflowMDL prints the -// `folder` line from it, so dropping it would make one side of a nanoflow diff -// print a folder and the other not — a false modification of exactly the kind -// this issue was about. -func TestNanoflowWrapperCarriesContainer(t *testing.T) { - nf := µflows.Nanoflow{Name: "NF", ContainerID: "folder-id"} - got := nanoflowAsMicroflow(nf) - if got.ContainerID != "folder-id" { - t.Errorf("ContainerID = %q, want folder-id — the folder line would differ between sides", got.ContainerID) - } - if nanoflowAsMicroflow(nil) != nil { - t.Error("nil nanoflow should wrap to nil") - } -} - -// A statement diff cannot compare must be reported, not dropped. Silently -// skipping made `diff` print "0 new, 0 modified, 0 unchanged" for a script that -// would genuinely add documents — worse than a wrong count, because there is -// nothing on screen to disbelieve. -func TestUnsupportedStatementIsAnError(t *testing.T) { - _, err := diffStatement(nil, &ast.CreateConstantStmt{}) - if err == nil { - t.Fatal("an unsupported statement returned no error — it would vanish from the summary") - } - var unsupported *unsupportedDiffError - if !errors.As(err, &unsupported) { - t.Fatalf("error = %T, want *unsupportedDiffError", err) - } - if !strings.Contains(unsupported.kind, "constant") { - t.Errorf("kind = %q, want it to name the statement", unsupported.kind) - } -} - -func TestStatementKindName(t *testing.T) { - if got := statementKindName(&ast.CreateConstantStmt{}); got != "create constant" { - t.Errorf("got %q, want \"create constant\"", got) - } -} diff --git a/mdl/executor/cmd_enumerations.go b/mdl/executor/cmd_enumerations.go index f01068889a..58b68da66a 100644 --- a/mdl/executor/cmd_enumerations.go +++ b/mdl/executor/cmd_enumerations.go @@ -447,14 +447,16 @@ func describeEnumeration(ctx *ExecContext, name ast.QualifiedName) error { comma = "" } caption := pickTextTranslation(v.Caption, lang) - fmt.Fprintf(ctx.Output, " %s '%s'%s\n", v.Name, caption, comma) + // mdlQuote doubles an apostrophe (mendixlabs/mxcli#394): a + // caption like "Won't fix" must re-parse. + fmt.Fprintf(ctx.Output, " %s %s%s\n", v.Name, mdlQuote(ctx, caption), comma) } // Emit the module folder so a moved enumeration round-trips (Bug 12b). // BuildFolderPath returns the module name at the module root and the // folder path when the enum lives in a folder. folderClause := "" if fp := h.BuildFolderPath(enum.ContainerID); fp != "" && fp != modName { - folderClause = fmt.Sprintf(" folder '%s'", fp) + folderClause = " folder " + mdlQuote(ctx, fp) } fmt.Fprintf(ctx.Output, ")%s;\n", folderClause) return nil diff --git a/mdl/executor/cmd_enumerations_caption_quote_test.go b/mdl/executor/cmd_enumerations_caption_quote_test.go new file mode 100644 index 0000000000..d469a4898e --- /dev/null +++ b/mdl/executor/cmd_enumerations_caption_quote_test.go @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/langver" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" +) + +// mendixlabs/mxcli#394: DESCRIBE ENUMERATION wrote a caption with an apostrophe +// as `'It's a test'`, which does not re-parse — the round trip of any caption a +// person would type ("Won't fix") was broken. The output must re-parse, under +// both describe languages, to the same caption; a folder path with an +// apostrophe is the same emit. +func TestDescribeEnumeration_CaptionApostropheRoundTrips(t *testing.T) { + const caption = "It's a test" + for _, lang := range []langver.Version{langver.V0, langver.V1} { + mod := mkModule("MyFirstModule") + folderID := nextID("folder") + enum := &model.Enumeration{ + BaseElement: model.BaseElement{ID: nextID("enum")}, + ContainerID: folderID, + Name: "TestEnum", + Values: []model.EnumerationValue{ + {BaseElement: model.BaseElement{ID: nextID("ev")}, Name: "Value1", + Caption: &model.Text{Translations: map[string]string{"en_US": caption}}}, + {BaseElement: model.BaseElement{ID: nextID("ev")}, Name: "Value2", + Caption: &model.Text{Translations: map[string]string{"en_US": "plain"}}}, + }, + } + h := mkHierarchy(mod) + withContainer(h, folderID, mod.ID) + h.folderNames[folderID] = "Bob's" + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListEnumerationsFunc: func() ([]*model.Enumeration, error) { return []*model.Enumeration{enum}, nil }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + ctx.describeLang = &lang + assertNoError(t, describeEnumeration(ctx, ast.QualifiedName{Module: "MyFirstModule", Name: "TestEnum"})) + out := buf.String() + + src := out + if lang >= langver.V1 { + src = "mdl 1;\n" + out + } + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("%v: describe output does not re-parse: %v\n%s", lang, errs, out) + } + var stmt *ast.CreateEnumerationStmt + for _, s := range prog.Statements { + if c, ok := s.(*ast.CreateEnumerationStmt); ok { + stmt = c + } + } + if stmt == nil || len(stmt.Values) != 2 { + t.Fatalf("%v: re-parsed to %#v\n%s", lang, prog.Statements, out) + } + if stmt.Values[0].Caption != caption { + t.Errorf("%v: caption re-parses as %q, want %q\n%s", lang, stmt.Values[0].Caption, caption, out) + } + if !strings.Contains(out, "Bob''s") { + t.Errorf("%v: folder path apostrophe not doubled:\n%s", lang, out) + } + } +} diff --git a/mdl/executor/cmd_microflows_builder_actions.go b/mdl/executor/cmd_microflows_builder_actions.go index 24dd44f0d0..f0ca35929e 100644 --- a/mdl/executor/cmd_microflows_builder_actions.go +++ b/mdl/executor/cmd_microflows_builder_actions.go @@ -1060,9 +1060,10 @@ func qualifiedNameString(qn ast.QualifiedName) string { func (fb *flowBuilder) addCastAction(s *ast.CastObjectStmt) model.ID { action := µflows.CastAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - ObjectVariable: s.ObjectVariable, - OutputVariable: s.OutputVariable, + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + ObjectVariable: s.ObjectVariable, + OutputVariable: s.OutputVariable, + ErrorHandlingType: fb.ehType(nil), } activity := µflows.ActionActivity{ @@ -1774,9 +1775,10 @@ func (fb *flowBuilder) addListOperationAction(s *ast.ListOperationStmt) model.ID } action := µflows.ListOperationAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - Operation: operation, - OutputVariable: s.OutputVariable, + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + Operation: operation, + OutputVariable: s.OutputVariable, + ErrorHandlingType: fb.ehType(nil), } // Track output variable type for operations that preserve/produce list types @@ -1902,10 +1904,11 @@ func (fb *flowBuilder) addAggregateListAction(s *ast.AggregateListStmt) model.ID } action := µflows.AggregateListAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - InputVariable: s.InputVariable, - OutputVariable: s.OutputVariable, - Function: function, + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + InputVariable: s.InputVariable, + OutputVariable: s.OutputVariable, + Function: function, + ErrorHandlingType: fb.ehType(nil), } // The fold Mendix stores beside the expression. REDUCE names both in MDL; @@ -1968,6 +1971,7 @@ func (fb *flowBuilder) addCreateListAction(s *ast.CreateListStmt) model.ID { BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, OutputVariable: s.Variable, EntityQualifiedName: entityQN, + ErrorHandlingType: fb.ehType(nil), } // Register variable type as list @@ -1999,10 +2003,11 @@ func (fb *flowBuilder) addAddToListAction(s *ast.AddToListStmt) model.ID { value = "$" + s.Item } action := µflows.ChangeListAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - Type: microflows.ChangeListTypeAdd, - ChangeVariable: s.List, - Value: value, + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + Type: microflows.ChangeListTypeAdd, + ChangeVariable: s.List, + Value: value, + ErrorHandlingType: fb.ehType(nil), } activity := µflows.ActionActivity{ @@ -2025,10 +2030,11 @@ func (fb *flowBuilder) addAddToListAction(s *ast.AddToListStmt) model.ID { // addRemoveFromListAction creates a REMOVE FROM list statement. func (fb *flowBuilder) addRemoveFromListAction(s *ast.RemoveFromListStmt) model.ID { action := µflows.ChangeListAction{ - BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - Type: microflows.ChangeListTypeRemove, - ChangeVariable: s.List, - Value: "$" + s.Item, + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + Type: microflows.ChangeListTypeRemove, + ChangeVariable: s.List, + Value: "$" + s.Item, + ErrorHandlingType: fb.ehType(nil), } activity := µflows.ActionActivity{ diff --git a/mdl/executor/cmd_microflows_builder_calls.go b/mdl/executor/cmd_microflows_builder_calls.go index f67d06ed7d..30ce92311e 100644 --- a/mdl/executor/cmd_microflows_builder_calls.go +++ b/mdl/executor/cmd_microflows_builder_calls.go @@ -373,6 +373,12 @@ func (fb *flowBuilder) addCallJavaActionAction(s *ast.CallJavaActionStmt) model. // Regular parameter: expression-based value valueExpr := fb.exprToString(arg.Value) if microflowTypeParams[arg.Name] { + // A Microflow parameter takes a name, never an expression + // (mendixlabs/mxcli#1210); the same decision reference + // validation reports. + if msg := microflowParamArgRefusal(actionQN, arg.Name, arg.Value); msg != "" { + fb.addError("%s", msg) + } value = µflows.MicroflowParameterValue{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, Microflow: strings.Trim(valueExpr, "'"), diff --git a/mdl/executor/cmd_microflows_builder_flows.go b/mdl/executor/cmd_microflows_builder_flows.go index 84e8e04589..7799b7cbd7 100644 --- a/mdl/executor/cmd_microflows_builder_flows.go +++ b/mdl/executor/cmd_microflows_builder_flows.go @@ -759,7 +759,14 @@ func (fb *flowBuilder) addErrorHandlerFlow(sourceActivityID model.ID, sourceX in var lastErrID model.ID var lastErrCase string var lastErrAnchor *ast.FlowAnchors + // prevOwnAnchor is the @anchor of the handler statement before this one, + // whose `from:` is the side the edge between them leaves. The loop below + // built its edges with the default sides and never applied either end, so + // an @anchor inside a handler parsed, passed check and exec, and did + // nothing (mendixlabs/mxcli#991). + var prevOwnAnchor *ast.FlowAnchors for _, stmt := range errorBody { + thisAnchor := stmtOwnAnchor(stmt) actID := errBuilder.addStatement(stmt) if errBuilder.pendingJoin != nil { // A handler whose FIRST statement is a join has no activity of its @@ -782,11 +789,18 @@ func (fb *flowBuilder) addErrorHandlerFlow(sourceActivityID model.ID, sourceX in if actID != "" { errBuilder.applyPendingAnnotations(actID) if lastErrID == "" { - // Connect source activity to first error handler activity - fb.flows = append(fb.flows, newErrorHandlerFlow(sourceActivityID, actID)) + // Connect source activity to first error handler activity. The + // statement's `to:` is the side its incoming edge — this one — + // enters; the side the edge leaves the source has no spelling. + flow := newErrorHandlerFlow(sourceActivityID, actID) + applyUserAnchors(flow, nil, thisAnchor) + fb.flows = append(fb.flows, flow) } else { - errBuilder.flows = append(errBuilder.flows, newHorizontalFlow(lastErrID, actID)) + flow := newHorizontalFlow(lastErrID, actID) + applyUserAnchors(flow, prevOwnAnchor, thisAnchor) + errBuilder.flows = append(errBuilder.flows, flow) } + prevOwnAnchor = thisAnchor if errBuilder.nextConnectionPoint != "" { lastErrID = errBuilder.nextConnectionPoint lastErrCase = errBuilder.nextFlowCase @@ -794,10 +808,15 @@ func (fb *flowBuilder) addErrorHandlerFlow(sourceActivityID model.ID, sourceX in errBuilder.nextConnectionPoint = "" errBuilder.nextFlowCase = "" errBuilder.nextFlowAnchor = nil + // A compound statement steers its own exits. + prevOwnAnchor = nil } else { lastErrID = actID lastErrCase = "" - lastErrAnchor = nil + // The edge that rejoins the main flow leaves this statement, so + // it takes the statement's `from:` — and only that: its `to:` + // belongs to the edge coming in. + lastErrAnchor = originOnly(thisAnchor) } } } @@ -812,6 +831,16 @@ func (fb *flowBuilder) addErrorHandlerFlow(sourceActivityID model.ID, sourceX in fb.flows = append(fb.flows, errBuilder.flows...) fb.annotationFlows = append(fb.annotationFlows, errBuilder.annotationFlows...) fb.errors = append(fb.errors, errBuilder.errors...) + // A @curve inside the handler is recorded on errBuilder, and applied by the + // builder that owns the flows once its graph is complete — this one, since + // the handler's flows (and the edge that rejoins the main flow) land here. + // Left on errBuilder it was never applied (mendixlabs/mxcli#991). + for id, c := range errBuilder.curveByOrigin { + if fb.curveByOrigin == nil { + fb.curveByOrigin = map[model.ID]*ast.FlowCurve{} + } + fb.curveByOrigin[id] = c + } if fb.annotationsByLabel == nil { fb.annotationsByLabel = errBuilder.annotationsByLabel } @@ -971,6 +1000,15 @@ func applyUserAnchors(flow *microflows.SequenceFlow, origin *ast.FlowAnchors, de } } +// originOnly is an anchor's `from:` alone, for an edge whose destination the +// statement does not own. +func originOnly(a *ast.FlowAnchors) *ast.FlowAnchors { + if a == nil || a.From == ast.AnchorSideUnset { + return nil + } + return &ast.FlowAnchors{From: a.From, To: ast.AnchorSideUnset} +} + func branchDestinationAnchor(branchAnchor, stmtAnchor *ast.FlowAnchors) *ast.FlowAnchors { // The split branch annotation owns the incoming edge to the first branch // activity. If it specifies `to`, it must win over the first statement's diff --git a/mdl/executor/cmd_microflows_builder_workflow.go b/mdl/executor/cmd_microflows_builder_workflow.go index b5b32bba49..76294cdedc 100644 --- a/mdl/executor/cmd_microflows_builder_workflow.go +++ b/mdl/executor/cmd_microflows_builder_workflow.go @@ -210,6 +210,7 @@ func (fb *flowBuilder) addLockWorkflowAction(s *ast.LockWorkflowStmt) model.ID { ErrorHandlingType: convertErrorHandlingType(s.ErrorHandling), PauseAllWorkflows: s.PauseAllWorkflows, WorkflowVariable: s.WorkflowVariable, + Workflow: s.Workflow, } return fb.wrapAction(action, s.ErrorHandling) } @@ -220,6 +221,7 @@ func (fb *flowBuilder) addUnlockWorkflowAction(s *ast.UnlockWorkflowStmt) model. ErrorHandlingType: convertErrorHandlingType(s.ErrorHandling), ResumeAllPausedWorkflows: s.ResumeAllPausedWorkflows, WorkflowVariable: s.WorkflowVariable, + Workflow: s.Workflow, } return fb.wrapAction(action, s.ErrorHandling) } diff --git a/mdl/executor/cmd_microflows_format_action.go b/mdl/executor/cmd_microflows_format_action.go index 559b10e39d..d09161ca24 100644 --- a/mdl/executor/cmd_microflows_format_action.go +++ b/mdl/executor/cmd_microflows_format_action.go @@ -1003,22 +1003,10 @@ func formatAction( return fmt.Sprintf("open workflow $%s;", a.WorkflowVariable) case *microflows.LockWorkflowAction: - if a.PauseAllWorkflows { - return "lock workflow all;" - } - if a.Workflow != "" { - return fmt.Sprintf("lock workflow %s;", a.Workflow) - } - return fmt.Sprintf("lock workflow $%s;", a.WorkflowVariable) + return "lock workflow " + workflowSelectionMDL(a.Workflow, a.WorkflowVariable, a.PauseAllWorkflows, "pause all") + ";" case *microflows.UnlockWorkflowAction: - if a.ResumeAllPausedWorkflows { - return "unlock workflow all;" - } - if a.Workflow != "" { - return fmt.Sprintf("unlock workflow %s;", a.Workflow) - } - return fmt.Sprintf("unlock workflow $%s;", a.WorkflowVariable) + return "unlock workflow " + workflowSelectionMDL(a.Workflow, a.WorkflowVariable, a.ResumeAllPausedWorkflows, "unpause all") + ";" case *microflows.JavaScriptActionCallAction: jsActionName := a.JavaScriptAction @@ -2206,3 +2194,25 @@ func templateArgsClause(ctx *ExecContext, args []string) string { } return " with (" + strings.Join(parts, ", ") + ")" } + +// workflowSelectionMDL spells a lock/unlock's workflow and its instances flag. +// The flag is Studio Pro's "Pause instances" / "Unpause instances" on the +// selected workflow, and was printed as a bare `all` in place of the workflow +// — a statement that dropped the selection and rebuilt as CE1825 +// (mendixlabs/mxcli#870). A stored activity with the flag and no selection +// still prints as `all`, which check and exec refuse: it has no buildable form. +func workflowSelectionMDL(workflow, variable string, flag bool, flagWords string) string { + var sel string + switch { + case workflow != "": + sel = workflow + case variable != "": + sel = "$" + variable + default: + return "all" + } + if flag { + sel += " " + flagWords + } + return sel +} diff --git a/mdl/executor/cmd_microflows_show_helpers.go b/mdl/executor/cmd_microflows_show_helpers.go index 839dfe7775..e9bc1db396 100644 --- a/mdl/executor/cmd_microflows_show_helpers.go +++ b/mdl/executor/cmd_microflows_show_helpers.go @@ -337,15 +337,22 @@ func emitAnchorAnnotationWithActivityMap( break } } + defaultTo := anchorSideKeyword(AnchorLeft) if incoming := flowsByDest[id]; len(incoming) > 0 { to = anchorSideKeyword(incoming[0].DestinationConnectionIndex) + // An error handler's first activity is entered by the error edge, which + // the builder draws into its TOP (newErrorHandlerFlow). Judged against + // the left side, `to: left` there read as the default and was left out, + // and the rebuild entered the top (mendixlabs/mxcli#991). + if len(incoming) == 1 && incoming[0].IsErrorHandler { + defaultTo = anchorSideKeyword(AnchorTop) + } } if from == "" && to == "" { return } defaultFrom := anchorSideKeyword(AnchorRight) - defaultTo := anchorSideKeyword(AnchorLeft) var parts []string if from != "" && from != defaultFrom { parts = append(parts, "from: "+from) @@ -2491,13 +2498,21 @@ func collectErrorHandlerStatementSpans( visited := make(map[model.ID]bool) stopID := firstReachableErrorHandlerMerge(startID, activityMap, flowsByOrigin) - // A note on a handler-body activity is emitted here or nowhere: this + // A handler-body activity's annotations are emitted here or nowhere: this // traversal is a second, smaller describer and the main one never reaches - // inside an `on error begin … end error` block. Without it the write path attaches the - // note and the read path drops it, which is the same round-trip loss #1077 - // is about, one nesting level down. + // inside an `on error begin … end error` block. It used to emit the notes + // alone (#1077) — so @position, @anchor, @curve, @caption, @color and + // @excluded, all of which the builder honours in a handler, came back + // missing and describe → exec put a laid-out handler back on + // auto-placement (mendixlabs/mxcli#991). Same emitter as the main path. + flowsByDest := map[model.ID][]*microflows.SequenceFlow{} + for _, flows := range flowsByOrigin { + for _, f := range flows { + flowsByDest[f.DestinationID] = append(flowsByDest[f.DestinationID], f) + } + } notes := func(obj microflows.MicroflowObject, indentStr string) { - statements = append(statements, annotationsByTarget.lines(obj.GetID(), obj.GetPosition(), objectHeight(obj), indentStr)...) + emitObjectAnnotations(obj, &statements, indentStr, annotationsByTarget, flowsByOrigin, flowsByDest, activityMap) } splitMergeMap := findErrorHandlerSplitMergePoints(ctx, activityMap, flowsByOrigin) diff --git a/mdl/executor/cmd_microflows_traverse_test.go b/mdl/executor/cmd_microflows_traverse_test.go index 70c303dad4..b0d326542a 100644 --- a/mdl/executor/cmd_microflows_traverse_test.go +++ b/mdl/executor/cmd_microflows_traverse_test.go @@ -1056,7 +1056,9 @@ func TestCollectErrorHandlerStatements_Simple(t *testing.T) { mkID("err_log"): {mkFlow("err_log", "err_end")}, } - stmts := e.collectErrorHandlerStatements(mkID("err_log"), activityMap, flowsByOrigin, nil, nil, nil) + // Annotation lines are left out: a handler-body statement now carries its + // @position/@anchor like any other (mendixlabs/mxcli#991). + stmts := statementLines(e.collectErrorHandlerStatements(mkID("err_log"), activityMap, flowsByOrigin, nil, nil, nil)) if len(stmts) != 2 { t.Fatalf("expected 2 statements, got %d: %v", len(stmts), stmts) } @@ -1085,7 +1087,7 @@ func TestCollectErrorHandlerStatements_StopsAtMerge(t *testing.T) { mkID("merge"): {mkFlow("merge", "after")}, } - stmts := e.collectErrorHandlerStatements(mkID("err_log"), activityMap, flowsByOrigin, nil, nil, nil) + stmts := statementLines(e.collectErrorHandlerStatements(mkID("err_log"), activityMap, flowsByOrigin, nil, nil, nil)) // Should stop at merge, not include "after" if len(stmts) != 1 { t.Fatalf("expected 1 statement (stop at merge), got %d: %v", len(stmts), stmts) @@ -1587,3 +1589,14 @@ func TestTraverseFlow_InheritanceSplitOmitsEmptyElse(t *testing.T) { t.Errorf("an (empty) branch WITH a body must still render:\n%s", out) } } + +// statementLines drops the @annotation lines from described statements. +func statementLines(lines []string) []string { + var out []string + for _, l := range lines { + if !strings.HasPrefix(strings.TrimSpace(l), "@") { + out = append(out, l) + } + } + return out +} diff --git a/mdl/executor/cmd_misc.go b/mdl/executor/cmd_misc.go index c9f7a5f1e0..01168c2d9b 100644 --- a/mdl/executor/cmd_misc.go +++ b/mdl/executor/cmd_misc.go @@ -12,6 +12,7 @@ import ( "github.com/mendixlabs/mxcli/cmd/mxcli/syntax" "github.com/mendixlabs/mxcli/mdl/ast" mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/srctext" "github.com/mendixlabs/mxcli/mdl/visitor" ) @@ -402,7 +403,13 @@ func execExecuteScript(ctx *ExecContext, s *ast.ExecuteScriptStmt) error { } // Read the script file - content, readErr := os.ReadFile(scriptPath) + raw, readErr := os.ReadFile(scriptPath) + if readErr != nil { + return mdlerrors.NewBackend("read script file '"+s.Path+"'", readErr) + } + // A BOM or UTF-16 is decoded as for a script given to check/exec + // (mendixlabs/mxcli#1253). + content, readErr := srctext.Decode(raw) if readErr != nil { return mdlerrors.NewBackend("read script file '"+s.Path+"'", readErr) } diff --git a/mdl/executor/cmd_move.go b/mdl/executor/cmd_move.go index 79fac38e33..7778fbdeb0 100644 --- a/mdl/executor/cmd_move.go +++ b/mdl/executor/cmd_move.go @@ -410,9 +410,25 @@ func moveEntity(ctx *ExecContext, name ast.QualifiedName, sourceModule, targetMo } fmt.Fprintf(ctx.Output, "Moved entity %s to %s\n", name.String(), targetModule.Name) - if len(convertedAssocs) > 0 { - fmt.Fprintf(ctx.Output, "Converted %d association(s) to cross-module associations:\n", len(convertedAssocs)) - for _, assocName := range types.MovedAssociationNames(convertedAssocs) { + // A move that brings both endpoints of a cross-association into one module turns + // it back into a plain association (ako/mxcli#628); report the two apart. + var toCross, toPlain []types.MovedAssociation + for _, a := range convertedAssocs { + if a.SameModule { + toPlain = append(toPlain, a) + } else { + toCross = append(toCross, a) + } + } + if len(toCross) > 0 { + fmt.Fprintf(ctx.Output, "Converted %d association(s) to cross-module associations:\n", len(toCross)) + for _, assocName := range types.MovedAssociationNames(toCross) { + fmt.Fprintf(ctx.Output, " - %s\n", assocName) + } + } + if len(toPlain) > 0 { + fmt.Fprintf(ctx.Output, "Converted %d cross-module association(s) to associations within %s:\n", len(toPlain), targetModule.Name) + for _, assocName := range types.MovedAssociationNames(toPlain) { fmt.Fprintf(ctx.Output, " - %s\n", assocName) } } diff --git a/mdl/executor/cmd_pages_builder_v3.go b/mdl/executor/cmd_pages_builder_v3.go index 7aef506c43..0048ab4d00 100644 --- a/mdl/executor/cmd_pages_builder_v3.go +++ b/mdl/executor/cmd_pages_builder_v3.go @@ -480,6 +480,12 @@ func (pb *pageBuilder) buildWidgetV3(w *ast.WidgetV3) (pages.Widget, error) { } } } + // A built-in Forms widget mxcli has no writer for: say so, rather than + // send the reader to `widget init` (ako/mxcli#563). Same message as + // MDL-WIDGET38, which refuses it before anything is written. + if msg := unbuildableWidgetMessage(w.Type); msg != "" { + return nil, mdlerrors.NewUnsupported(msg) + } if pb.pluggableEngineErr != nil { return nil, mdlerrors.NewUnsupported(fmt.Sprintf("unsupported widget type: %s (%v)", w.Type, pb.pluggableEngineErr)) } diff --git a/mdl/executor/cmd_pages_builder_v3_widgets.go b/mdl/executor/cmd_pages_builder_v3_widgets.go index 8224bd07c3..728b9eaba0 100644 --- a/mdl/executor/cmd_pages_builder_v3_widgets.go +++ b/mdl/executor/cmd_pages_builder_v3_widgets.go @@ -465,6 +465,9 @@ func (pb *pageBuilder) buildTextBoxV3(w *ast.WidgetV3) (*pages.TextBox, error) { } tb.AttributePath, tb.AttributeRefSteps, tb.SourceVariable = path, steps, sv } + if sv := pb.pageVariableInputBinding(w); sv != nil { + tb.SourceVariable = sv + } if err := pb.checkInputBinding(w, pb.entityContext); err != nil { return nil, err } @@ -532,6 +535,9 @@ func (pb *pageBuilder) buildTextAreaV3(w *ast.WidgetV3) (*pages.TextArea, error) } ta.AttributePath, ta.AttributeRefSteps, ta.SourceVariable = path, steps, sv } + if sv := pb.pageVariableInputBinding(w); sv != nil { + ta.SourceVariable = sv + } if err := pb.checkInputBinding(w, pb.entityContext); err != nil { return nil, err } @@ -583,6 +589,9 @@ func (pb *pageBuilder) buildDatePickerV3(w *ast.WidgetV3) (*pages.DatePicker, er } dp.AttributePath, dp.AttributeRefSteps, dp.SourceVariable = path, steps, sv } + if sv := pb.pageVariableInputBinding(w); sv != nil { + dp.SourceVariable = sv + } if err := pb.checkInputBinding(w, pb.entityContext); err != nil { return nil, err } @@ -623,6 +632,9 @@ func (pb *pageBuilder) buildDropdownV3(w *ast.WidgetV3) (*pages.DropDown, error) } dd.AttributePath, dd.AttributeRefSteps, dd.SourceVariable = path, steps, sv } + if sv := pb.pageVariableInputBinding(w); sv != nil { + dd.SourceVariable = sv + } if err := pb.checkInputBinding(w, pb.entityContext); err != nil { return nil, err } @@ -663,6 +675,9 @@ func (pb *pageBuilder) buildCheckBoxV3(w *ast.WidgetV3) (*pages.CheckBox, error) } cb.AttributePath, cb.AttributeRefSteps, cb.SourceVariable = path, steps, sv } + if sv := pb.pageVariableInputBinding(w); sv != nil { + cb.SourceVariable = sv + } if err := pb.checkInputBinding(w, pb.entityContext); err != nil { return nil, err } @@ -738,6 +753,9 @@ func (pb *pageBuilder) buildRadioButtonsV3(w *ast.WidgetV3) (*pages.RadioButtons } rb.AttributePath, rb.AttributeRefSteps, rb.SourceVariable = path, steps, sv } + if sv := pb.pageVariableInputBinding(w); sv != nil { + rb.SourceVariable = sv + } if err := pb.checkInputBinding(w, pb.entityContext); err != nil { return nil, err } diff --git a/mdl/executor/cmd_pages_describe_output.go b/mdl/executor/cmd_pages_describe_output.go index 4dd2d09e72..058d522379 100644 --- a/mdl/executor/cmd_pages_describe_output.go +++ b/mdl/executor/cmd_pages_describe_output.go @@ -1073,7 +1073,11 @@ func outputWidgetMDLV3(ctx *ExecContext, w rawWidget, indent int) { fmt.Fprintf(ctx.Output, "%s}\n", prefix) case "Footer": - fmt.Fprintf(ctx.Output, "%sfooter %s {\n", prefix, mdlIdent(w.Name)) + if w.Name == "" { + fmt.Fprintf(ctx.Output, "%sfooter {\n", prefix) + } else { + fmt.Fprintf(ctx.Output, "%sfooter %s {\n", prefix, mdlIdent(w.Name)) + } for _, child := range w.Children { outputWidgetMDLV3(ctx, child, indent+1) } diff --git a/mdl/executor/cmd_pages_describe_parse.go b/mdl/executor/cmd_pages_describe_parse.go index 38213ff7a6..7c2547e398 100644 --- a/mdl/executor/cmd_pages_describe_parse.go +++ b/mdl/executor/cmd_pages_describe_parse.go @@ -905,8 +905,10 @@ func parseDataViewChildren(ctx *ExecContext, w map[string]any, entityContext ... // Get footer widgets footerWidgets := getBsonArrayElements(w["FooterWidgets"]) if len(footerWidgets) > 0 { - // Create a special footer container with synthetic name - footer := rawWidget{Type: "Footer", Name: "footer1"} + // The footer is a region of the data view, stored without a name: it is + // printed unnamed and addressed as `.footer` (ako/mxcli#528). + // The `footer1` printed here before named nothing ALTER could find. + footer := rawWidget{Type: "Footer"} for _, child := range footerWidgets { if childMap, ok := child.(map[string]any); ok { footer.Children = append(footer.Children, parseRawWidget(ctx, childMap, entCtx)...) @@ -1203,10 +1205,18 @@ func extractSnippetCallParams(w map[string]any) string { // `$dataView1.FullName`, which the builder resolves back to the same pair; // describe used to print the bare attribute and exec dropped the variable // (ako/mxcli#826). Every other binding reads as before. +// +// An input bound directly to a page variable stores no AttributeRef and names +// the variable in SourceVariable.LocalVariable; it prints as `$ShowAll`, the +// spelling the builder takes. describe used to print it unbound, so a round +// trip silently cut the binding (mendixlabs/mxcli#1235). func extractInputAttribute(ctx *ExecContext, w map[string]any) string { sv, _ := w["SourceVariable"].(map[string]any) widget := extractString(sv["Widget"]) attrRef, _ := w["AttributeRef"].(map[string]any) + if local := extractString(sv["LocalVariable"]); local != "" && widget == "" && attrRef == nil { + return "$" + local + } if widget == "" || attrRef == nil { return extractAttributeRef(ctx, w) } diff --git a/mdl/executor/cmd_pages_input_binding_context.go b/mdl/executor/cmd_pages_input_binding_context.go index b16b114b50..6fad9253f8 100644 --- a/mdl/executor/cmd_pages_input_binding_context.go +++ b/mdl/executor/cmd_pages_input_binding_context.go @@ -9,6 +9,7 @@ import ( "github.com/mendixlabs/mxcli/mdl/ast" mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/sdk/pages" ) // An `Attribute:` binding is only storable when there is an object to bind to. @@ -45,14 +46,30 @@ import ( // c is the context the widget sits in. noEntity says that nothing could qualify // a bare name here — the builder knows that; the check-time walk has no project // to ask, passes false, and lets the context alone decide. -func inputBindingProblem(w *ast.WidgetV3, c pageArgContext, noEntity bool) string { +// +// isPageVariable answers whether a bare `$name` is one of the document's page +// variables, which an input binds to directly (mendixlabs/mxcli#1235); nil +// means the caller cannot know (an ALTER fragment at check time), and the +// builder, which can, decides. +func inputBindingProblem(w *ast.WidgetV3, c pageArgContext, noEntity bool, isPageVariable func(string) bool) string { raw, present := lookupPropCI(w, "Attribute") if !present || raw == nil { return "" } kind := strings.ToLower(w.Type) + if name, ok := bareVariableReference(raw); ok { + if isPageVariable == nil || isPageVariable(name) { + return "" + } + } attr, isString := raw.(string) if !isString { + if name, ok := bareVariableReference(raw); ok { + return fmt.Sprintf("%s `%s`: `Attribute: $%s` — `$%s` is not a page variable of this document, so the "+ + "widget would be written with no binding at all. Declare it (`Variables: ( $%s: Boolean = 'true' )`) to "+ + "bind the input to it, or bind an attribute by name inside a data container", + kind, w.Name, name, name, name) + } return fmt.Sprintf("%s `%s`: `Attribute: %s` is not an attribute binding MDL can store — the widget "+ "would be written with no binding at all, inside a data view or outside one. Bind the attribute by "+ "name inside a data container over that object: `dataview dv (DataSource: $Param) { %s %s "+ @@ -117,7 +134,7 @@ func nonStringAttributeText(v any) string { // checkInputBinding is the builder's refusal: the widget being built must not // reach the writer with a binding the writer will turn into nothing. func (pb *pageBuilder) checkInputBinding(w *ast.WidgetV3, entity string) error { - if msg := inputBindingProblem(w, pb.argCtx, entity == ""); msg != "" { + if msg := inputBindingProblem(w, pb.argCtx, entity == "", pb.isLocalVariable); msg != "" { return mdlerrors.NewValidation(msg) } return nil @@ -137,7 +154,9 @@ func validateInputBindingContext(w *ast.WidgetV3, c pageArgContext, locationPref if own := argContextForOwnAction(w, c); own != c { c = own } - msg := inputBindingProblem(w, c, false) + // A bare `$name` is judged against the document's Variables by + // validatePageVariableBindings, which has them; this walk does not. + msg := inputBindingProblem(w, c, false, nil) if msg == "" { return nil } @@ -148,3 +167,69 @@ func validateInputBindingContext(w *ast.WidgetV3, c pageArgContext, locationPref Suggestion: "An input or dynamic text shows an attribute of the object a data view, list view, gallery or data grid supplies — wrap it in one.", }} } + +// bareVariableReference reads `Attribute: $name` — a variable named with no +// attribute after it — and returns the name without the "$". `$P/Name` is a +// different shape (ContextVariable set) and is not one. +func bareVariableReference(raw any) (string, bool) { + ds, ok := raw.(*ast.DataSourceV3) + if !ok || ds.ContextVariable != "" || len(ds.Args) > 0 || ds.Where != "" { + return "", false + } + name, ok := strings.CutPrefix(ds.Reference, "$") + if !ok || name == "" || strings.ContainsAny(name, "./ ") { + return "", false + } + return name, true +} + +func (pb *pageBuilder) isLocalVariable(name string) bool { + return pb.localVariables[name] +} + +// pageVariableInputBinding is the SourceVariable of an input bound directly to +// a page variable — `checkbox cb (Attribute: $ShowAll)` — or nil. Studio Pro +// stores that binding as a Forms$PageVariable naming the variable in its +// LocalVariable slot, with no AttributeRef (mendixlabs/mxcli#1235). +func (pb *pageBuilder) pageVariableInputBinding(w *ast.WidgetV3) *pages.WidgetVariable { + raw, _ := lookupPropCI(w, "Attribute") + name, ok := bareVariableReference(raw) + if !ok || !pb.localVariables[name] { + return nil + } + return &pages.WidgetVariable{Variable: name, Kind: "local"} +} + +// validatePageVariableBindings is MDL-WIDGET34 for `Attribute: $name` on a +// whole page or snippet, where the document's Variables are known: a name that +// is not one of them is a binding the writer would drop. +func validatePageVariableBindings(widgets []*ast.WidgetV3, variables []ast.PageVariable, locationPrefix string) []linter.Violation { + declared := make(map[string]bool, len(variables)) + for _, v := range variables { + declared[strings.TrimPrefix(v.Name, "$")] = true + } + isDeclared := func(name string) bool { return declared[name] } + var out []linter.Violation + var walk func(ws []*ast.WidgetV3) + walk = func(ws []*ast.WidgetV3) { + for _, w := range ws { + if w == nil { + continue + } + raw, _ := lookupPropCI(w, "Attribute") + if _, ok := bareVariableReference(raw); ok { + if msg := inputBindingProblem(w, pageArgContext{}, false, isDeclared); msg != "" { + out = append(out, linter.Violation{ + RuleID: "MDL-WIDGET34", + Severity: linter.SeverityError, + Message: locationPrefix + ": " + msg, + Suggestion: "Bind an input to a page variable by declaring it: `Variables: ( $name: Boolean = 'true' )`, then `Attribute: $name`.", + }) + } + } + walk(w.Children) + } + } + walk(widgets) + return out +} diff --git a/mdl/executor/cmd_pages_input_binding_context_test.go b/mdl/executor/cmd_pages_input_binding_context_test.go index 26b3ecd285..39bf13a08f 100644 --- a/mdl/executor/cmd_pages_input_binding_context_test.go +++ b/mdl/executor/cmd_pages_input_binding_context_test.go @@ -155,7 +155,9 @@ func TestValidateInputBindingWithoutDataContext(t *testing.T) { varRooted := []*ast.WidgetV3{{ Type: "dataview", Name: "dv", Properties: map[string]any{"DataSource": &ast.DataSourceV3{Type: "parameter", Reference: "$P"}}, - Children: []*ast.WidgetV3{inputWidget("textbox", "t", &ast.DataSourceV3{Type: "parameter", Reference: "$P"})}, + // The shape the visitor builds for `$P/Name` (a bare `$P` is a page + // variable reference, judged against the document's Variables). + Children: []*ast.WidgetV3{inputWidget("textbox", "t", &ast.DataSourceV3{Type: "association", ContextVariable: "P", Reference: "Name"})}, }} if n, _ := hits(varRooted, false); n != 1 { t.Errorf("`Attribute: $P/…` inside a data view: MDL-WIDGET34 = %d, want 1", n) diff --git a/mdl/executor/cmd_published_rest.go b/mdl/executor/cmd_published_rest.go index 40533f572a..6533f0288b 100644 --- a/mdl/executor/cmd_published_rest.go +++ b/mdl/executor/cmd_published_rest.go @@ -11,6 +11,7 @@ import ( "github.com/mendixlabs/mxcli/mdl/ast" mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" ) // listPublishedRestServices handles SHOW PUBLISHED REST SERVICES [IN module] command. @@ -135,8 +136,11 @@ func describePublishedRestService(ctx *ExecContext, name ast.QualifiedName) erro if op.Path != "" { opPath = " " + mdlQuoted(op.Path) } - fmt.Fprintf(ctx.Output, " %s%s%s%s;%s\n", - strings.ToLower(op.HTTPMethod), opPath, mf, deprecated, summary) + fmt.Fprintf(ctx.Output, " %s%s%s%s%s;%s\n", + strings.ToLower(op.HTTPMethod), opPath, mf, deprecated, publishedRestBindingClauses(op), summary) + for _, note := range publishedRestParameterNotes(op) { + fmt.Fprintf(ctx.Output, " -- %s\n", note) + } } fmt.Fprintln(ctx.Output, " }") } @@ -159,6 +163,55 @@ func describePublishedRestService(ctx *ExecContext, name ast.QualifiedName) erro return mdlerrors.NewNotFound("published rest service", name.String()) } +// publishedRestBindingClauses prints an operation's mapping bindings and its +// commit option, in the grammar's order. Commit is printed when it is not +// "Yes", the value exec writes when the statement has no commit clause. +func publishedRestBindingClauses(op *model.PublishedRestOperation) string { + var b strings.Builder + if op.ImportMapping != "" { + b.WriteString(" import mapping " + op.ImportMapping) + } + if op.ExportMapping != "" { + b.WriteString(" export mapping " + op.ExportMapping) + } + if op.Commit != "" && op.Commit != "Yes" { + b.WriteString(" commit " + op.Commit) + } + return b.String() +} + +// publishedRestParameterNotes names the operation parameters MDL cannot state: +// it derives every parameter from the microflow, so a header or form +// parameter, a parameter renamed away from its microflow parameter, or one +// with a description has no spelling. create or modify on the same project +// keeps them; a fresh create derives the parameter again. +func publishedRestParameterNotes(op *model.PublishedRestOperation) []string { + var notes []string + for _, p := range op.OperationParameters { + bound := p.MicroflowParameter + if i := strings.LastIndex(bound, "."); i >= 0 { + bound = bound[i+1:] + } + var why []string + switch p.ParameterType { + case "Path", "Query", "Body": + default: + why = append(why, strings.ToLower(p.ParameterType)+" parameter") + } + if bound != p.Name { + why = append(why, fmt.Sprintf("bound to $%s", bound)) + } + if p.Description != "" { + why = append(why, "description "+mdlQuoted(strings.ReplaceAll(p.Description, "\n", " "))) + } + if len(why) > 0 { + notes = append(notes, fmt.Sprintf("parameter %s: %s (not expressible in MDL; kept by create or modify on this project)", + p.Name, strings.Join(why, ", "))) + } + } + return notes +} + // findPublishedRestService looks up a published REST service by module and name. func findPublishedRestService(ctx *ExecContext, moduleName, name string) (*model.PublishedRestService, error) { @@ -234,21 +287,16 @@ func execCreatePublishedRestService(ctx *ExecContext, s *ast.CreatePublishedRest } for _, resDef := range s.Resources { - resource := &model.PublishedRestResource{ - Name: resDef.Name, - } - for _, opDef := range resDef.Operations { - op := &model.PublishedRestOperation{ - HTTPMethod: opDef.HTTPMethod, - Path: opDef.Path, - Microflow: opDef.Microflow.String(), - Summary: "", - Deprecated: opDef.Deprecated, - } - resource.Operations = append(resource.Operations, op) + resource, err := astResourceDefToModel(resDef) + if err != nil { + return err } svc.Resources = append(svc.Resources, resource) } + if existing != nil { + carryStoredOperations(svc, existing) + } + deriveOperationParameters(ctx, svc) if existing != nil { if s.Folder == "" { @@ -309,17 +357,212 @@ func execDropPublishedRestService(ctx *ExecContext, s *ast.DropPublishedRestServ // astResourceDefToModel converts an AST PublishedRestResourceDef to the // runtime model type used by the writer. -func astResourceDefToModel(def *ast.PublishedRestResourceDef) *model.PublishedRestResource { +func astResourceDefToModel(def *ast.PublishedRestResourceDef) (*model.PublishedRestResource, error) { resource := &model.PublishedRestResource{Name: def.Name} for _, opDef := range def.Operations { + commit, err := publishedRestCommit(opDef.Commit) + if err != nil { + return nil, mdlerrors.NewValidation(fmt.Sprintf("resource '%s', operation %s %s: %v", + def.Name, opDef.HTTPMethod, opDef.Path, err)) + } resource.Operations = append(resource.Operations, &model.PublishedRestOperation{ - HTTPMethod: opDef.HTTPMethod, - Path: opDef.Path, - Microflow: opDef.Microflow.String(), - Deprecated: opDef.Deprecated, + HTTPMethod: opDef.HTTPMethod, + Path: opDef.Path, + Microflow: opDef.Microflow.String(), + Deprecated: opDef.Deprecated, + ImportMapping: opDef.ImportMapping, + ExportMapping: opDef.ExportMapping, + Commit: commit, }) } - return resource + return resource, nil +} + +// publishedRestCommitValues are the values of Rest$PublishedRestServiceOperation.Commit. +var publishedRestCommitValues = []string{"Yes", "YesWithoutEvents", "No"} + +// publishedRestCommit returns the stored spelling of an operation's commit +// clause ("" when the statement has none). Any other value used to parse and +// be thrown away; it is refused, since there is nothing correct to write. +func publishedRestCommit(v string) (string, error) { + if v == "" { + return "", nil + } + for _, c := range publishedRestCommitValues { + if strings.EqualFold(v, c) { + return c, nil + } + } + return "", fmt.Errorf("commit %s is not a commit option (allowed: %s)", v, strings.Join(publishedRestCommitValues, ", ")) +} + +// carryStoredOperations copies onto each operation a statement declares what +// MDL cannot state about it — summary, documentation, the import mapping's +// object handling, and the stored parameters (their names, kinds and +// descriptions) — from the stored operation it restates. An operation is +// matched by its resource's name, its method and its path, in order; one +// without a match is new and gets the defaults. +func carryStoredOperations(svc, stored *model.PublishedRestService) { + pool := map[string][]*model.PublishedRestOperation{} + key := func(res string, op *model.PublishedRestOperation) string { + return res + "\x00" + strings.ToUpper(op.HTTPMethod) + "\x00" + op.Path + } + for _, res := range stored.Resources { + for _, op := range res.Operations { + k := key(res.Name, op) + pool[k] = append(pool[k], op) + } + } + for _, res := range svc.Resources { + for _, op := range res.Operations { + k := key(res.Name, op) + if len(pool[k]) == 0 { + continue + } + old := pool[k][0] + pool[k] = pool[k][1:] + op.Summary = old.Summary + op.Documentation = old.Documentation + op.ObjectHandlingBackup = old.ObjectHandlingBackup + if old.Microflow == op.Microflow { + op.OperationParameters = old.OperationParameters + } + } + } +} + +// deriveOperationParameters gives every operation the parameters Studio Pro +// derives from its microflow (ako/mxcli#571, mendixlabs/mxcli#1206): a +// parameter named in the path is a path parameter, an object or a list is the +// body, System.HttpRequest and System.HttpResponse are the request and the +// response themselves, and anything else is a query parameter — each with the +// microflow parameter's type. Without the query and body parameters mx check +// reports CE0350, and a String path parameter bound to an Integer is CE6539. +// +// The operation's stored parameters (op.OperationParameters on entry) win for +// the microflow parameter they bind: Studio Pro lets a parameter be renamed, +// described or turned into a header, and MDL has no spelling for that, so a +// rewrite keeps it. Only the type follows the microflow, and a path parameter +// follows the path. +func deriveOperationParameters(ctx *ExecContext, svc *model.PublishedRestService) { + var microflowsByName map[string]*microflows.Microflow + for _, resource := range svc.Resources { + for _, op := range resource.Operations { + if op.Microflow == "" { + continue + } + if microflowsByName == nil { + microflowsByName = liveMicroflowsByQualifiedName(ctx) + } + mf := microflowsByName[op.Microflow] + if mf == nil { + if len(op.OperationParameters) == 0 && !ctx.Quiet { + fmt.Fprintf(ctx.Output, "Warning: microflow %s not found, so operation %s %s gets only its path parameters, as String -- "+ + "create the microflow before the service, or its other parameters fail mx check with CE0350\n", + op.Microflow, strings.ToUpper(op.HTTPMethod), op.Path) + } + continue + } + op.OperationParameters = mergeOperationParameters(op.OperationParameters, operationParametersOf(op.Microflow, mf, op.PathParameterNames())) + } + } +} + +// liveMicroflowsByQualifiedName indexes the project's live microflows (an +// excluded twin never shadows the live one, #914). +func liveMicroflowsByQualifiedName(ctx *ExecContext) map[string]*microflows.Microflow { + out := map[string]*microflows.Microflow{} + all, err := ctx.Backend.ListMicroflows() + if err != nil { + return out + } + h, err := getHierarchy(ctx) + if err != nil { + return out + } + for _, mf := range all { + qn := h.GetQualifiedName(mf.ContainerID, mf.Name) + if prev, ok := out[qn]; ok && !prev.Excluded { + continue + } + out[qn] = mf + } + return out +} + +// operationParametersOf maps a microflow's parameters to operation parameters +// by the rule Studio Pro applies. +func operationParametersOf(mfName string, mf *microflows.Microflow, pathNames []string) []*model.PublishedRestOperationParameter { + inPath := make(map[string]bool, len(pathNames)) + for _, name := range pathNames { + inPath[name] = true + } + var params []*model.PublishedRestOperationParameter + for _, p := range mf.Parameters { + param := &model.PublishedRestOperationParameter{ + Name: p.Name, + ParameterType: "Query", + MicroflowParameter: mfName + "." + p.Name, + } + if p.Type != nil { + param.DataType = p.Type.GetTypeName() + } + switch t := p.Type.(type) { + case *microflows.ObjectType: + if t.EntityQualifiedName == "System.HttpRequest" || t.EntityQualifiedName == "System.HttpResponse" { + continue + } + param.ParameterType, param.DataType, param.QualifiedName = "Body", "Object", t.EntityQualifiedName + case *microflows.ListType: + param.ParameterType, param.DataType, param.QualifiedName = "Body", "List", t.EntityQualifiedName + case *microflows.EnumerationType: + param.DataType, param.QualifiedName = "Enumeration", t.EnumerationQualifiedName + } + if inPath[p.Name] { + param.ParameterType = "Path" + } + params = append(params, param) + } + return params +} + +// mergeOperationParameters keeps each stored parameter whose microflow +// parameter the microflow still has — in stored order, with the derived type, +// and the derived kind where either side is a path parameter — drops the ones +// it no longer has, keeps an unbound one as stored, and appends the derived +// parameters no stored one binds. +func mergeOperationParameters(stored, derived []*model.PublishedRestOperationParameter) []*model.PublishedRestOperationParameter { + byBinding := make(map[string]*model.PublishedRestOperationParameter, len(derived)) + for _, d := range derived { + byBinding[d.MicroflowParameter] = d + } + used := map[string]bool{} + var out []*model.PublishedRestOperationParameter + for _, sp := range stored { + if sp.MicroflowParameter == "" { + out = append(out, sp) + continue + } + d, ok := byBinding[sp.MicroflowParameter] + if !ok || used[sp.MicroflowParameter] { + continue + } + used[sp.MicroflowParameter] = true + kept := *sp + if d.DataType != "" { // "" is a type the reader could not read: keep the stored one + kept.DataType, kept.QualifiedName = d.DataType, d.QualifiedName + } + if d.ParameterType == "Path" || sp.ParameterType == "Path" { + kept.Name, kept.ParameterType = d.Name, d.ParameterType + } + out = append(out, &kept) + } + for _, d := range derived { + if !used[d.MicroflowParameter] { + out = append(out, d) + } + } + return out } // execAlterPublishedRestService applies SET / ADD RESOURCE / DROP RESOURCE @@ -363,7 +606,11 @@ func execAlterPublishedRestService(ctx *ExecContext, s *ast.AlterPublishedRestSe return mdlerrors.NewAlreadyExistsMsg("resource", a.Resource.Name, fmt.Sprintf("resource '%s' already exists on %s.%s", a.Resource.Name, s.Name.Module, s.Name.Name)) } } - svc.Resources = append(svc.Resources, astResourceDefToModel(a.Resource)) + resource, err := astResourceDefToModel(a.Resource) + if err != nil { + return err + } + svc.Resources = append(svc.Resources, resource) case *ast.PublishedRestDropResourceAction: idx := -1 @@ -383,6 +630,7 @@ func execAlterPublishedRestService(ctx *ExecContext, s *ast.AlterPublishedRestSe } } + deriveOperationParameters(ctx, svc) if err := ctx.Backend.UpdatePublishedRestService(svc); err != nil { return mdlerrors.NewBackend("alter published rest service", err) } diff --git a/mdl/executor/cmd_published_rest_params_test.go b/mdl/executor/cmd_published_rest_params_test.go new file mode 100644 index 0000000000..011a5c0fdb --- /dev/null +++ b/mdl/executor/cmd_published_rest_params_test.go @@ -0,0 +1,314 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// publishedRestFixture is a module RestQ holding the microflows of +// mendixlabs/mxcli#1206's repro, with the given stored services. +type publishedRestFixture struct { + mod *model.Module + stored []*model.PublishedRestService + created *model.PublishedRestService + updated *model.PublishedRestService +} + +func newPublishedRestFixture(t *testing.T, stored ...*model.PublishedRestService) (*publishedRestFixture, *ExecContext, *strings.Builder) { + t.Helper() + f := &publishedRestFixture{mod: mkModule("RestQ"), stored: stored} + for _, s := range stored { + s.ContainerID = f.mod.ID + } + param := func(name string, dt microflows.DataType) *microflows.MicroflowParameter { + return µflows.MicroflowParameter{Name: name, Type: dt} + } + mf := func(name string, params ...*microflows.MicroflowParameter) *microflows.Microflow { + return µflows.Microflow{ + BaseElement: model.BaseElement{ID: nextID("mf")}, + ContainerID: f.mod.ID, Name: name, Parameters: params, + } + } + mfs := []*microflows.Microflow{ + mf("GetStatus", + param("orderNumber", µflows.StringType{}), + param("count", µflows.IntegerType{}), + param("httpRequest", µflows.ObjectType{EntityQualifiedName: "System.HttpRequest"})), + mf("GetById", + param("id", µflows.IntegerType{}), + param("verbose", µflows.BooleanType{})), + mf("PutFile", param("file", µflows.ObjectType{EntityQualifiedName: "RestQ.Upload"})), + mf("PutMany", param("items", µflows.ListType{EntityQualifiedName: "RestQ.Item"})), + } + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{f.mod}, nil }, + ListMicroflowsFunc: func() ([]*microflows.Microflow, error) { + return mfs, nil + }, + ListPublishedRestServicesFunc: func() ([]*model.PublishedRestService, error) { return f.stored, nil }, + CreatePublishedRestServiceFunc: func(svc *model.PublishedRestService) error { + f.created = svc + return nil + }, + UpdatePublishedRestServiceFunc: func(svc *model.PublishedRestService) error { + f.updated = svc + return nil + }, + } + h := mkHierarchy(f.mod) + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + out := &strings.Builder{} + ctx.Output = out + return f, ctx, out +} + +func execPublishedRest(t *testing.T, ctx *ExecContext, src string) error { + t.Helper() + prog := parseMDL(t, src) + for _, stmt := range prog.Statements { + var err error + switch s := stmt.(type) { + case *ast.CreatePublishedRestServiceStmt: + err = execCreatePublishedRestService(ctx, s) + case *ast.AlterPublishedRestServiceStmt: + err = execAlterPublishedRestService(ctx, s) + default: + t.Fatalf("unexpected statement %T", stmt) + } + if err != nil { + return err + } + } + return nil +} + +func paramsString(op *model.PublishedRestOperation) string { + var parts []string + for _, p := range op.OperationParameters { + s := p.ParameterType + " " + p.Name + ":" + p.DataType + if p.QualifiedName != "" { + s += "(" + p.QualifiedName + ")" + } + s += "->" + p.MicroflowParameter + parts = append(parts, s) + } + return strings.Join(parts, "; ") +} + +// TestCreatePublishedRestService_DerivesParameters is mendixlabs/mxcli#1206: +// only the path's placeholders were written, each a String, so every query and +// body parameter failed mx check with CE0350 and an Integer {id} with CE6539. +func TestCreatePublishedRestService_DerivesParameters(t *testing.T) { + f, ctx, _ := newPublishedRestFixture(t) + assertNoError(t, execPublishedRest(t, ctx, `create published rest service RestQ.Orders (Path: 'rest/orders/v1') { + resource 'orders' { + get 'status' microflow RestQ.GetStatus; + get 'items/{id}' microflow RestQ.GetById; + post 'upload' microflow RestQ.PutFile; + put 'many' microflow RestQ.PutMany; + } +};`)) + if f.created == nil { + t.Fatal("nothing created") + } + ops := f.created.Resources[0].Operations + want := []string{ + "Query orderNumber:String->RestQ.GetStatus.orderNumber; Query count:Integer->RestQ.GetStatus.count", + "Path id:Integer->RestQ.GetById.id; Query verbose:Boolean->RestQ.GetById.verbose", + "Body file:Object(RestQ.Upload)->RestQ.PutFile.file", + "Body items:List(RestQ.Item)->RestQ.PutMany.items", + } + for i, w := range want { + if got := paramsString(ops[i]); got != w { + t.Errorf("operation %d parameters:\n got %s\nwant %s", i, got, w) + } + } +} + +// TestCreatePublishedRestService_WritesMappingBindings is ako/mxcli#571: the +// import mapping, export mapping and commit clauses parsed and were thrown away. +func TestCreatePublishedRestService_WritesMappingBindings(t *testing.T) { + f, ctx, _ := newPublishedRestFixture(t) + assertNoError(t, execPublishedRest(t, ctx, `create published rest service RestQ.Orders (Path: 'rest/orders/v1') { + resource 'orders' { + post 'upload' microflow RestQ.PutFile import mapping RestQ.IMM_Upload export mapping "RestQ"."EMM_Result" commit yeswithoutevents; + get 'status' microflow RestQ.GetStatus; + } +};`)) + op := f.created.Resources[0].Operations[0] + if op.ImportMapping != "RestQ.IMM_Upload" || op.ExportMapping != "RestQ.EMM_Result" || op.Commit != "YesWithoutEvents" { + t.Errorf("bindings = import %q export %q commit %q, want RestQ.IMM_Upload, RestQ.EMM_Result, YesWithoutEvents", + op.ImportMapping, op.ExportMapping, op.Commit) + } + if other := f.created.Resources[0].Operations[1]; other.ImportMapping != "" || other.ExportMapping != "" || other.Commit != "" { + t.Errorf("an operation without clauses got bindings: %+v", other) + } +} + +// TestCreatePublishedRestService_RefusesUnknownCommit: `commit Maybe` parsed +// and was thrown away; there is nothing correct to write, so it is refused by +// exec and by check (MDL-REST03). +func TestCreatePublishedRestService_RefusesUnknownCommit(t *testing.T) { + src := `create published rest service RestQ.Orders (Path: 'rest/orders/v1') { + resource 'orders' { post 'upload' microflow RestQ.PutFile import mapping RestQ.IMM commit Maybe; } +};` + f, ctx, _ := newPublishedRestFixture(t) + err := execPublishedRest(t, ctx, src) + if err == nil || !strings.Contains(err.Error(), "commit Maybe") { + t.Fatalf("exec error = %v, want a refusal naming commit Maybe", err) + } + if f.created != nil { + t.Error("the service was written despite the refusal") + } + v := ValidatePublishedRestCommit(parseMDL(t, src)) + if len(v) != 1 || v[0].RuleID != "MDL-REST03" { + t.Errorf("check = %+v, want one MDL-REST03", v) + } + if v := ValidatePublishedRestCommit(parseMDL(t, strings.Replace(src, "Maybe", "No", 1))); len(v) != 0 { + t.Errorf("control: commit No flagged: %+v", v) + } +} + +// TestCreateOrModifyPublishedRestService_KeepsStoredParameters: a parameter +// Studio Pro lets the user rename, describe or turn into a header has no MDL +// spelling; create or modify of the same operation keeps it, as it keeps the +// summary, documentation and object handling, while the type follows the +// microflow and a new microflow parameter is derived. +func TestCreateOrModifyPublishedRestService_KeepsStoredParameters(t *testing.T) { + stored := &model.PublishedRestService{ + BaseElement: model.BaseElement{ID: nextID("prs")}, + Name: "Orders", + Path: "rest/orders/v1", + Resources: []*model.PublishedRestResource{{ + Name: "orders", + Operations: []*model.PublishedRestOperation{{ + HTTPMethod: "Get", Path: "status", Microflow: "RestQ.GetStatus", + Summary: "Order status", Documentation: "docs", ObjectHandlingBackup: "Error", Commit: "No", + OperationParameters: []*model.PublishedRestOperationParameter{ + {Name: "X-Count", ParameterType: "Header", MicroflowParameter: "RestQ.GetStatus.count", DataType: "String", Description: "how many"}, + }, + }}, + }}, + } + f, ctx, out := newPublishedRestFixture(t, stored) + assertNoError(t, execPublishedRest(t, ctx, `create or modify published rest service RestQ.Orders (Path: 'rest/orders/v1') { + resource 'orders' { get 'status' microflow RestQ.GetStatus commit No; } +};`)) + if f.updated == nil { + t.Fatalf("nothing updated; output:\n%s", out) + } + op := f.updated.Resources[0].Operations[0] + if got, want := paramsString(op), "Header X-Count:Integer->RestQ.GetStatus.count; Query orderNumber:String->RestQ.GetStatus.orderNumber"; got != want { + t.Errorf("parameters:\n got %s\nwant %s", got, want) + } + if op.OperationParameters[0].Description != "how many" { + t.Errorf("description lost: %+v", op.OperationParameters[0]) + } + if op.Summary != "Order status" || op.Documentation != "docs" || op.ObjectHandlingBackup != "Error" { + t.Errorf("unstated operation properties not carried: %+v", op) + } +} + +// TestAlterPublishedRestService_DerivesAddedOperations: ALTER writes every +// operation again, so an added resource derives its parameters and keeps its +// bindings, and the untouched operations keep theirs. +func TestAlterPublishedRestService_DerivesAddedOperations(t *testing.T) { + stored := &model.PublishedRestService{ + BaseElement: model.BaseElement{ID: nextID("prs")}, + Name: "Orders", + Path: "rest/orders/v1", + Resources: []*model.PublishedRestResource{{ + Name: "orders", + Operations: []*model.PublishedRestOperation{{ + HTTPMethod: "Post", Path: "upload", Microflow: "RestQ.PutFile", + ImportMapping: "RestQ.IMM_Upload", Commit: "No", + OperationParameters: []*model.PublishedRestOperationParameter{ + {Name: "file", ParameterType: "Body", MicroflowParameter: "RestQ.PutFile.file", DataType: "Object", QualifiedName: "RestQ.Upload"}, + }, + }}, + }}, + } + f, ctx, _ := newPublishedRestFixture(t, stored) + assertNoError(t, execPublishedRest(t, ctx, `alter published rest service RestQ.Orders + add resource 'items' { get '{id}' microflow RestQ.GetById export mapping RestQ.EMM_Item; };`)) + if f.updated == nil { + t.Fatal("nothing updated") + } + kept := f.updated.Resources[0].Operations[0] + if kept.ImportMapping != "RestQ.IMM_Upload" || kept.Commit != "No" || paramsString(kept) != "Body file:Object(RestQ.Upload)->RestQ.PutFile.file" { + t.Errorf("untouched operation changed: %+v %s", kept, paramsString(kept)) + } + added := f.updated.Resources[1].Operations[0] + if added.ExportMapping != "RestQ.EMM_Item" { + t.Errorf("added operation export mapping = %q", added.ExportMapping) + } + if got, want := paramsString(added), "Path id:Integer->RestQ.GetById.id; Query verbose:Boolean->RestQ.GetById.verbose"; got != want { + t.Errorf("added operation parameters:\n got %s\nwant %s", got, want) + } +} + +// TestCreatePublishedRestService_MissingMicroflowWarns: without the microflow +// only the path is known; the operation keeps today's path-only parameters and +// exec says what mx check will report. +func TestCreatePublishedRestService_MissingMicroflowWarns(t *testing.T) { + f, ctx, out := newPublishedRestFixture(t) + assertNoError(t, execPublishedRest(t, ctx, `create published rest service RestQ.Orders (Path: 'rest/orders/v1') { + resource 'orders' { get '{id}' microflow RestQ.NotYet; } +};`)) + if len(f.created.Resources[0].Operations[0].OperationParameters) != 0 { + t.Errorf("parameters derived for a missing microflow: %s", paramsString(f.created.Resources[0].Operations[0])) + } + if !strings.Contains(out.String(), "microflow RestQ.NotYet not found") { + t.Errorf("no warning; output:\n%s", out) + } +} + +// TestDescribePublishedRestService_PrintsBindings: describe prints the +// bindings exec now writes, so executing its output restates them, and names +// the parameters MDL cannot state. +func TestDescribePublishedRestService_PrintsBindings(t *testing.T) { + stored := &model.PublishedRestService{ + BaseElement: model.BaseElement{ID: nextID("prs")}, + Name: "Orders", + Path: "rest/orders/v1", + Resources: []*model.PublishedRestResource{{ + Name: "orders", + Operations: []*model.PublishedRestOperation{ + { + HTTPMethod: "Post", Path: "upload", Microflow: "RestQ.PutFile", + ImportMapping: "RestQ.IMM_Upload", ExportMapping: "RestQ.EMM_Result", Commit: "YesWithoutEvents", + }, + { + HTTPMethod: "Get", Path: "status", Microflow: "RestQ.GetStatus", Commit: "Yes", + OperationParameters: []*model.PublishedRestOperationParameter{ + {Name: "X-Count", ParameterType: "Header", MicroflowParameter: "RestQ.GetStatus.count", DataType: "Integer"}, + {Name: "orderNumber", ParameterType: "Query", MicroflowParameter: "RestQ.GetStatus.orderNumber", DataType: "String"}, + }, + }, + }, + }}, + } + _, ctx, out := newPublishedRestFixture(t, stored) + assertNoError(t, describePublishedRestService(ctx, ast.QualifiedName{Module: "RestQ", Name: "Orders"})) + got := out.String() + assertContainsStr(t, got, "post 'upload' microflow RestQ.PutFile import mapping RestQ.IMM_Upload export mapping RestQ.EMM_Result commit YesWithoutEvents;") + assertContainsStr(t, got, "get 'status' microflow RestQ.GetStatus;") + assertContainsStr(t, got, "-- parameter X-Count: header parameter, bound to $count") + if strings.Contains(got, "parameter orderNumber") { + t.Errorf("a derivable parameter was flagged:\n%s", got) + } + // The output re-parses, and its bindings parse back to what is stored. + prog := parseMDL(t, got) + op := prog.Statements[0].(*ast.CreatePublishedRestServiceStmt).Resources[0].Operations[0] + if op.ImportMapping != "RestQ.IMM_Upload" || op.ExportMapping != "RestQ.EMM_Result" || op.Commit != "YesWithoutEvents" { + t.Errorf("describe output parses back to %+v", op) + } +} diff --git a/mdl/executor/cmd_translations.go b/mdl/executor/cmd_translations.go index 51b5f76bb3..b25dae21cd 100644 --- a/mdl/executor/cmd_translations.go +++ b/mdl/executor/cmd_translations.go @@ -107,31 +107,14 @@ func execCreateTranslations(ctx *ExecContext, s *ast.CreateTranslationsStmt) err return err } src := sourceLanguage(ctx) - if strings.EqualFold(s.Language, src) { - return mdlerrors.NewValidationf( - "%s is the project's source language — writing translations into it would "+ - "overwrite the strings the rest of the model is keyed on", s.Language) + if err := translationsRefusal(ctx, s, src, scope); err != nil { + return err } mode := translations.ModeMerge switch s.Mode { case ast.TranslationsReplace: mode = translations.ModeReplace - case ast.TranslationsCreate: - // The language is the thing that exists, so bare CREATE refuses when it - // already has translations — the same contract every other CREATE has. - existing, err := translations.Languages(ctx.Backend, scope) - if err != nil { - return mdlerrors.NewBackend("read languages", err) - } - for _, l := range existing { - if strings.EqualFold(l, s.Language) { - return mdlerrors.NewValidationf( - "%s already has translations — use `create or modify translations` to "+ - "merge these in, or `create or replace translations` to make this file "+ - "authoritative (which REMOVES translations it does not name)", s.Language) - } - } } dict := make(translations.Dictionary, len(s.Entries)) diff --git a/mdl/executor/describe_unqualified_name_test.go b/mdl/executor/describe_unqualified_name_test.go new file mode 100644 index 0000000000..d14feae474 --- /dev/null +++ b/mdl/executor/describe_unqualified_name_test.go @@ -0,0 +1,31 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// mendixlabs/mxcli#1218 (3): `describe entity String` failed with "module name +// is required: objects must be created within a module" — the create path's +// message, which reads as if describe were creating something. It now names +// the unqualified name. +func TestDescribeUnqualifiedEntityNamesTheName(t *testing.T) { + for _, typ := range []ast.DescribeObjectType{ast.DescribeEntity, ast.DescribeAssociation} { + ctx, _ := newMockCtx(t) + err := execDescribe(ctx, &ast.DescribeStmt{ObjectType: typ, Name: ast.QualifiedName{Name: "String"}}) + if err == nil { + t.Fatalf("%v: no error for an unqualified name", typ) + } + msg := err.Error() + if strings.Contains(msg, "created") { + t.Errorf("%v: %q still speaks of creating objects", typ, msg) + } + if !strings.Contains(msg, `"String" is not a qualified`) { + t.Errorf("%v: %q does not name the unqualified name", typ, msg) + } + } +} diff --git a/mdl/executor/design_property_keyword_keys_test.go b/mdl/executor/design_property_keyword_keys_test.go index 5f9186abf3..534447fbb7 100644 --- a/mdl/executor/design_property_keyword_keys_test.go +++ b/mdl/executor/design_property_keyword_keys_test.go @@ -161,10 +161,10 @@ func TestValidateDesignProperties_SlotDesignPropsAreReportedDropped(t *testing.T footer ft (DesignProperties: ['Card style': on]) { dynamictext f (Content: 'x') } } }`, reg) - // A layout grid's rows and columns have no stored name, so the visitor - // drops the one written here (MDL-DEPR005, #749); the message names them - // by kind and parent instead. - for _, name := range []string{`row inside layoutgrid "lg"`, `column inside row sets`, `column inside row "topRow"`, `footer "ft"`} { + // A layout grid's rows and columns, and a data view's footer, have no + // stored name, so the visitor drops the one written here (MDL-DEPR005, + // #749, ako/mxcli#528); the message names them by kind and parent instead. + for _, name := range []string{`row inside layoutgrid "lg"`, `column inside row sets`, `column inside row "topRow"`, `footer inside dataview "dv"`} { var found bool for _, v := range vs { if v.RuleID == "MDL-WIDGET07" && strings.Contains(v.Message, name) && diff --git a/mdl/executor/executor.go b/mdl/executor/executor.go index 19fd3c4ec7..c16e62c045 100644 --- a/mdl/executor/executor.go +++ b/mdl/executor/executor.go @@ -255,12 +255,13 @@ type Executor struct { settings map[string]any cache *executorCache catalog *catalog.Catalog - quiet bool // suppress connection and status messages - tally *mutationTally // collapses a program run's "Unchanged" reports into one line - accessRun *accessRuleRun // the open run of access-rule statements, if any (#872, #890) - format OutputFormat // output format (table, json) - logger *diaglog.Logger // session diagnostics logger (nil = no logging) - tracer *backend.Tracer // MCP tool-call tracer (--mcp-trace; nil = off) + quiet bool // suppress connection and status messages + tally *mutationTally // collapses a program run's "Unchanged" reports into one line + accessRun *accessRuleRun // the open run of access-rule statements, if any (#872, #890) + format OutputFormat // output format (table, json) + logger *diaglog.Logger // session diagnostics logger (nil = no logging) + tracer *backend.Tracer // MCP tool-call tracer (--mcp-trace; nil = off) + stmtGuard func(ast.Statement) (ast.Statement, error) fragments map[string]*ast.DefineFragmentStmt // script-scoped fragment definitions sqlMgr *sqllib.Manager // external SQL connection manager (lazy init) themeRegistry *ThemeRegistry // cached theme design property definitions (lazy init) @@ -326,12 +327,29 @@ func (e *Executor) SetTracer(t *backend.Tracer) { e.tracer = t } +// SetStatementGuard installs a function every statement passes through before +// it runs — including the statements of a nested EXECUTE SCRIPT, which are +// dispatched through Execute too. It returns the statement to run in its place, +// or an error to refuse it. `mxcli diff` uses it to keep a script it runs on a +// scratch copy from reaching anything but the copy. +func (e *Executor) SetStatementGuard(g func(ast.Statement) (ast.Statement, error)) { + e.stmtGuard = g +} + // Execute runs a single MDL statement with output-line and wall-clock guards. // Each statement gets a fresh line budget. If the statement exceeds maxOutputLines // lines of output or runs longer than the configured timeout, it is aborted with an error. func (e *Executor) Execute(stmt ast.Statement) error { start := time.Now() + if e.stmtGuard != nil { + s, err := e.stmtGuard(stmt) + if err != nil { + return err + } + stmt = s + } + // Announce the MDL command so the PED calls it triggers (reported by the MCP // client) group under it (--mcp-trace, level 2; no-op otherwise). e.tracer.Statement(stmtSummary(stmt)) @@ -376,7 +394,20 @@ func (e *Executor) Execute(stmt ast.Statement) error { } // ExecuteProgram runs all statements in a program. -func (e *Executor) ExecuteProgram(prog *ast.Program) (err error) { +func (e *Executor) ExecuteProgram(prog *ast.Program) error { + _, err := e.ExecuteProgramReportingStop(prog) + return err +} + +// ExecuteProgramReportingStop is ExecuteProgram that also says where it +// stopped: the 0-based index of the statement whose error ended the run, or +// -1 when no statement failed (an error from the run's end-of-program work — +// the access-rule flush, reconciliation — is not one statement's). A caller +// running several statements uses it to say which one failed and how many +// were not run, rather than leaving the rest silently skipped +// (mendixlabs/mxcli#1218). +func (e *Executor) ExecuteProgramReportingStop(prog *ast.Program) (stoppedAt int, err error) { + stoppedAt = -1 if e.beginTally() { defer e.flushTally() } @@ -401,19 +432,19 @@ func (e *Executor) ExecuteProgram(prog *ast.Program) (err error) { } }() - for _, stmt := range prog.Statements { + for i, stmt := range prog.Statements { if err := rules.step(e, stmt); err != nil { - return err + return -1, err } if err := e.Execute(stmt); err != nil { - return annotateForwardRef(err, stmt, created, allDefined) + return i, annotateForwardRef(err, stmt, created, allDefined) } created.collectSingle(stmt) } if err := rules.end(); err != nil { - return err + return -1, err } - return e.finalizeProgramExecution() + return -1, e.finalizeProgramExecution() } // enterLanguage runs the following statements under a program's language diff --git a/mdl/executor/executor_query.go b/mdl/executor/executor_query.go index 57e13c1af6..a0352f458a 100644 --- a/mdl/executor/executor_query.go +++ b/mdl/executor/executor_query.go @@ -201,6 +201,15 @@ func execDescribe(ctx *ExecContext, s *ast.DescribeStmt) error { s.ObjectType = resolved } + // An entity or association is only ever found by its module. Without one + // the lookup failed with the create-path message "objects must be created + // within a module", which reads as if describe were creating something + // (mendixlabs/mxcli#1218). + if (s.ObjectType == ast.DescribeEntity || s.ObjectType == ast.DescribeAssociation) && s.Name.Module == "" { + label := describeObjectTypeLabel(s.ObjectType) + return mdlerrors.NewValidation(fmt.Sprintf("%q is not a qualified %s name: write Module.%s", s.Name.Name, label, s.Name.Name)) + } + // Determine the object type label and name for JSON wrapping. objectType := describeObjectTypeLabel(s.ObjectType) name := s.Name.String() diff --git a/mdl/executor/flow_handler_annotations_test.go b/mdl/executor/flow_handler_annotations_test.go new file mode 100644 index 0000000000..efcbb2c1a5 --- /dev/null +++ b/mdl/executor/flow_handler_annotations_test.go @@ -0,0 +1,193 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +func buildCollectionFromMDL(t *testing.T, nanoflow bool, src string) *microflows.MicroflowObjectCollection { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatal(errs) + } + var body []ast.MicroflowStatement + switch s := prog.Statements[len(prog.Statements)-1].(type) { + case *ast.CreateMicroflowStmt: + body = s.Body + case *ast.CreateNanoflowStmt: + body = s.Body + } + fb := &flowBuilder{posX: 100, posY: 100, spacing: HorizontalSpacing, isNanoflow: nanoflow} + return fb.buildFlowGraph(body, nil) +} + +func objectAtPos(oc *microflows.MicroflowObjectCollection, x, y int) model.ID { + for _, o := range oc.Objects { + if p := o.GetPosition(); p.X == x && p.Y == y { + return o.GetID() + } + } + return "" +} + +// mendixlabs/mxcli#991. Inside `on error … begin … end error` the statements' +// @position was honoured but @anchor and @curve were not: the handler loop +// created its flows with the builder's default sides and never recorded a +// curve, so check and exec passed and the edges came out straight, on default +// sides. They now mean what they mean outside a handler — @anchor's `to:` is +// the side the statement's incoming edge enters (here, the error edge), `from:` +// and @curve shape the edge leaving it. +func TestErrorHandlerBody_HonoursAnchorAndCurve(t *testing.T) { + src := `mdl 1; +create nanoflow M.NF () +begin + @position(240, 200) + call microflow M.SUB() on error without rollback begin + @position(240, 380) + @anchor(from: right, to: left) + @curve(from: (0, 30), to: (0, -30)) + log error node 'R' 'one'; + @position(440, 380) + @anchor(to: top) + log error node 'R' 'two'; + end error; + @position(560, 200) + return; +end;` + oc := buildCollectionFromMDL(t, true, src) + one, two := objectAtPos(oc, 240, 380), objectAtPos(oc, 440, 380) + if one == "" || two == "" { + t.Fatal("handler activities not at their @position") + } + var errEdge, between *microflows.SequenceFlow + for _, f := range oc.Flows { + switch { + case f.IsErrorHandler && f.DestinationID == one: + errEdge = f + case f.OriginID == one && f.DestinationID == two: + between = f + } + } + if errEdge == nil || between == nil { + t.Fatalf("missing flows: error edge %v, between %v", errEdge, between) + } + if errEdge.DestinationConnectionIndex != AnchorLeft { + t.Errorf("error edge enters side %d, want left (%d) from @anchor(to: left)", errEdge.DestinationConnectionIndex, AnchorLeft) + } + if between.OriginConnectionIndex != AnchorRight || between.DestinationConnectionIndex != AnchorTop { + t.Errorf("handler edge is (%d,%d), want (right,top) = (%d,%d)", between.OriginConnectionIndex, between.DestinationConnectionIndex, AnchorRight, AnchorTop) + } + if between.OriginControlVector != "0;30" || between.DestinationControlVector != "0;-30" { + t.Errorf("handler edge curve is %q/%q, want 0;30/0;-30", between.OriginControlVector, between.DestinationControlVector) + } +} + +// The last handler statement's @anchor(from:) and @curve shape the edge that +// rejoins the main flow; its `to:` stays on its own incoming edge. +func TestErrorHandlerBody_TailEdgeTakesTheLastStatementsFromAndCurve(t *testing.T) { + src := `mdl 1; +create nanoflow M.NF () +begin + @position(240, 200) + call microflow M.SUB() on error without rollback begin + @position(240, 380) + @anchor(from: top, to: left) + @curve(from: (0, -30), to: (-30, 0)) + log error node 'R' 'one'; + end error; + @position(560, 200) + return; +end;` + oc := buildCollectionFromMDL(t, true, src) + one := objectAtPos(oc, 240, 380) + var tail *microflows.SequenceFlow + for _, f := range oc.Flows { + if f.OriginID == one && !f.IsErrorHandler { + tail = f + } + } + if tail == nil { + t.Fatal("no edge leaves the handler") + } + if tail.OriginConnectionIndex != AnchorTop { + t.Errorf("rejoin edge leaves side %d, want top (%d)", tail.OriginConnectionIndex, AnchorTop) + } + if tail.DestinationConnectionIndex == AnchorLeft { + t.Errorf("the statement's `to:` leaked onto its outgoing edge") + } + if tail.OriginControlVector != "0;-30" || tail.DestinationControlVector != "-30;0" { + t.Errorf("rejoin edge curve is %q/%q, want 0;-30/-30;0", tail.OriginControlVector, tail.DestinationControlVector) + } +} + +// mendixlabs/mxcli#992, end to end: the one-sided per-case form describe emits +// now reaches the true edge. +func TestSplitBranch_OneSidedAnchorReachesTheEdge(t *testing.T) { + src := `mdl 1; +create nanoflow M.NF ($Q: Integer) +begin + @position(360, 200) + @anchor(true: (to: top)) + if $Q = 0 then + @position(560, 80) + log info node 'a' 'b'; + else + @position(560, 320) + log info node 'a' 'c'; + end if; +end;` + oc := buildCollectionFromMDL(t, true, src) + dest := objectAtPos(oc, 560, 80) + for _, f := range oc.Flows { + if f.DestinationID == dest { + if f.DestinationConnectionIndex != AnchorTop { + t.Errorf("true edge enters side %d, want top (%d)", f.DestinationConnectionIndex, AnchorTop) + } + return + } + } + t.Fatal("no true edge") +} + +// mendixlabs/mxcli#991, the read half. DESCRIBE printed no @position — nor +// @anchor, @curve, @caption, @color or @excluded — for a statement inside an +// error handler, though the builder honours each there. So describe → exec of a +// flow whose handler someone laid out put the handler back on auto-placement, +// reporting success. The handler body now carries the same annotations as the +// main path, and a second describe is a fixed point. +func TestErrorHandlerBody_DescribeKeepsItsLayout(t *testing.T) { + src := `mdl 1; +create microflow M.F () +begin + @position(240, 200) + call microflow M.SUB() on error without rollback begin + @position(240, 380) + @anchor(from: right, to: left) + @curve(from: (0, 30), to: (0, -30)) + @caption 'Handled' + @color Red + log error node 'R' 'one'; + end error; + @position(560, 200) + return; +end;` + oc := buildCollectionFromMDL(t, false, src) + body := describeBody(t, µflows.Microflow{ObjectCollection: oc}) + for _, want := range []string{"@position(240, 380)", "to: left", "@curve(from: (0, 30), to: (0, -30))", "@caption 'Handled'", "@color Red"} { + if !strings.Contains(body, want) { + t.Errorf("describe dropped %q inside the handler:\n%s", want, body) + } + } + again := buildCollectionFromMDL(t, false, "mdl 1;\ncreate microflow M.F ()\nbegin\n"+body+"\nend;") + if second := describeBody(t, µflows.Microflow{ObjectCollection: again}); second != body { + t.Errorf("describe is not a fixed point:\n--- first\n%s\n--- second\n%s", body, second) + } +} diff --git a/mdl/executor/javaaction_flow_param.go b/mdl/executor/javaaction_flow_param.go new file mode 100644 index 0000000000..c23ac40822 --- /dev/null +++ b/mdl/executor/javaaction_flow_param.go @@ -0,0 +1,74 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "regexp" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// A Java action parameter of type Microflow stores a document reference — the +// microflow's qualified name — and Studio Pro only lets you pick one. There is +// no dynamic form. Given a variable, mxcli wrote the TEXT `$Name` as that +// reference; check and exec both accepted it, and the project no longer loaded +// (mendixlabs/mxcli#1210, Mendix 11.14.0): +// +// StorageLoadException: … Microflow parameter value … has an invalid value '' +// for property Microflow. The text '$Name' is not a valid MicroflowIdentifier. +// +// That is a silent wrong write — an unloadable project, recoverable only from +// version control — so it is refused under every language version (ADR-0011). +// microflowParamArgRefusal is the one decision: the flow builder refuses the +// call with it, and reference validation (check -p, and exec's pre-flight) +// reports it from the same function, so the two cannot disagree. + +// microflowIdentifierRe is the shape a stored microflow reference has. +var microflowIdentifierRe = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]*$`) + +// microflowParamArgRefusal returns why v cannot be written into the +// Microflow-typed parameter param of action, or "" when it can. `empty` is not +// passed here: it is the unbound marker, handled before the value is built. +func microflowParamArgRefusal(action, param string, v ast.Expression) string { + switch e := v.(type) { + case *ast.QualifiedNameExpr: + return "" + case *ast.LiteralExpr: + if s, ok := e.Value.(string); ok && e.Kind == ast.LiteralString && microflowIdentifierRe.MatchString(s) { + return "" + } + } + return fmt.Sprintf("call java action %s: parameter %s is of type Microflow, which takes a microflow "+ + "name ('Module.Microflow'), not %s — Mendix has no dynamic microflow reference here, and the text "+ + "would be stored as the reference, leaving a project that no longer loads", action, param, + expressionToString(v)) +} + +// microflowTypedParams names the parameters a CREATE JAVA ACTION declares as +// `Microflow` — the spelling astDataTypeToJavaActionParamType turns into a +// javaactions.MicroflowType. +func microflowTypedParams(params []ast.JavaActionParam) map[string]bool { + out := map[string]bool{} + for _, p := range params { + if bareDataTypeName(p.Type) == "Microflow" { + out[p.Name] = true + } + } + return out +} + +// microflowParamArgErrors applies microflowParamArgRefusal to the arguments of +// one call, given the action's Microflow-typed parameters. +func microflowParamArgErrors(ref codeActionCallRef, flowParams map[string]bool) []string { + var out []string + for _, a := range ref.args { + if !flowParams[a.Name] || isEmptyJavaActionArgument(a.Value) { + continue + } + if msg := microflowParamArgRefusal(ref.name, a.Name, a.Value); msg != "" { + out = append(out, msg) + } + } + return out +} diff --git a/mdl/executor/lock_workflow_selection_test.go b/mdl/executor/lock_workflow_selection_test.go new file mode 100644 index 0000000000..faa6ce4de8 --- /dev/null +++ b/mdl/executor/lock_workflow_selection_test.go @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// mendixlabs/mxcli#870. A Lock workflow activity always names a workflow +// definition; PauseAllWorkflows is Studio Pro's "Pause instances" checkbox +// (Unlock's is "Unpause instances", ResumeAllPausedWorkflows), on by default. +// WorkflowCommons.ACT_WorkflowDefinition_Lock in ako/TestApp stores both +// branches of that choice: PauseAllWorkflows true WITH a WorkflowDefinition +// variable selection, and false with the same selection. +// +// MDL read the flag as "all workflows": `lock workflow all;` wrote the flag and +// no selection, which mxbuild rejects with CE1825 "The 'Workflow' property is +// required", and DESCRIBE printed the Studio Pro activity as that same +// `lock workflow all;` — dropping the workflow, so describe → exec turned a +// working microflow into one that does not build. + +func lockStmt(t *testing.T, src string) ast.MicroflowStatement { + t.Helper() + prog, errs := visitor.Build("mdl 1;\ncreate microflow M.MF ($Wf: System.WorkflowDefinition)\nbegin\n" + src + "\nend;") + if len(errs) > 0 { + t.Fatalf("%s: %v", src, errs) + } + return prog.Statements[len(prog.Statements)-1].(*ast.CreateMicroflowStmt).Body[0] +} + +func TestLockWorkflow_PauseInstancesIsAFlagOnASelection(t *testing.T) { + for _, tc := range []struct { + src string + variable, name string + flag bool + }{ + {"lock workflow $Wf;", "Wf", "", false}, + {"lock workflow $Wf pause all;", "Wf", "", true}, + {"lock workflow M.Approve pause all;", "", "M.Approve", true}, + {"unlock workflow $Wf;", "Wf", "", false}, + {"unlock workflow $Wf unpause all;", "Wf", "", true}, + {"unlock workflow M.Approve;", "", "M.Approve", false}, + } { + fb := &flowBuilder{posX: 100, posY: 100, spacing: HorizontalSpacing} + oc := fb.buildFlowGraph([]ast.MicroflowStatement{lockStmt(t, tc.src)}, nil) + var variable, name string + var flag, found bool + for _, o := range oc.Objects { + aa, ok := o.(*microflows.ActionActivity) + if !ok { + continue + } + switch a := aa.Action.(type) { + case *microflows.LockWorkflowAction: + variable, name, flag, found = a.WorkflowVariable, a.Workflow, a.PauseAllWorkflows, true + case *microflows.UnlockWorkflowAction: + variable, name, flag, found = a.WorkflowVariable, a.Workflow, a.ResumeAllPausedWorkflows, true + } + } + if !found { + t.Errorf("%s: no action built", tc.src) + continue + } + if variable != tc.variable || name != tc.name || flag != tc.flag { + t.Errorf("%s: built var=%q name=%q flag=%v, want var=%q name=%q flag=%v", + tc.src, variable, name, flag, tc.variable, tc.name, tc.flag) + } + } +} + +// The Studio Pro activity describes as a statement that rebuilds it. +func TestLockWorkflow_DescribeKeepsTheSelection(t *testing.T) { + for _, tc := range []struct { + action microflows.MicroflowAction + want string + }{ + {µflows.LockWorkflowAction{PauseAllWorkflows: true, WorkflowVariable: "WorkflowDefinition"}, "lock workflow $WorkflowDefinition pause all;"}, + {µflows.LockWorkflowAction{WorkflowVariable: "WorkflowDefinition"}, "lock workflow $WorkflowDefinition;"}, + {µflows.LockWorkflowAction{PauseAllWorkflows: true, Workflow: "M.Approve"}, "lock workflow M.Approve pause all;"}, + {µflows.UnlockWorkflowAction{ResumeAllPausedWorkflows: true, WorkflowVariable: "WorkflowDefinition"}, "unlock workflow $WorkflowDefinition unpause all;"}, + {µflows.UnlockWorkflowAction{Workflow: "M.Approve"}, "unlock workflow M.Approve;"}, + } { + if got := formatAction(&ExecContext{}, tc.action, nil, nil); got != tc.want { + t.Errorf("describe: got %q, want %q", got, tc.want) + } + // ... and the description parses back to the same activity. + lockStmt(t, tc.want) + } +} + +// `lock workflow all` names no workflow, and there is no model for it: refused +// at check and exec rather than written into a microflow that cannot build. +func TestLockWorkflow_AllWithoutAWorkflowIsRefused(t *testing.T) { + for _, src := range []string{"lock workflow all;", "unlock workflow all;"} { + prog, errs := visitor.Build("create microflow M.MF ()\nbegin\n" + src + "\nend;") + if len(errs) > 0 { + t.Fatal(errs) + } + mf := prog.Statements[len(prog.Statements)-1].(*ast.CreateMicroflowStmt) + if vs := ValidateMicroflow(mf); !hasErrorRule(vs, lockWorkflowAllRule) { + t.Errorf("%s: check passed it: %+v", src, vs) + } + if err := validateMicroflowRules(mf); err == nil { + t.Errorf("%s: exec accepted it", src) + } + } + // CONTROL: a lock that names its workflow is clean. + prog, _ := visitor.Build("create microflow M.MF ($Wf: System.WorkflowDefinition)\nbegin\nlock workflow $Wf pause all;\nunlock workflow $Wf;\nend;") + if vs := ValidateMicroflow(prog.Statements[len(prog.Statements)-1].(*ast.CreateMicroflowStmt)); hasErrorRule(vs, lockWorkflowAllRule) { + t.Errorf("a lock naming its workflow was refused: %+v", vs) + } +} diff --git a/mdl/executor/microflow_error_handler_authoring_test.go b/mdl/executor/microflow_error_handler_authoring_test.go index 2a9442a387..0a4ad09c65 100644 --- a/mdl/executor/microflow_error_handler_authoring_test.go +++ b/mdl/executor/microflow_error_handler_authoring_test.go @@ -368,3 +368,44 @@ func containsSubstringAny(errs []string, want string) bool { } return false } + +// mendixlabs/mxcli#591: `call nanoflow … on error continue` in a nanoflow passed +// check and exec, then failed the build with CE6035. MEASURED on 11.14.0 (an +// ako/TestApp copy, one nanoflow per cell): create, commit, call nanoflow and +// call microflow each accept only a handler WITHOUT rollback in a nanoflow — +// `on error continue`, `on error rollback` and a custom handler with rollback +// are all CE6035. Retrieve and delete accepted all four, and are the control. +func TestNanoflow_RefusesAllButWithoutRollbackOnCreateCommitAndCalls(t *testing.T) { + stmts := map[string]string{ + "create": "$O = create M.Car (Brand = 'x')", + "commit": "commit $Car", + "call nanoflow": "call nanoflow M.NF_Sub()", + "call microflow": "call microflow M.MF_Sub()", + } + refused := map[string]string{ + "continue": " on error continue;", + "rollback": " on error rollback;", + "custom": " on error begin log info node 'B' 'e'; end error;", + } + for name, stmt := range stmts { + for form, clause := range refused { + if errs := nanoflowErrorsFor(t, stmt+clause); !containsSubstringAny(errs, "on error") { + t.Errorf("%s with %s was accepted in a nanoflow, but mxbuild reports CE6035: %v", name, form, errs) + } + } + // The one handler the four accept, and no clause at all. + for _, ok := range []string{" on error without rollback begin log info node 'B' 'e'; end error;", ";"} { + if errs := nanoflowErrorsFor(t, stmt+ok); containsSubstringAny(errs, "on error") { + t.Errorf("%s%s was refused, but it builds: %v", name, ok, errs) + } + } + } + // CONTROL: retrieve and delete accept every form in a nanoflow. + for _, stmt := range []string{"retrieve $L from M.Car", "delete $Car"} { + for _, clause := range refused { + if errs := nanoflowErrorsFor(t, stmt+clause); containsSubstringAny(errs, "on error") { + t.Errorf("%s%s was refused, but it builds: %v", stmt, clause, errs) + } + } + } +} diff --git a/mdl/executor/microflow_error_handling_test.go b/mdl/executor/microflow_error_handling_test.go index dcab0d12fe..3df7afec93 100644 --- a/mdl/executor/microflow_error_handling_test.go +++ b/mdl/executor/microflow_error_handling_test.go @@ -134,3 +134,43 @@ func TestMDL076_OnlyContinueIsRefused(t *testing.T) { } } } + +// mendixlabs/mxcli#175. `call workflow … on error continue` passed check and +// exec, then failed the build: CE6035 "Error handling type is not supported" at +// Call workflow activity. MEASURED on 11.14.0 (ako/TestApp, one microflow per +// clause): continue is CE6035; no clause, rollback, a custom handler and a +// custom handler without rollback all build at 0 errors. +func TestMDL076_ReportsContinueOnCallWorkflow(t *testing.T) { + v := µflowValidator{mfName: "M.ACT_X"} + v.checkErrorHandlingContinueSupported(&ast.CallWorkflowStmt{ + ErrorHandling: &ast.ErrorHandlingClause{Type: ast.ErrorHandlingContinue}, + }) + if len(v.violations) != 1 || v.violations[0].RuleID != continueUnsupportedRule { + t.Fatalf("`call workflow … on error continue` was accepted: %+v", v.violations) + } +} + +// CONTROL for the above: every other form was measured to build, so refusing +// one would reject a working script. +func TestMDL076_CallWorkflowAcceptsTheOtherClauses(t *testing.T) { + for _, eh := range []*ast.ErrorHandlingClause{ + nil, + {Type: ast.ErrorHandlingRollback}, + {Type: ast.ErrorHandlingCustom}, + {Type: ast.ErrorHandlingCustomWithoutRollback}, + } { + v := µflowValidator{mfName: "M.ACT_X"} + v.checkErrorHandlingContinueSupported(&ast.CallWorkflowStmt{ErrorHandling: eh}) + if len(v.violations) != 0 { + t.Errorf("clause %+v was reported on a call workflow: %+v", eh, v.violations) + } + } +} + +// MDL076 is a rule exec enforces, not only check: a script run with --no-check, +// through -c or the REPL wrote the CE6035 activity before. +func TestMDL076_IsExecEnforced(t *testing.T) { + if !execEnforcedMicroflowRules[continueUnsupportedRule] { + t.Fatalf("%s is reported by check but not refused by exec", continueUnsupportedRule) + } +} diff --git a/mdl/executor/nanoflow_list_error_handling_test.go b/mdl/executor/nanoflow_list_error_handling_test.go new file mode 100644 index 0000000000..cf8022e077 --- /dev/null +++ b/mdl/executor/nanoflow_list_error_handling_test.go @@ -0,0 +1,105 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// mendixlabs/mxcli#591. The list activities — create list, add to / remove from +// list, list operations, aggregates and cast — carried no error-handling type +// at all, and both writers stamped a literal "Rollback" on them. In a NANOFLOW +// that is CE6035 "Error handling type is not supported" on every one of them, +// measured on 11.14.0 (ako/TestApp copy, one nanoflow per activity): Create list, +// Change list (add and remove), Aggregate list and List operation (head, filter, +// sort) all failed. Studio Pro stores "Abort" on every action of every nanoflow +// in ako/TestApp (130 actions over 11 nanoflows), and "Rollback" on the same +// actions in a microflow — so the value is the flow flavour's default, which is +// what the builder now supplies. +func TestListActions_ErrorHandlingFollowsTheFlowFlavour(t *testing.T) { + src := `mdl 1; +create microflow M.F ($Products: List of M.Product, $P: M.Product) +begin + $L = create list of M.Product; + add $P to $L; + remove $P from $L; + $H = head $Products; + $N = count $Products; +end;` + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatal(errs) + } + body := prog.Statements[len(prog.Statements)-1].(*ast.CreateMicroflowStmt).Body + + for _, tc := range []struct { + nanoflow bool + want microflows.ErrorHandlingType + }{ + {true, microflows.ErrorHandlingTypeAbort}, + {false, microflows.ErrorHandlingTypeRollback}, + } { + fb := &flowBuilder{posX: 100, posY: 100, spacing: HorizontalSpacing, isNanoflow: tc.nanoflow} + oc := fb.buildFlowGraph(body, nil) + seen := 0 + for _, obj := range oc.Objects { + aa, ok := obj.(*microflows.ActionActivity) + if !ok { + continue + } + var got microflows.ErrorHandlingType + switch a := aa.Action.(type) { + case *microflows.CreateListAction: + got = a.ErrorHandlingType + case *microflows.ChangeListAction: + got = a.ErrorHandlingType + case *microflows.ListOperationAction: + got = a.ErrorHandlingType + case *microflows.AggregateListAction: + got = a.ErrorHandlingType + default: + continue + } + seen++ + if got != tc.want { + t.Errorf("nanoflow=%v: %T has error handling %q, want %q", tc.nanoflow, aa.Action, got, tc.want) + } + } + if seen != 5 { + t.Fatalf("nanoflow=%v: found %d list activities, want 5", tc.nanoflow, seen) + } + } +} + +// The cast activity is built by a different path from the list ones (an +// inheritance split's case body), so it is pinned on its own. +func TestCastAction_ErrorHandlingFollowsTheFlowFlavour(t *testing.T) { + for _, tc := range []struct { + nanoflow bool + want microflows.ErrorHandlingType + }{ + {true, microflows.ErrorHandlingTypeAbort}, + {false, microflows.ErrorHandlingTypeRollback}, + } { + fb := &flowBuilder{posX: 100, posY: 100, spacing: HorizontalSpacing, isNanoflow: tc.nanoflow} + oc := fb.buildFlowGraph([]ast.MicroflowStatement{&ast.CastObjectStmt{ObjectVariable: "Obj", OutputVariable: "Sub"}}, nil) + found := false + for _, obj := range oc.Objects { + if aa, ok := obj.(*microflows.ActionActivity); ok { + if c, ok := aa.Action.(*microflows.CastAction); ok { + found = true + if c.ErrorHandlingType != tc.want { + t.Errorf("nanoflow=%v: cast has error handling %q, want %q", tc.nanoflow, c.ErrorHandlingType, tc.want) + } + } + } + } + if !found { + t.Fatal("no cast action built") + } + } +} diff --git a/mdl/executor/nanoflow_validation.go b/mdl/executor/nanoflow_validation.go index bfee0f9599..2863a306e9 100644 --- a/mdl/executor/nanoflow_validation.go +++ b/mdl/executor/nanoflow_validation.go @@ -94,50 +94,12 @@ func checkDisallowedNanoflowAction(stmt ast.MicroflowStatement) string { return "" } -// getErrorHandling extracts the ErrorHandlingClause from statements that have one. -// -// Only statements reachable in nanoflows (i.e., NOT in the denylist) need coverage -// here. Disallowed actions are rejected by checkDisallowedNanoflowAction before -// this function is called. Statements like ListOperationStmt that have no -// ErrorHandling field are also omitted (they return nil implicitly via default). +// getErrorHandling extracts the ErrorHandlingClause from a statement that has +// one. It is stmtErrorHandling: a second hand-kept list of the statements that +// take a clause drifted from the first, which is how a handler body went +// unwalked here. func getErrorHandling(stmt ast.MicroflowStatement) *ast.ErrorHandlingClause { - switch s := stmt.(type) { - case *ast.CreateObjectStmt: - return s.ErrorHandling - case *ast.MfCommitStmt: - return s.ErrorHandling - case *ast.DeleteObjectStmt: - return s.ErrorHandling - case *ast.RetrieveStmt: - return s.ErrorHandling - case *ast.CallMicroflowStmt: - return s.ErrorHandling - case *ast.CallNanoflowStmt: - return s.ErrorHandling - case *ast.CallJavaScriptActionStmt: - return s.ErrorHandling - // The eight statements mendixlabs/mxcli#1078 gave an onErrorClause. None is - // on the denylist above, so all eight are reachable in a nanoflow — and - // without them here their handler BODIES are never walked, so a Java action - // or REST call nested inside `declare … on error begin … end error` would go unreported. - case *ast.DeclareStmt: - return s.ErrorHandling - case *ast.MfSetStmt: - return s.ErrorHandling - case *ast.ChangeObjectStmt: - return s.ErrorHandling - case *ast.LogStmt: - return s.ErrorHandling - case *ast.ShowPageStmt: - return s.ErrorHandling - case *ast.ClosePageStmt: - return s.ErrorHandling - case *ast.ShowMessageStmt: - return s.ErrorHandling - case *ast.ValidationFeedbackStmt: - return s.ErrorHandling - } - return nil + return stmtErrorHandling(stmt) } // validateNanoflowReturnType checks that the return type is allowed for nanoflows. @@ -241,11 +203,14 @@ func checkNanoflowErrorHandling(stmt ast.MicroflowStatement) string { keyword = "show message" case *ast.ValidationFeedbackStmt: keyword = "validation feedback" + case *ast.CreateObjectStmt, *ast.MfCommitStmt, *ast.CallNanoflowStmt, *ast.CallMicroflowStmt: + return checkNanoflowWithoutRollbackOnly(stmt) default: // declare and set are deliberately absent: both variable activities accept - // every form in a nanoflow. So do the statements that could already carry - // the clause (commit, create, retrieve, the calls) — unmeasured here, and - // refusing them would reject nanoflows that build today. + // every form in a nanoflow, and so do retrieve and delete (measured, + // mendixlabs/mxcli#591). The remaining statements that carry the clause + // are unmeasured here, and refusing them would reject nanoflows that + // build today. return "" } return "`" + keyword + " ... on error` is not supported in a nanoflow — Mendix rejects " + @@ -253,3 +218,58 @@ func checkNanoflowErrorHandling(stmt ast.MicroflowStatement) string { " there with CE6035 \"Error handling type is not supported\". Drop the clause " + "(a nanoflow activity aborts the flow on error by default)" } + +// nanoflowWithoutRollbackOnly names the nanoflow activities that accept exactly +// one error-handling clause: a custom handler WITHOUT rollback. +// +// MEASURED on Mendix 11.14.0 (an ako/TestApp copy, one nanoflow per cell), +// mendixlabs/mxcli#591: +// +// activity (in a NANOFLOW) continue rollback custom without rollback +// Create object CE6035 CE6035 CE6035 ok +// Commit CE6035 CE6035 CE6035 ok +// Call nanoflow CE6035 CE6035 CE6035 ok +// Call microflow CE6035 CE6035 CE6035 ok +// Retrieve ok ok ok ok +// Delete ok ok ok ok +// +// With no clause the activity stores the nanoflow default Abort, which builds; +// a nanoflow has no transaction for "rollback" to undo, and Studio Pro's own +// nanoflows in ako/TestApp store only Abort and CustomWithoutRollBack. +var nanoflowWithoutRollbackOnly = map[string]string{ + "create": "Create object activity", + "commit": "Commit object(s) activity", + "call nanoflow": "Nanoflow call action activity", + "call microflow": "Call microflow activity", +} + +func checkNanoflowWithoutRollbackOnly(stmt ast.MicroflowStatement) string { + eh := getErrorHandling(stmt) + if eh == nil || eh.Type == ast.ErrorHandlingCustomWithoutRollback { + return "" + } + var keyword string + switch stmt.(type) { + case *ast.CreateObjectStmt: + keyword = "create" + case *ast.MfCommitStmt: + keyword = "commit" + case *ast.CallNanoflowStmt: + keyword = "call nanoflow" + case *ast.CallMicroflowStmt: + keyword = "call microflow" + default: + return "" + } + form := "on error rollback" + switch eh.Type { + case ast.ErrorHandlingContinue: + form = "on error continue" + case ast.ErrorHandlingCustom: + form = "on error begin … end error" + } + return "`" + keyword + " ... " + form + "` is not supported in a nanoflow — Mendix rejects it on a " + + nanoflowWithoutRollbackOnly[keyword] + " there with CE6035 \"Error handling type is not supported\". " + + "A nanoflow has no transaction to roll back: drop the clause (the activity aborts the flow on error), " + + "or handle the error with `on error without rollback begin … end error`" +} diff --git a/mdl/executor/page_silent_changes_integration_test.go b/mdl/executor/page_silent_changes_integration_test.go new file mode 100644 index 0000000000..ce6f3a59b2 --- /dev/null +++ b/mdl/executor/page_silent_changes_integration_test.go @@ -0,0 +1,153 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package executor + +import ( + "strings" + "testing" +) + +// mendixlabs/mxcli#1214: `alter page … set Editable = true on ` +// printed "Altered page" and stored nothing. describe is the read the user has. +func TestAlterPage_SetListViewEditable(t *testing.T) { + env := setupTestEnv(t) + defer env.teardown() + + entity := testModule + ".EditRow" + page := testModule + ".EditRowsPage" + if err := env.executeMDL(`create or modify persistent entity ` + entity + ` (Note: String(200));`); err != nil { + t.Fatal(err) + } + if err := env.executeMDL(`create page ` + page + ` (Title: 'Rows', Layout: Atlas_Core.Atlas_Default) { + listview lvRows (DataSource: database ` + entity + `) { textbox txtNote (Attribute: Note) } + }`); err != nil { + t.Fatal(err) + } + before, err := env.describeMDL(`describe page ` + page + `;`) + if err != nil { + t.Fatal(err) + } + // Control: the list view starts read-only, so the assertion below can fail. + if strings.Contains(before, "Editable: true") { + t.Fatalf("control: list view already editable before the alter:\n%s", before) + } + if err := env.executeMDL(`alter page ` + page + ` { set (Editable: true) on lvRows };`); err != nil { + t.Fatal(err) + } + after, err := env.describeMDL(`describe page ` + page + `;`) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(after, "listview lvRows (DataSource: database from "+entity+", Editable: true)") { + t.Errorf("set Editable = true on a list view was not stored:\n%s", after) + } + // The input enum is not a list view's value: refused, not swallowed. + if err := env.executeMDL(`alter page ` + page + ` { set (Editable: true) on txtNote };`); err == nil { + t.Error("set Editable = true on a text box succeeded; its Editable is Always/Never") + } +} + +// ako/mxcli#528, mendixlabs/mxcli#293: a data view's footer is a region with no +// stored name. It is addressed as `.footer`, a replacement may restate +// the names of the widgets it removes, and describe prints the footer unnamed — +// so its output re-executes onto the same page. +func TestAlterPage_DataViewFooterRegion(t *testing.T) { + env := setupTestEnv(t) + defer env.teardown() + env.requireMinVersion(t, 11, 0) + + entity := testModule + ".FootRow" + page := testModule + ".FootRowPage" + if err := env.executeMDL(`create or modify persistent entity ` + entity + ` (Note: String(200));`); err != nil { + t.Fatal(err) + } + if err := env.executeMDL(`create page ` + page + ` (Title: 'Foot', Layout: Atlas_Core.Atlas_Default, Params: { $Row: ` + entity + ` }) { + dataview dvMain (DataSource: $Row) { + textbox txtNote (Attribute: Note) + footer footerButtons { actionbutton btnSave (Caption: 'Save', Action: save_changes) } + } + }`); err != nil { + t.Fatal(err) + } + if err := env.executeMDL(`alter page ` + page + ` { + insert into dvMain.footer { actionbutton btnCancel (Caption: 'Cancel', Action: cancel_changes) } + };`); err != nil { + t.Fatalf("insert into dvMain.footer: %v", err) + } + if err := env.executeMDL(`alter page ` + page + ` { + replace dvMain.footer with { footer { + actionbutton btnSave (Caption: 'Save it', Action: save_changes) + actionbutton btnClose (Caption: 'Close', Action: close_page) + } } + };`); err != nil { + t.Fatalf("replace dvMain.footer restating btnSave: %v", err) + } + out, err := env.describeMDL(`describe page ` + page + `;`) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(out, "footer {") || strings.Contains(out, "footer1") || strings.Contains(out, "footerButtons") { + t.Errorf("describe should print the footer unnamed:\n%s", out) + } + if !strings.Contains(out, "'Save it'") || !strings.Contains(out, "btnClose") || strings.Contains(out, "btnCancel") { + t.Errorf("the footer was not replaced as a whole:\n%s", out) + } + // The name a script wrote on the footer, and the old `footer1`, name + // nothing; the miss points at the address that works. + err = env.executeMDL(`alter page ` + page + ` { drop footer1 };`) + if err == nil || !strings.Contains(err.Error(), "dvMain.footer") { + t.Errorf("drop footer1: %v, want a not-found naming dvMain.footer", err) + } + // describe's output re-executes onto the page it came from. + if err := env.executeMDL(stripDescribeArtifacts(out)); err != nil { + t.Fatalf("re-executing describe output: %v\n%s", err, out) + } + again, err := env.describeMDL(`describe page ` + page + `;`) + if err != nil { + t.Fatal(err) + } + if again != out { + t.Errorf("describe changed after re-executing its own output:\n--- before\n%s\n--- after\n%s", out, again) + } +} + +// mendixlabs/mxcli#1235: an input bound to a page variable is written, read +// back by describe in the same spelling, and describe's output re-executes +// onto the same page. +func TestPageVariableInputBinding_RoundTrip(t *testing.T) { + env := setupTestEnv(t) + defer env.teardown() + env.requireMinVersion(t, 11, 0) + + page := testModule + ".VarBoundPage" + if err := env.executeMDL(`create page ` + page + ` ( + Title: 'Orders', Layout: Atlas_Core.Atlas_Default, + Variables: { $ShowAll: Boolean = 'true', $Filter: String = '''x''' } + ) { + checkbox cbShowAll (Label: 'Show all', Attribute: $ShowAll) + textbox tbFilter (Label: 'Filter', Attribute: $Filter) + }`); err != nil { + t.Fatalf("create: %v", err) + } + out, err := env.describeMDL(`describe page ` + page + `;`) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"Attribute: $ShowAll", "Attribute: $Filter"} { + if !strings.Contains(out, want) { + t.Errorf("describe dropped the page-variable binding %q:\n%s", want, out) + } + } + if err := env.executeMDL(stripDescribeArtifacts(out)); err != nil { + t.Fatalf("re-executing describe output: %v\n%s", err, out) + } + again, err := env.describeMDL(`describe page ` + page + `;`) + if err != nil { + t.Fatal(err) + } + if again != out { + t.Errorf("describe changed after re-executing its own output:\n--- before\n%s\n--- after\n%s", out, again) + } +} diff --git a/mdl/executor/page_variable_input_binding_test.go b/mdl/executor/page_variable_input_binding_test.go new file mode 100644 index 0000000000..d455ebe8c4 --- /dev/null +++ b/mdl/executor/page_variable_input_binding_test.go @@ -0,0 +1,152 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// mendixlabs/mxcli#1235: an input widget bound to a page variable. Studio Pro +// stores it with no AttributeRef and a SourceVariable naming the variable — +// measured on TestApp (Studio Pro 11, Pages.Probe1235_pagevar, read back over +// MCP): `sourceVariable: {Pages$PageVariable, localVariable: "ShowAll"}` and no +// attributeRef, with ped_check_errors reporting no errors. MDL refused to write +// it (MDL-WIDGET34) and describe printed the widget unbound. + +func pageVarBuilder(vars ...string) *pageBuilder { + pb := &pageBuilder{ + widgetScope: map[string]model.ID{}, + paramScope: map[string]model.ID{}, + paramEntityNames: map[string]string{}, + localVariables: map[string]bool{}, + } + pb.argCtx = atDocumentRoot() + for _, v := range vars { + pb.localVariables[v] = true + } + return pb +} + +func pageVarRef(name string) *ast.DataSourceV3 { + return &ast.DataSourceV3{Type: "parameter", Reference: "$" + name} +} + +func TestPageVariableBinding_Builders(t *testing.T) { + want := &pages.WidgetVariable{Variable: "ShowAll", Kind: "local"} + check := func(t *testing.T, path string, sv *pages.WidgetVariable) { + t.Helper() + if path != "" { + t.Errorf("AttributePath = %q, want none — a page variable is not an attribute", path) + } + if sv == nil || *sv != *want { + t.Errorf("SourceVariable = %+v, want %+v", sv, want) + } + } + for _, typ := range []string{"checkbox", "textbox", "textarea", "datepicker", "dropdown", "radiobuttons"} { + t.Run(typ, func(t *testing.T) { + pb := pageVarBuilder("ShowAll") + w := &ast.WidgetV3{Name: "in1", Type: typ, Properties: map[string]any{"Attribute": pageVarRef("ShowAll")}} + switch typ { + case "checkbox": + x, err := pb.buildCheckBoxV3(w) + if err != nil { + t.Fatal(err) + } + check(t, x.AttributePath, x.SourceVariable) + case "textbox": + x, err := pb.buildTextBoxV3(w) + if err != nil { + t.Fatal(err) + } + check(t, x.AttributePath, x.SourceVariable) + case "textarea": + x, err := pb.buildTextAreaV3(w) + if err != nil { + t.Fatal(err) + } + check(t, x.AttributePath, x.SourceVariable) + case "datepicker": + x, err := pb.buildDatePickerV3(w) + if err != nil { + t.Fatal(err) + } + check(t, x.AttributePath, x.SourceVariable) + case "dropdown": + x, err := pb.buildDropdownV3(w) + if err != nil { + t.Fatal(err) + } + check(t, x.AttributePath, x.SourceVariable) + case "radiobuttons": + x, err := pb.buildRadioButtonsV3(w) + if err != nil { + t.Fatal(err) + } + check(t, x.AttributePath, x.SourceVariable) + } + }) + } +} + +// A `$name` that is not a page variable is still refused — writing it would +// store an input with no binding at all. +func TestPageVariableBinding_UndeclaredIsRefused(t *testing.T) { + pb := pageVarBuilder("ShowAll") + w := &ast.WidgetV3{Name: "cb", Type: "checkbox", Properties: map[string]any{"Attribute": pageVarRef("Other")}} + if _, err := pb.buildCheckBoxV3(w); err == nil { + t.Fatal("checkbox bound to an undeclared $Other was built; want a refusal") + } +} + +// check-time: the page's own Variables decide, with no project needed. +func TestPageVariableBinding_Check(t *testing.T) { + page := func(attr string) *ast.CreatePageStmtV3 { + return &ast.CreatePageStmtV3{ + Name: ast.QualifiedName{Module: "M", Name: "P"}, + Variables: []ast.PageVariable{{Name: "ShowAll", DataType: "Boolean", DefaultValue: "true"}}, + Widgets: []*ast.WidgetV3{{Name: "cb", Type: "checkbox", Properties: map[string]any{ + "Label": "Show all", "Attribute": pageVarRef(attr)}}}, + } + } + reg, err := NewWidgetRegistry() + if err != nil { + t.Fatal(err) + } + for _, v := range ValidateWidgetPropertiesForStatement(page("ShowAll"), reg) { + if v.RuleID == "MDL-WIDGET34" { + t.Errorf("a declared page variable was refused: %s", v.Message) + } + } + found := false + for _, v := range ValidateWidgetPropertiesForStatement(page("Other"), reg) { + if v.RuleID == "MDL-WIDGET34" { + found = true + } + } + if !found { + t.Error("an undeclared $Other passed check; exec refuses it") + } +} + +// describe reads the binding back in the spelling the builder takes. +func TestPageVariableBinding_Describe(t *testing.T) { + w := map[string]any{ + "$Type": "Forms$CheckBox", + "Name": "cbShowAll", + "AttributeRef": nil, + "SourceVariable": map[string]any{"$Type": "Forms$PageVariable", "LocalVariable": "ShowAll", "PageParameter": "", "Widget": ""}, + } + if got := extractInputAttribute(nil, w); got != "$ShowAll" { + t.Errorf("describe binding = %q, want $ShowAll", got) + } + // Control: an input bound to its data context still reads as before. + bound := map[string]any{"AttributeRef": map[string]any{"Attribute": "M.E.Name"}, "SourceVariable": nil} + if got := extractInputAttribute(nil, bound); strings.HasPrefix(got, "$") { + t.Errorf("a data-context binding described as %q", got) + } +} diff --git a/mdl/executor/roundtrip_microflow_test.go b/mdl/executor/roundtrip_microflow_test.go index a6359ddaa5..07ed281e85 100644 --- a/mdl/executor/roundtrip_microflow_test.go +++ b/mdl/executor/roundtrip_microflow_test.go @@ -681,12 +681,14 @@ func TestRoundtripMicroflow_ErrorHandlingContinue(t *testing.T) { createMDL := `create microflow ` + mfName + ` () returns Boolean begin $Obj = create RoundtripTest.MfCommitItem; - commit $Obj on error continue; + delete $Obj on error continue; return true; end;` + // delete, not commit: `commit … on error continue` is CE6035 (MDL076), + // which exec now refuses as check always did (mendixlabs/mxcli#175). assertMicroflowContains(t, env, mfName, createMDL, - []string{"commit", "on error continue", "return"}, + []string{"delete", "on error continue", "return"}, nil, ) } diff --git a/mdl/executor/roundtrip_nanoflow_test.go b/mdl/executor/roundtrip_nanoflow_test.go index 2f750d5c5e..47d4555962 100644 --- a/mdl/executor/roundtrip_nanoflow_test.go +++ b/mdl/executor/roundtrip_nanoflow_test.go @@ -239,12 +239,16 @@ end;` nfName := testModule + ".RT_NF_ErrorHandling" createMDL := `create nanoflow ` + nfName + ` () returns Boolean begin - $Result = call microflow ` + mfName + ` () on error continue; + $Result = call microflow ` + mfName + ` () on error without rollback begin + log error node 'RT' 'call failed'; + end error; return $Result; end;` + // A call in a nanoflow takes only a handler without rollback; `on error + // continue` there is CE6035 and refused (mendixlabs/mxcli#591). assertNanoflowContains(t, env, nfName, createMDL, - []string{"nanoflow", "call microflow", "on error continue", "return"}, + []string{"nanoflow", "call microflow", "on error without rollback", "return"}, nil, ) } diff --git a/mdl/executor/validate.go b/mdl/executor/validate.go index 1926355eb1..f247213d60 100644 --- a/mdl/executor/validate.go +++ b/mdl/executor/validate.go @@ -15,6 +15,7 @@ import ( "github.com/mendixlabs/mxcli/mdl/linter" "github.com/mendixlabs/mxcli/model" "github.com/mendixlabs/mxcli/sdk/domainmodel" + "github.com/mendixlabs/mxcli/sdk/javaactions" ) // scriptContext holds objects defined within a script for reference validation. @@ -36,6 +37,23 @@ type scriptContext struct { menus map[string]bool // Menu documents created (Module.Menu) constants map[string]bool // Constants created (Module.Constant) workflows map[string]bool // Workflows created (Module.Workflow) + // Task queues created (Module.Queue, lower-cased — a queue is resolved + // case-insensitively, like buildQueueQualifiedNames). Missing, a queue the + // script creates was "not found" for every `in queue` call to it + // (mendixlabs/mxcli#1211). + queues map[string]bool + + // Microflow-typed parameters of the Java actions the script creates + // (Module.Action -> parameter name). Such a parameter takes a microflow + // NAME, never an expression (mendixlabs/mxcli#1210). + javaActionFlowParams map[string]map[string]bool + + // Entities the script declares, and those it alters, for resolving the + // bare members of a retrieve constraint on an entity the project does not + // have yet (mendixlabs/mxcli#1213). An altered entity's member list is + // not the declaration's, so it is not judged from it. + entityDecls map[string]*ast.CreateEntityStmt + alteredEntities map[string]bool // Java/JavaScript actions created in the script, mapped to their declared // parameter names. A bool would be enough to stop the false "not found", @@ -93,6 +111,11 @@ func newScriptContext() *scriptContext { layouts: make(map[string]bool), menus: make(map[string]bool), constants: make(map[string]bool), + queues: make(map[string]bool), + + javaActionFlowParams: make(map[string]map[string]bool), + entityDecls: make(map[string]*ast.CreateEntityStmt), + alteredEntities: make(map[string]bool), javaActions: make(map[string][]string), javaScriptActions: make(map[string][]string), @@ -172,7 +195,10 @@ func (sc *scriptContext) collectSingle(stmt ast.Statement) { sc.entities[s.Name.String()] = true sc.recordEntityAttrs(s) sc.recordEntityGeneralization(s) + sc.entityDecls[s.Name.String()] = s } + case *ast.AlterEntityStmt: + sc.alteredEntities[s.Name.String()] = true case *ast.CreateAssociationStmt: sc.recordAssociation(s) case *ast.CreateViewEntityStmt: @@ -226,6 +252,11 @@ func (sc *scriptContext) collectSingle(stmt ast.Statement) { case *ast.CreateJavaActionStmt: if s.Name.Module != "" { sc.javaActions[s.Name.String()] = codeActionParamNames(s.Parameters) + sc.javaActionFlowParams[s.Name.String()] = microflowTypedParams(s.Parameters) + } + case *ast.CreateQueueStmt: + if s.Name.Module != "" { + sc.queues[strings.ToLower(s.Name.String())] = true } case *ast.CreateJavaScriptActionStmt: if s.Name.Module != "" { @@ -333,6 +364,9 @@ func validateProgramWithWarnings(ctx *ExecContext, prog *ast.Program) ([]error, } } errors = append(errors, validateForwardPageRefs(ctx, prog)...) + // The same for the other references exec resolves in statement order, where + // the later definition is a `create or modify` (mendixlabs/mxcli#1211, #1212). + errors = append(errors, validateForwardDefRefs(ctx, prog)...) // Resolve icon-collection references. Needs the project (the collections // are documents in it), so it belongs here rather than in the no-project // pass — MxBuild otherwise reports the typo as CE1613. @@ -1111,7 +1145,7 @@ func validateFlowBodyReferences(ctx *ExecContext, body []ast.MicroflowStatement, if len(refs.queues) > 0 { known := buildQueueQualifiedNames(ctx) for _, ref := range refs.queues { - if !known[strings.ToLower(ref)] { + if !known[strings.ToLower(ref)] && !sc.queues[strings.ToLower(ref)] { errors = append(errors, fmt.Sprintf("task queue not found: %s (referenced by in queue)", ref)) } } @@ -1159,6 +1193,7 @@ func validateFlowBodyReferences(ctx *ExecContext, body []ast.MicroflowStatement, // output. mxcli-chat FINDINGS §37. if declared, inScript := sc.javaActions[ref.name]; inScript { errors = append(errors, validateCodeActionParams("java action", ref, declared)...) + errors = append(errors, microflowParamArgErrors(ref, sc.javaActionFlowParams[ref.name])...) continue } if !known[ref.name] { @@ -1167,10 +1202,15 @@ func validateFlowBodyReferences(ctx *ExecContext, body []ast.MicroflowStatement, } if ja, err := ctx.Backend.ReadJavaActionByName(ref.name); err == nil && ja != nil { var declared []string + flowParams := map[string]bool{} for _, p := range ja.Parameters { declared = append(declared, p.Name) + if _, isFlow := p.ParameterType.(*javaactions.MicroflowType); isFlow { + flowParams[p.Name] = true + } } errors = append(errors, validateCodeActionParams("java action", ref, declared)...) + errors = append(errors, microflowParamArgErrors(ref, flowParams)...) } } } @@ -1211,6 +1251,7 @@ func validateFlowBodyReferences(ctx *ExecContext, body []ast.MicroflowStatement, if len(refs.retrieves) > 0 { errors = append(errors, validateRetrieveConstraints(ctx, refs.retrieves)...) errors = append(errors, validateXPathAssociations(ctx, refs.retrieves, sc)...) + errors = append(errors, validateRetrieveMembers(ctx, refs.retrieves, sc)...) } return errors @@ -1333,6 +1374,7 @@ type flowRefCollector struct { type codeActionCallRef struct { name string argNames []string + args []ast.CallArgument } // callArgNames extracts the written parameter names from a code-action call's @@ -1397,6 +1439,7 @@ type entityRef struct { type retrieveConstraintRef struct { entity string // entity qualified name (database retrieve only) constraint string // bracketed XPath constraint, e.g. "[System.owner = '[%CurrentUser%]']" + stored string // the constraint as the flow builder writes it (retrieveXPathConstraint) } // addQueue records an `IN QUEUE Module.Name` target. A queue that does not exist @@ -1434,7 +1477,7 @@ func (c *flowRefCollector) collectFromStatements(stmts []ast.MicroflowStatement) case *ast.CallJavaActionStmt: if s.ActionName.Module != "" { c.javaActions = append(c.javaActions, codeActionCallRef{ - name: s.ActionName.String(), argNames: callArgNames(s.Arguments), + name: s.ActionName.String(), argNames: callArgNames(s.Arguments), args: s.Arguments, }) } c.addQueue(s.Queue) @@ -1460,6 +1503,7 @@ func (c *flowRefCollector) collectFromStatements(stmts []ast.MicroflowStatement) c.retrieves = append(c.retrieves, retrieveConstraintRef{ entity: s.Source.String(), constraint: expressionToXPath(s.Where), + stored: retrieveXPathConstraint(s.Where, s.Source.String()), }) } } @@ -1560,6 +1604,10 @@ var execEnforcedMicroflowRules = map[string]bool{ "MDL047": true, "MDL048": true, "MDL055": true, + // MDL091: `startsWith()` / `endsWith()` in a retrieve constraint are + // CE0161, measured on mxbuild 11.13.0 against `starts-with()` as the + // control (mendixlabs/mxcli#1213). + "MDL091": true, // MDL057: `synchronize` in a microflow is CE0009 at build time, verified on // mxbuild 11.13.0 — the same class of "check caught it, exec did not" gap // that #833 was about. @@ -1577,9 +1625,20 @@ var execEnforcedMicroflowRules = map[string]bool{ // otherwise `check` catches the typo and the write that follows does not. "MDL059": true, "MDL060": true, + // MDL092: an @anchor parameter the visitor cannot use is dropped, the edge + // keeping its default sides — the same class as MDL060 (mendixlabs/mxcli#992). + "MDL092": true, + // MDL076: `on error continue` on an activity that rejects it is CE6035 at + // build time — every row of continueUnsupportedOn was measured on 11.14.0. + // check reported it; exec without the pre-check (-c, the REPL, --no-check) + // wrote it (mendixlabs/mxcli#175). + "MDL076": true, // MDL-WF16: a notify workflow with no target is CE0166 at build time, // measured on the 11.6, 11.10 and 11.13 mxbuilds. "MDL-WF16": true, + // MDL-WF17: `lock workflow all` / `unlock workflow all` is CE1825, measured + // on 11.13.0 and 11.14.0 (mendixlabs/mxcli#870). + "MDL-WF17": true, } // validateMicroflowRules runs the MDL0xx microflow rule set (ValidateMicroflow) @@ -1674,6 +1733,17 @@ func documentWidgets(stmt ast.Statement) (label string, widgets []*ast.WidgetV3, return "", nil, false } +// documentVariables is a page's or snippet's `Variables:` declarations. +func documentVariables(stmt ast.Statement) []ast.PageVariable { + switch s := stmt.(type) { + case *ast.CreatePageStmtV3: + return s.Variables + case *ast.CreateSnippetStmtV3: + return s.Variables + } + return nil +} + // validateViewEntityAttributeSet reports an ADD/DROP ATTRIBUTE whose target is a // view entity — one the script creates, or one already in the project. An entity // the script (re)creates as anything else is judged by that statement instead. diff --git a/mdl/executor/validate_create_shape.go b/mdl/executor/validate_create_shape.go index 562aa7205b..7a3da881cd 100644 --- a/mdl/executor/validate_create_shape.go +++ b/mdl/executor/validate_create_shape.go @@ -45,6 +45,10 @@ func ValidateCreateIsQualified(prog *ast.Program) []linter.Violation { continue } docType, name, _ := stmtCreateInfo(stmt) + // A demo user is named by its user name, which has no module. + if docType == "demo-user" { + continue + } if docType == "" || name == "" || strings.Contains(name, ".") { continue } diff --git a/mdl/executor/validate_definition_order.go b/mdl/executor/validate_definition_order.go index 7377f3f1f6..395afa6e9d 100644 --- a/mdl/executor/validate_definition_order.go +++ b/mdl/executor/validate_definition_order.go @@ -23,6 +23,7 @@ package executor import ( "fmt" + "strings" "github.com/mendixlabs/mxcli/mdl/ast" "github.com/mendixlabs/mxcli/mdl/linter" @@ -36,6 +37,7 @@ const ( defMicroflow = "microflow" defNanoflow = "nanoflow" defPage = "page" + defQueue = "task queue" ) // defKey identifies a document by kind and qualified name. Kind is part of the @@ -156,6 +158,8 @@ func definedBy(stmt ast.Statement) (key defKey, plain bool) { return qualifiedKey(defNanoflow, s.Name), !s.CreateOrModify case *ast.CreatePageStmtV3: return qualifiedKey(defPage, s.Name), !s.IsModify && !s.IsReplace + case *ast.CreateQueueStmt: + return qualifiedKey(defQueue, s.Name), !s.CreateOrModify } return defKey{}, false } @@ -217,6 +221,16 @@ func eagerDefRefs(stmt ast.Statement) []defRef { } return out + case *ast.CreatePageStmtV3: + if s.Excluded { + // An excluded page may name a flow the project lacks; the builder + // writes it by name (danglingRefOK), so the order does not fail. + return nil + } + return widgetFlowDefRefs("page "+s.Name.String(), allPageWidgets(s)) + case *ast.CreateSnippetStmtV3: + return widgetFlowDefRefs("snippet "+s.Name.String(), s.Widgets) + case *ast.GrantEntityAccessStmt: return oneDefRef(s.Entity, defEntity, "grant on entity "+s.Entity.String()) case *ast.GrantMicroflowAccessStmt: @@ -250,8 +264,45 @@ func flowDefRefs(site string, params []ast.MicroflowParam, ret *ast.MicroflowRet switch c := stmt.(type) { case *ast.CallMicroflowStmt: out = append(out, oneDefRef(c.MicroflowName, defMicroflow, site)...) + out = append(out, queueDefRef(c.Queue, site)...) case *ast.CallNanoflowStmt: out = append(out, oneDefRef(c.NanoflowName, defNanoflow, site)...) + case *ast.CallJavaActionStmt: + // The action itself resolves lazily (measured, #955); its IN QUEUE + // does not — buildQueueSettings refuses a queue it cannot find. + out = append(out, queueDefRef(c.Queue, site)...) + } + } + return out +} + +// queueDefRef is an `IN QUEUE` target, which the flow builder resolves when it +// writes the call (buildQueueSettings) — so a queue created further down the +// script fails the flow (mendixlabs/mxcli#1211). +func queueDefRef(q *ast.QualifiedName, site string) []defRef { + if q == nil { + return nil + } + return oneDefRef(*q, defQueue, site+": in queue") +} + +// widgetFlowDefRefs is the microflows and nanoflows a page or snippet's widgets +// name — action targets and data sources. The page builder resolves each when +// it writes the page, so one created further down the script stops exec at the +// page, after everything before it was written (mendixlabs/mxcli#1212). +func widgetFlowDefRefs(site string, widgets []*ast.WidgetV3) []defRef { + refs := &widgetRefCollector{} + refs.collectFromWidgets(widgets) + refs.dedupe() + var out []defRef + for _, mf := range refs.microflows { + if mf = unquoteQualifiedName(mf); strings.Contains(mf, ".") { + out = append(out, defRef{name: mf, kinds: []string{defMicroflow}, site: site}) + } + } + for _, nf := range refs.nanoflows { + if nf = unquoteQualifiedName(nf); strings.Contains(nf, ".") { + out = append(out, defRef{name: nf, kinds: []string{defNanoflow}, site: site}) } } return out @@ -326,3 +377,101 @@ func errorHandlerBody(stmt ast.MicroflowStatement) []ast.MicroflowStatement { } return nil } + +// validateForwardDefRefs is MDL-ORDER01's project-tier half. MDL-ORDER01 needs +// no project because it only judges a reference whose later definition is a +// PLAIN create — which asserts the document is not in the project yet. A later +// `create or modify` asserts nothing, so whether the earlier reference resolves +// depends on the project: when it has the document, exec resolves against it; +// when it does not, exec fails at the referring statement with the statements +// before it already written, while the whole-script reference pass, which +// counts every name the script defines anywhere, reported it clean. +// +// The references are eagerDefRefs — the same index MDL-ORDER01 walks — so the +// two tiers cannot disagree about what is resolved eagerly. +func validateForwardDefRefs(ctx *ExecContext, prog *ast.Program) []error { + if prog == nil || !ctx.Connected() { + return nil + } + plain := plainCreateIndex(prog) + anyCreate := map[defKey]int{} + for i, stmt := range prog.Statements { + if key, _ := definedBy(stmt); key.name != "" { + if _, seen := anyCreate[key]; !seen { + anyCreate[key] = i + } + } + } + if len(anyCreate) == 0 { + return nil + } + stored := map[string]map[string]bool{} + storedOf := func(kind string) map[string]bool { + if m, ok := stored[kind]; ok { + return m + } + var m map[string]bool + switch kind { + case defEntity: + m = buildEntityQualifiedNames(ctx) + case defEnumeration: + m = buildEnumerationQualifiedNames(ctx) + case defMicroflow: + m = buildMicroflowQualifiedNames(ctx) + case defNanoflow: + m = buildNanoflowQualifiedNames(ctx) + case defPage: + m = buildPageQualifiedNames(ctx) + case defQueue: + m = buildQueueQualifiedNames(ctx) + } + stored[kind] = m + return m + } + inProject := func(kind, name string) bool { + m := storedOf(kind) + // An empty listing is a backend that could not answer, as often as a + // project without any — silence, not a finding (cf. checkAlterTarget). + // Queues are the exception: a project with none is ordinary. + if len(m) == 0 && kind != defQueue { + return true + } + if kind == defQueue { + return m[strings.ToLower(name)] + } + return m[name] + } + + created := map[defKey]bool{} + var errs []error + for i, stmt := range prog.Statements { + for _, ref := range eagerDefRefs(stmt) { + resolved := false + for _, k := range ref.kinds { + if created[defKey{kind: k, name: ref.name}] || inProject(k, ref.name) { + resolved = true + break + } + } + if resolved { + continue + } + for _, k := range ref.kinds { + key := defKey{kind: k, name: ref.name} + at, later := anyCreate[key] + if !later || at <= i { + continue + } + if _, isPlain := plain[key]; isPlain { + break // MDL-ORDER01 reports it, without a project + } + errs = append(errs, fmt.Errorf( + "statement %d: %s references %s %s before it is created (statement %d), and the project does not have it — exec resolves this in statement order, so it fails here with the statements before it already written. Move the create statement for %s above this one", + i+1, ref.site, k, ref.name, at+1, ref.name)) + break + } + } + markCreated(created, stmt) + } + return errs +} diff --git a/mdl/executor/validate_duplicates.go b/mdl/executor/validate_duplicates.go index bce92772bf..55858b7169 100644 --- a/mdl/executor/validate_duplicates.go +++ b/mdl/executor/validate_duplicates.go @@ -163,6 +163,12 @@ func stmtCreateKind(stmt ast.Statement) (docType, name string, idempotent bool) return "agent", s.Name.String(), s.CreateOrModify case *ast.CreateImageCollectionStmt: return "image-collection", s.Name.String(), s.CreateOrModify + case *ast.CreateDemoUserStmt: + // Not a document either, and like a module role it was missed by the + // document sweep: exec refuses a plain CREATE of a user that exists + // ("demo user already exists"), and check passed it (ako/mxcli#906). + // The name is the user name, which is not module-qualified. + return "demo-user", s.UserName, s.CreateOrModify } return "", "", false } @@ -223,6 +229,8 @@ func stmtDropInfo(stmt ast.Statement) (docType, name string) { return "agent", s.Name.String() case *ast.DropImageCollectionStmt: return "image-collection", s.Name.String() + case *ast.DropDemoUserStmt: + return "demo-user", s.UserName } return "", "" } @@ -276,6 +284,8 @@ func friendlyDocType(docType string) string { return "javascript action" case "module-role": return "module role" + case "demo-user": + return "demo user" case "json-structure": return "JSON structure" case "knowledge-base": @@ -392,6 +402,7 @@ type projectNameSets struct { rules map[string]bool javaScriptActs map[string]bool moduleRoles map[string]bool + demoUsers map[string]bool } // projectSetFor returns the existence set for the given doc-type key, or nil @@ -448,6 +459,8 @@ func (ps *projectNameSets) setFor(docType string) map[string]bool { return ps.javaScriptActs case "module-role": return ps.moduleRoles + case "demo-user": + return ps.demoUsers } // "module" is deliberately absent: CREATE MODULE on an existing module is a // no-op that prints "already exists" and exits 0, so `create module M;` is @@ -590,6 +603,16 @@ func loadProjectNameSets(ctx *ExecContext) *projectNameSets { // Module roles ps.moduleRoles = buildModuleRoleQualifiedNames(ctx) + // Demo users: the same lookup execCreateDemoUser refuses on. + ps.demoUsers = make(map[string]bool) + if sec, err := ctx.Backend.GetProjectSecurity(); err == nil && sec != nil { + for _, du := range sec.DemoUsers { + if du != nil { + ps.demoUsers[du.UserName] = true + } + } + } + // Image collections ps.imageCollections = make(map[string]bool) if ics, err := ctx.Backend.ListImageCollections(); err == nil { @@ -682,5 +705,8 @@ func CheckProjectConflicts(ctx *ExecContext, prog *ast.Program) []error { // A create over a name another kind already has (ako/mxcli#793). errs = append(errs, CheckProjectNameClashes(ctx, prog)...) + // The refusals exec decides from project state other than a document + // listing: jar dependencies, translations (ako/mxcli#906). + errs = append(errs, CheckExecRefusals(ctx, prog)...) return errs } diff --git a/mdl/executor/validate_exec_refusals.go b/mdl/executor/validate_exec_refusals.go new file mode 100644 index 0000000000..a71db3de87 --- /dev/null +++ b/mdl/executor/validate_exec_refusals.go @@ -0,0 +1,178 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/translations" + "github.com/mendixlabs/mxcli/mdl/types" +) + +// Refusals exec makes from project state that is not a document listing, and +// that `check -p` therefore did not predict (ako/mxcli#906). check predicted +// "already exists" for every document kind, because those go through one +// registry (CheckProjectConflicts); these each had their own lookup inside the +// exec handler, so check passed a statement exec then refused: +// +// - `alter module … add jar dependency` for a coordinate the module has +// (formula1, every run after the first); +// - `create … translations … for ` (captrack, every run); +// - `create demo user` for a user that exists — that one is a create, and now +// goes through the registry like the other creates (stmtCreateKind). +// +// What keeps check and exec from disagreeing again is that the prediction CALLS +// exec's decision instead of restating it: the jar actions are replayed through +// applyJarDepAction on a copy of the stored settings, and the translations +// refusal is translationsRefusal, which execCreateTranslations calls too. A +// prediction is only made while the stored project is what exec will see; once +// an earlier statement of the script changes that state, check stays silent +// rather than guess. + +// CheckExecRefusals returns the refusals exec would make for the statements of +// prog, given the connected project. +func CheckExecRefusals(ctx *ExecContext, prog *ast.Program) []error { + if prog == nil || !ctx.Connected() { + return nil + } + var errs []error + errs = append(errs, checkJarDependencyActions(ctx, prog)...) + errs = append(errs, checkTranslationTargets(ctx, prog)...) + return errs +} + +// checkJarDependencyActions replays every `alter module … jar dependency` +// statement against a copy of the stored module settings, in script order, with +// the function exec applies them with. A refusal is reported once per module: +// after it exec has stopped, and what the module holds is no longer known. +func checkJarDependencyActions(ctx *ExecContext, prog *ast.Program) []error { + settings := map[string]*types.ModuleSettings{} + unknown := map[string]bool{} // modules whose state the script makes unpredictable + var errs []error + for i, stmt := range prog.Statements { + switch s := stmt.(type) { + case *ast.CreateModuleStmt: + unknown[s.Name] = true + case *ast.DropModuleStmt: + unknown[s.Name] = true + case *ast.AlterModuleJarDepStmt: + if unknown[s.ModuleName] { + continue + } + ms := settings[s.ModuleName] + if ms == nil { + module, err := ctx.Backend.GetModuleByName(s.ModuleName) + if err != nil || module == nil { + unknown[s.ModuleName] = true // a missing module is reported as itself + continue + } + stored, err := ctx.Backend.GetModuleSettings(module.ID) + if err != nil || stored == nil { + unknown[s.ModuleName] = true + continue + } + ms = cloneJarDependencies(stored) + settings[s.ModuleName] = ms + } + for _, action := range s.Actions { + if err := applyJarDepAction(ms, action, s.ModuleName); err != nil { + errs = append(errs, fmt.Errorf("statement %d: %w", i+1, err)) + unknown[s.ModuleName] = true + break + } + } + } + } + return errs +} + +// cloneJarDependencies copies the part of the settings applyJarDepAction +// mutates, so the prediction never touches what the backend handed out. +func cloneJarDependencies(ms *types.ModuleSettings) *types.ModuleSettings { + out := *ms + out.JarDependencies = make([]*types.JarDependency, 0, len(ms.JarDependencies)) + for _, d := range ms.JarDependencies { + if d == nil { + continue + } + c := *d + c.Exclusions = make([]*types.JarDependencyExclusion, 0, len(d.Exclusions)) + for _, e := range d.Exclusions { + if e != nil { + ec := *e + c.Exclusions = append(c.Exclusions, &ec) + } + } + out.JarDependencies = append(out.JarDependencies, &c) + } + return &out +} + +// checkTranslationTargets reports a translations statement exec refuses. Once +// the script changes the language settings the stored source language is no +// longer what exec sees, and once it has written translations for a language a +// later plain CREATE of that language is decided by the script, not the +// project — the prediction stops at both. +func checkTranslationTargets(ctx *ExecContext, prog *ast.Program) []error { + src := sourceLanguage(ctx) + if src == "" { + return nil + } + written := map[string]bool{} + var errs []error + for i, stmt := range prog.Statements { + switch s := stmt.(type) { + case *ast.AlterSettingsStmt: + if strings.EqualFold(s.Section, "LANGUAGE") { + return errs + } + case *ast.CreateTranslationsStmt: + lang := strings.ToLower(s.Language) + if written[lang] { + continue + } + written[lang] = true + scope, err := translationScope(ctx, s.Module) + if err != nil { + continue // a missing module is reported by exec as itself + } + if err := translationsRefusal(ctx, s, src, scope); err != nil { + errs = append(errs, fmt.Errorf("statement %d: %w", i+1, err)) + } + } + } + return errs +} + +// translationsRefusal is the decision execCreateTranslations makes before it +// writes anything: no translations into the source language, and a plain CREATE +// does not overwrite a language that already has translations. check -p calls +// it too, so the two cannot disagree. +func translationsRefusal(ctx *ExecContext, s *ast.CreateTranslationsStmt, src string, scope translations.Scope) error { + if strings.EqualFold(s.Language, src) { + return mdlerrors.NewValidationf( + "%s is the project's source language — writing translations into it would "+ + "overwrite the strings the rest of the model is keyed on", s.Language) + } + if s.Mode != ast.TranslationsCreate { + return nil + } + // The language is the thing that exists, so bare CREATE refuses when it + // already has translations — the same contract every other CREATE has. + existing, err := translations.Languages(ctx.Backend, scope) + if err != nil { + return mdlerrors.NewBackend("read languages", err) + } + for _, l := range existing { + if strings.EqualFold(l, s.Language) { + return mdlerrors.NewValidationf( + "%s already has translations — use `create or modify translations` to "+ + "merge these in, or `create or replace translations` to make this file "+ + "authoritative (which REMOVES translations it does not name)", s.Language) + } + } + return nil +} diff --git a/mdl/executor/validate_microflow.go b/mdl/executor/validate_microflow.go index 5902bbfec8..f888c406f5 100644 --- a/mdl/executor/validate_microflow.go +++ b/mdl/executor/validate_microflow.go @@ -215,7 +215,21 @@ func (v *microflowValidator) walkBody(body []ast.MicroflowStatement) { v.checkErrorHandlingContinueSupported(s) v.checkErrorHandlingSupported(s) v.checkStmtExprFunctions(s) + // An annotation inside `on error begin … end error` is honoured like one + // outside it, so a malformed one there must be refused too — this walk + // never entered a handler body (mendixlabs/mxcli#991). + if eh := stmtErrorHandling(s); eh != nil { + v.walkAnnotations(eh.Body) + } switch stmt := s.(type) { + case *ast.LockWorkflowStmt: + if stmt.WorkflowVariable == "" && stmt.Workflow == "" { + v.refuseWorkflowAll("lock", "pause", "a Lock") + } + case *ast.UnlockWorkflowStmt: + if stmt.WorkflowVariable == "" && stmt.Workflow == "" { + v.refuseWorkflowAll("unlock", "unpause", "an Unlock") + } case *ast.NotifyWorkflowStmt: // MDL-WF16. A notify reaches one named element of the workflow, and the // build refuses one that names none: CE0166 "The 'Target' property is @@ -361,6 +375,7 @@ func (v *microflowValidator) walkBody(body []ast.MicroflowStatement) { if stmt.Where != nil { xp := expressionToXPath(stmt.Where) v.checkXPathAssociationEmpty(stmt.Variable, xp) + v.checkXPathFunctionNames(stmt.Variable, xp) v.checkXPathIdConstraint(stmt.Variable, xp) v.checkXPathVariableTraversal(stmt.Variable, xp) } @@ -674,36 +689,99 @@ func exprHasSlashDivision(expr ast.Expression) bool { } // xpathAssocEmptyRe matches a module-qualified association compared directly to -// `empty` in an XPath constraint (`Ledger.Transaction_Category = empty`). The -// leading boundary class excludes a `/` (so an attribute-over-association path -// like `Assoc/Ledger.Category = empty` is NOT matched — that is a valid -// attribute nullability test) and a `.`/word char (so it captures the whole -// qualified name, not the tail of a 3-part enum literal). -var xpathAssocEmptyRe = regexp.MustCompile(`(^|[^\w./])([A-Za-z_]\w*\.[A-Za-z_]\w*)\s*=\s*empty\b`) - -// xpathAssociationEmptyMatches returns the module-qualified association names an -// XPath constraint compares directly to `empty` (`Ledger.Transaction_Category = -// empty`). Shared by the microflow-retrieve check (MDL047) and the page/widget -// datasource check. Empty result → nothing to flag. -func xpathAssociationEmptyMatches(xpath string) []string { - var out []string - for _, m := range xpathAssocEmptyRe.FindAllStringSubmatch(xpath, -1) { - out = append(out, m[2]) +// `empty` in an XPath constraint (`Ledger.Transaction_Category = empty`, or +// `!= empty`). The leading boundary class excludes a `/` (so an +// attribute-over-association path like `Assoc/Ledger.Category = empty` is NOT +// matched — that is a valid attribute nullability test) and a `.`/word char (so +// it captures the whole qualified name, not the tail of a 3-part enum literal). +var xpathAssocEmptyRe = regexp.MustCompile(`(^|[^\w./])([A-Za-z_]\w*\.[A-Za-z_]\w*)\s*(!=|=)\s*empty\b`) + +// xpathAssocEmptyHit is one association compared directly to `empty`. +type xpathAssocEmptyHit struct { + Assoc string // Module.Association + Op string // "=" or "!=" +} + +// test renders the comparison as written, for the message. +func (h xpathAssocEmptyHit) test() string { return h.Assoc + " " + h.Op + " empty" } + +// fix is the spelling Mendix XPath accepts: a path to the associated object +// tests its presence, `not(…)` around it its absence. +func (h xpathAssocEmptyHit) fix() string { + if h.Op == "!=" { + return fmt.Sprintf("Test for the presence of the associated object with a path: `[%s/]`.", h.Assoc) + } + return fmt.Sprintf("Test for the absence of the associated object with negation: `[not(%s/)]`.", h.Assoc) +} + +// xpathAssociationEmptyMatches returns the module-qualified associations an +// XPath constraint compares directly to `empty`. Shared by the +// microflow-retrieve check (MDL047) and the page/widget datasource check. +// Empty result → nothing to flag. +// +// `!= empty` was missed: only `=` was matched, and measured on mxbuild 11.13.0 +// `[Mod.Order_Customer != empty]` is CE0161 as well — in a retrieve and in a +// list view's database source — while `[Mod.Order_Customer/Mod.Customer]` +// builds clean (mendixlabs/mxcli#1213). +func xpathAssociationEmptyMatches(xpath string) []xpathAssocEmptyHit { + var out []xpathAssocEmptyHit + for _, m := range xpathAssocEmptyRe.FindAllStringSubmatch(blankXPathLiterals(xpath), -1) { + out = append(out, xpathAssocEmptyHit{Assoc: m[2], Op: m[3]}) } return out } -// checkXPathAssociationEmpty flags `[Module.Association = empty]` in a retrieve -// constraint. Mendix XPath has no `= empty` test for an association — it fails -// the build with CE0161; the nullability test is `not(Module.Association/Module.Target)`. -// A bare attribute (`Name = empty`) is valid and is not module-qualified, so it -// never matches. (ledger finding #25) +// checkXPathAssociationEmpty flags `[Module.Association = empty]` (and `!=`) in a +// retrieve constraint. Mendix XPath has no `empty` test for an association — it +// fails the build with CE0161; the test is a path to the associated object, +// negated with `not(…)` for absence. A bare attribute (`Name = empty`) is valid +// and is not module-qualified, so it never matches. (ledger finding #25) func (v *microflowValidator) checkXPathAssociationEmpty(variable, xpath string) { - for _, assoc := range xpathAssociationEmptyMatches(xpath) { + for _, h := range xpathAssociationEmptyMatches(xpath) { v.addViolation("MDL047", linter.SeverityError, - fmt.Sprintf("retrieve '$%s' constraint tests association `%s = empty`, which Mendix XPath does not support "+ - "(CE0161 \"Error(s) in XPath constraint\") — `= empty` works on attributes, not associations", variable, assoc), - fmt.Sprintf("Test for the absence of the associated object with negation: `[not(%s/)]`.", assoc)) + fmt.Sprintf("retrieve '$%s' constraint tests association `%s`, which Mendix XPath does not support "+ + "(CE0161 \"Error(s) in XPath constraint\") — `empty` compares attributes, not associations", variable, h.test()), + h.fix()) + } +} + +// xpathExpressionOnlyFunctions maps the Mendix EXPRESSION functions an XPath +// constraint does not have to the XPath function that does the same thing. The +// two languages sit side by side in a microflow and are easy to mix: measured on +// mxbuild 11.13.0, `startsWith(Title, 'X')` and `endsWith(Title, 'X')` in a +// retrieve constraint are CE0161 "Error(s) in XPath constraint" while +// `starts-with(Title, 'X')` builds clean (mendixlabs/mxcli#1213). The list is +// the measured pairs, not every function the two languages spell differently. +var xpathExpressionOnlyFunctions = map[string]string{ + "startsWith": "starts-with", + "endsWith": "ends-with", +} + +// xpathFunctionCallRe matches a function call and the character before it. A +// `-` or word character before the name is part of another name. +var xpathFunctionCallRe = regexp.MustCompile(`(^|[^\w.-])([A-Za-z_][\w-]*)\s*\(`) + +// xpathExpressionFunctionHits returns the expression-only functions an XPath +// constraint calls, outside string literals. +func xpathExpressionFunctionHits(xpath string) []string { + var out []string + for _, m := range xpathFunctionCallRe.FindAllStringSubmatch(blankXPathLiterals(xpath), -1) { + if _, ok := xpathExpressionOnlyFunctions[m[2]]; ok { + out = append(out, m[2]) + } + } + return out +} + +// checkXPathFunctionNames flags (MDL091) an expression-language function in a +// retrieve constraint. +func (v *microflowValidator) checkXPathFunctionNames(variable, xpath string) { + for _, fn := range xpathExpressionFunctionHits(xpath) { + xp := xpathExpressionOnlyFunctions[fn] + v.addViolation("MDL091", linter.SeverityError, + fmt.Sprintf("retrieve '$%s' constraint calls `%s()`, which is a Mendix expression function — XPath "+ + "does not have it, and mxbuild reports CE0161 \"Error(s) in XPath constraint\"", variable, fn), + fmt.Sprintf("Use the XPath function `%s()` in a retrieve constraint.", xp)) } } @@ -1486,6 +1564,9 @@ var knownActivityAnnotations = map[string]bool{ "start": true, } +// invalidAnchorRule refuses an @anchor parameter the visitor could not use. +const invalidAnchorRule = "MDL092" + // checkUnknownAnnotations rejects an @annotation name the visitor does not // implement. // @@ -1509,6 +1590,15 @@ func (v *microflowValidator) checkUnknownAnnotations(s ast.MicroflowStatement) { "A sequence flow's shape is two bezier control vectors, each a pixel offset from its end "+ "of the line: `@curve(from: (40, -90), to: (-40, 90))`. Only `from:` and `to:` are accepted.") } + for _, bad := range ann.InvalidAnchors { + v.addViolation(invalidAnchorRule, linter.SeverityError, + fmt.Sprintf("`@anchor` parameter `%s` is not one mxcli understands, so the edge it "+ + "names keeps its default sides", bad), + "A side is top, right, bottom or left. The flow leaving a statement is "+ + "`@anchor(from: right, to: left)`; an IF's branches are `@anchor(true: (from: …, to: …), "+ + "false: (…))`, a loop's `@anchor(iterator: (…), tail: (…))` — either side of a pair may be "+ + "left out (mendixlabs/mxcli#992).") + } for _, name := range ann.UnknownNames { v.addViolation("MDL059", linter.SeverityError, fmt.Sprintf("unknown annotation `@%s` — it parses but does nothing, so whatever it was "+ @@ -1583,3 +1673,21 @@ func (v *microflowValidator) checkCaptionOnLoop(ann *ast.ActivityAnnotations, wh "Use @annotation to attach a note to the loop instead.", "Replace @caption with @annotation to label the loop") } + +// lockWorkflowAllRule refuses `lock workflow all` / `unlock workflow all`. +const lockWorkflowAllRule = "MDL-WF17" + +// refuseWorkflowAll reports a lock or unlock that names no workflow. A Lock +// workflow activity always targets one definition; PauseAllWorkflows is the +// "Pause instances" option ON that definition, not "every workflow", and the +// metamodel has no all-definitions selection. The bare form built as CE1825 +// "The 'Workflow' property is required" (mendixlabs/mxcli#870). +func (v *microflowValidator) refuseWorkflowAll(verb, flag, activity string) { + label := strings.ToUpper(flag[:1]) + flag[1:] + v.addViolation(lockWorkflowAllRule, linter.SeverityError, + fmt.Sprintf("`%s workflow all` names no workflow — %s workflow activity always targets one "+ + "workflow definition, and the build fails CE1825 \"The 'Workflow' property is required\"", verb, activity), + fmt.Sprintf("Name the workflow: `%s workflow $WorkflowDefinition;` or `%s workflow Module.Workflow;`. "+ + "To %s the running instances of that workflow as well (Studio Pro's \"%s instances\"), add `%s all`: "+ + "`%s workflow $WorkflowDefinition %s all;`.", verb, verb, flag, label, flag, verb, flag)) +} diff --git a/mdl/executor/validate_microflow_error_handling.go b/mdl/executor/validate_microflow_error_handling.go index f63c0b10c5..4645aeac8c 100644 --- a/mdl/executor/validate_microflow_error_handling.go +++ b/mdl/executor/validate_microflow_error_handling.go @@ -58,6 +58,10 @@ var continueUnsupportedOn = map[string]string{ "close page": "Close page activity", "show message": "Show message activity", "validation feedback": "Validation feedback activity", + // mendixlabs/mxcli#175, measured on 11.14.0 in ako/TestApp: continue is + // CE6035 on a Call workflow activity; no clause, rollback and both custom + // handlers build. + "call workflow": "Call workflow activity", } // checkErrorHandlingContinueSupported reports `ON ERROR CONTINUE` on a statement @@ -114,6 +118,8 @@ func continueUnsupportedStatement(stmt ast.MicroflowStatement) (keyword, activit keyword = "show message" case *ast.ValidationFeedbackStmt: keyword = "validation feedback" + case *ast.CallWorkflowStmt: + keyword = "call workflow" default: // DeclareStmt and MfSetStmt are deliberately absent: create-variable and // change-variable accept Continue on 11.14.0. diff --git a/mdl/executor/validate_nanoflow.go b/mdl/executor/validate_nanoflow.go index f2d65739ff..a3599b31d7 100644 --- a/mdl/executor/validate_nanoflow.go +++ b/mdl/executor/validate_nanoflow.go @@ -28,9 +28,60 @@ func ValidateNanoflow(stmt *ast.CreateNanoflowStmt) []linter.Violation { varKinds: map[string]exprcheck.TypeKind{}, } v.walkExprFunctions(stmt.Body) + v.walkAnnotations(stmt.Body) + // The nanoflow restrictions exec's build refuses (validateNanoflow): an + // action a nanoflow cannot hold, an error-handling clause its activity + // rejects (CE6035), a Binary return. They ran only inside exec, so `check` + // passed what exec then refused (mendixlabs/mxcli#591). + for _, msg := range validateNanoflowBody(stmt.Body) { + v.addViolation(nanoflowActivityRule, linter.SeverityError, msg, "") + } + if msg := validateNanoflowReturnType(stmt.ReturnType); msg != "" { + v.addViolation(nanoflowActivityRule, linter.SeverityError, msg, "") + } return v.violations } +// nanoflowActivityRule reports what a nanoflow cannot hold — the messages of +// validateNanoflow, which exec refuses in its build. A check-side ID for them. +const nanoflowActivityRule = "MDL091" + +// walkAnnotations applies checkUnknownAnnotations (MDL059, MDL060, +// MDL079) to every statement of a body, nested ones and error-handler bodies +// included. A nanoflow runs it over its whole body; a microflow's walkBody +// covers its own statements and hands it the handler bodies, which walkBody +// does not enter. A nanoflow is where layout annotations matter most — Studio Pro's +// PED API does not write nanoflows, so MDL is their only scripted writer — and +// an annotation that parsed but was dropped there passed check AND exec +// (mendixlabs/mxcli#992). +func (v *microflowValidator) walkAnnotations(body []ast.MicroflowStatement) { + for _, s := range body { + v.checkUnknownAnnotations(s) + switch stmt := s.(type) { + case *ast.IfStmt: + v.walkAnnotations(stmt.ThenBody) + v.walkAnnotations(stmt.ElseBody) + case *ast.EnumSplitStmt: + for _, c := range stmt.Cases { + v.walkAnnotations(c.Body) + } + v.walkAnnotations(stmt.ElseBody) + case *ast.InheritanceSplitStmt: + for _, c := range stmt.Cases { + v.walkAnnotations(c.Body) + } + v.walkAnnotations(stmt.ElseBody) + case *ast.LoopStmt: + v.walkAnnotations(stmt.Body) + case *ast.WhileStmt: + v.walkAnnotations(stmt.Body) + } + if eh := stmtErrorHandling(s); eh != nil { + v.walkAnnotations(eh.Body) + } + } +} + // walkExprFunctions applies checkStmtExprFunctions to every statement in the // body, including branches, loops and error-handler bodies. func (v *microflowValidator) walkExprFunctions(body []ast.MicroflowStatement) { diff --git a/mdl/executor/validate_nanoflow_check_test.go b/mdl/executor/validate_nanoflow_check_test.go new file mode 100644 index 0000000000..79fbf77ca7 --- /dev/null +++ b/mdl/executor/validate_nanoflow_check_test.go @@ -0,0 +1,108 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +func nanoflowViolations(t *testing.T, body string) []linter.Violation { + t.Helper() + prog, errs := visitor.Build("mdl 1;\ncreate nanoflow M.NF_T()\nbegin\n" + body + "\nend;") + if len(errs) > 0 { + t.Fatalf("parsing:\n%s\nerrors: %v", body, errs) + } + return ValidateNanoflow(prog.Statements[len(prog.Statements)-1].(*ast.CreateNanoflowStmt)) +} + +func hasErrorRule(vs []linter.Violation, rule string) bool { + for _, v := range vs { + if v.RuleID == rule && v.Severity == linter.SeverityError { + return true + } + } + return false +} + +// `mxcli check` reported nothing a nanoflow could not hold: the nanoflow rules +// ran only inside exec's build (validateNanoflow), and the annotation rules +// (MDL059/MDL060) ran only for CREATE MICROFLOW. So `call nanoflow … on error +// continue` (mendixlabs/mxcli#591), `show home page` in a nanoflow, and the +// per-case `@curve(true: …)` mxcli does not implement (mendixlabs/mxcli#992) +// all passed check — the curve one passed exec too, and was dropped. +func TestValidateNanoflow_ReportsWhatExecWouldRefuse(t *testing.T) { + for _, tc := range []struct{ name, body, rule string }{ + {"error handling", "call nanoflow M.NF_Sub() on error continue;", nanoflowActivityRule}, + {"disallowed action", "show home page;", nanoflowActivityRule}, + {"per-case curve", "@curve(true: (from: (15, 0), to: (0, -30)))\nif true then\n log info node 'a' 'b';\nend if;", "MDL060"}, + {"unknown annotation", "@postion(1, 2)\nlog info node 'a' 'b';", "MDL059"}, + {"inside a handler", "call microflow M.MF() on error without rollback begin\n @curve(true: (from: (1, 0)))\n log info node 'a' 'b';\nend error;", "MDL060"}, + } { + t.Run(tc.name, func(t *testing.T) { + if vs := nanoflowViolations(t, tc.body); !hasErrorRule(vs, tc.rule) { + t.Errorf("check passed a nanoflow exec refuses or drops; want %s, got %+v", tc.rule, vs) + } + }) + } + // CONTROL: a nanoflow with none of these is clean. + if vs := nanoflowViolations(t, "@position(100, 100)\n@curve(from: (0, 30), to: (0, -30))\nlog info node 'a' 'b';"); hasErrorRule(vs, nanoflowActivityRule) || hasErrorRule(vs, "MDL060") || hasErrorRule(vs, "MDL059") { + t.Errorf("a valid nanoflow was reported: %+v", vs) + } +} + +// exec refuses MDL060 and MDL059 on a nanoflow as it does on a microflow. +func TestValidateNanoflowRules_RefusesAnIgnoredCurve(t *testing.T) { + prog, errs := visitor.Build("mdl 1;\ncreate nanoflow M.NF_T()\nbegin\n@curve(true: (from: (15, 0)))\nif true then\n log info node 'a' 'b';\nend if;\nend;") + if len(errs) > 0 { + t.Fatal(errs) + } + if err := validateNanoflowRules(prog.Statements[len(prog.Statements)-1].(*ast.CreateNanoflowStmt)); err == nil { + t.Fatal("exec accepted a @curve it would drop") + } +} + +// A microflow's annotation rules did not enter an error-handler body either. +func TestValidateMicroflow_ChecksAnnotationsInsideAHandler(t *testing.T) { + prog, errs := visitor.Build("mdl 1;\ncreate microflow M.MF_T()\nbegin\ncall microflow M.MF() on error without rollback begin\n @postion(1, 2)\n log info node 'a' 'b';\nend error;\nend;") + if len(errs) > 0 { + t.Fatal(errs) + } + if vs := ValidateMicroflow(prog.Statements[len(prog.Statements)-1].(*ast.CreateMicroflowStmt)); !hasErrorRule(vs, "MDL059") { + t.Fatalf("an unknown annotation inside a handler passed: %+v", vs) + } +} + +// mendixlabs/mxcli#992: an @anchor parameter the visitor cannot use was +// skipped, and the edge kept the builder's default sides. Refused at check and +// exec, on a microflow and a nanoflow alike. +func TestAnchorParameterItCannotUseIsRefused(t *testing.T) { + const body = "@anchor(true: (to: middle))\nif true then\n log info node 'a' 'b';\nend if;" + prog, errs := visitor.Build("mdl 1;\ncreate microflow M.MF_T()\nbegin\n" + body + "\nend;\ncreate nanoflow M.NF_T()\nbegin\n" + body + "\nend;") + if len(errs) > 0 { + t.Fatal(errs) + } + n := len(prog.Statements) + mf := prog.Statements[n-2].(*ast.CreateMicroflowStmt) + nf := prog.Statements[n-1].(*ast.CreateNanoflowStmt) + if vs := ValidateMicroflow(mf); !hasErrorRule(vs, invalidAnchorRule) { + t.Errorf("microflow: check passed an @anchor it drops: %+v", vs) + } + if vs := ValidateNanoflow(nf); !hasErrorRule(vs, invalidAnchorRule) { + t.Errorf("nanoflow: check passed an @anchor it drops: %+v", vs) + } + if err := validateMicroflowRules(mf); err == nil { + t.Error("microflow: exec accepted an @anchor it drops") + } + if err := validateNanoflowRules(nf); err == nil { + t.Error("nanoflow: exec accepted an @anchor it drops") + } + // CONTROL: the one-sided per-case form is valid. + prog, _ = visitor.Build("mdl 1;\ncreate nanoflow M.NF_T()\nbegin\n@anchor(true: (to: top))\nif true then\n log info node 'a' 'b';\nend if;\nend;") + if vs := ValidateNanoflow(prog.Statements[len(prog.Statements)-1].(*ast.CreateNanoflowStmt)); hasErrorRule(vs, invalidAnchorRule) { + t.Errorf("a valid per-case @anchor was refused: %+v", vs) + } +} diff --git a/mdl/executor/validate_program.go b/mdl/executor/validate_program.go index 9eeb51bab2..484912c802 100644 --- a/mdl/executor/validate_program.go +++ b/mdl/executor/validate_program.go @@ -289,6 +289,10 @@ func ValidateProgram(prog *ast.Program, projectPath string) []linter.Violation { // operation, so the mapping would be dropped in silence (#843). violations = append(violations, ValidateRestClientMappings(prog)...) + // Flag a published REST operation whose commit option is not one Mendix + // has; it used to parse and be thrown away (ako/mxcli#571). + violations = append(violations, ValidatePublishedRestCommit(prog)...) + // Flag a scheduled event whose Repeat and fields disagree (a Multiplier on // a Daily repeat, an HourOfDay of 99). Decidable from the statement, so it // runs here rather than at exec, where the script would already have diff --git a/mdl/executor/validate_rest_mapping.go b/mdl/executor/validate_rest_mapping.go index 186b777be8..d4f22a6975 100644 --- a/mdl/executor/validate_rest_mapping.go +++ b/mdl/executor/validate_rest_mapping.go @@ -54,3 +54,40 @@ func ValidateRestClientMappings(prog *ast.Program) []linter.Violation { } return out } + +// ValidatePublishedRestCommit reports (MDL-REST03) a published REST operation +// whose commit clause names no commit option. exec refuses the same statement +// (publishedRestCommit), so check predicts it. +func ValidatePublishedRestCommit(prog *ast.Program) []linter.Violation { + var out []linter.Violation + visit := func(res *ast.PublishedRestResourceDef) { + if res == nil { + return + } + for _, op := range res.Operations { + if _, err := publishedRestCommit(op.Commit); err != nil { + out = append(out, linter.Violation{ + RuleID: "MDL-REST03", + Severity: linter.SeverityError, + Message: "resource '" + res.Name + "', operation " + op.HTTPMethod + " " + op.Path + ": " + err.Error(), + Suggestion: "Write commit Yes, commit YesWithoutEvents or commit No, or leave the clause out (Yes).", + }) + } + } + } + for _, stmt := range prog.Statements { + switch s := stmt.(type) { + case *ast.CreatePublishedRestServiceStmt: + for _, res := range s.Resources { + visit(res) + } + case *ast.AlterPublishedRestServiceStmt: + for _, a := range s.Actions { + if add, ok := a.(*ast.PublishedRestAddResourceAction); ok { + visit(add.Resource) + } + } + } + } + return out +} diff --git a/mdl/executor/validate_retrieve_members.go b/mdl/executor/validate_retrieve_members.go new file mode 100644 index 0000000000..6bb63973ee --- /dev/null +++ b/mdl/executor/validate_retrieve_members.go @@ -0,0 +1,101 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// A retrieve constraint naming a member its entity does not have passed +// `check --references` and `exec`, and mxbuild then reported CE0161 "Error(s) +// in XPath constraint" (mendixlabs/mxcli#1213). The widget data source had this +// check since #1049; the microflow retrieve did not, and neither looked at an +// entity the same script declares — the ordinary shape, since the entity and +// the microflow querying it are usually one script. +// +// Measured on mxbuild 11.13.0 (entity with `CreatedDate: AutoCreatedDate`): +// +// [NoSuchAttr = 'x'] CE0161 +// [CreatedDate > '[%CurrentDateTime%]'] CE0161 (the spelling describe prints) +// [createdDate > '[%CurrentDateTime%]'] clean +// +// The resolution is unresolvableXPathSteps, the widget check's walk, over the +// constraint exactly as the flow builder stores it (retrieveXPathConstraint), so +// what is checked is what is written. Silence where the entity cannot be +// established, as everywhere in that walk. + +// validateRetrieveMembers reports the members of each database retrieve's +// constraint that resolve to nothing on the retrieved entity. +func validateRetrieveMembers(ctx *ExecContext, retrieves []retrieveConstraintRef, sc *scriptContext) []string { + if len(retrieves) == 0 { + return nil + } + m := &execXPathModel{ctx: ctx} + var assocs map[string]string // unqualified association name -> qualified, loaded on demand + resolve := func(entityQN, member string) memberResolution { + decl := sc.entityDecls[entityQN] + if decl == nil || sc.alteredEntities[entityQN] { + return resolveMemberOnEntity(ctx, entityQN, member) + } + // Declared by the script: its attribute list is what exec writes. An + // inherited member cannot be judged from the declaration. + if decl.Generalization != nil { + return memberUnknown + } + for _, a := range decl.Attributes { + if a.Name == member && !isAutoSystemMemberType(a.Type.Kind) { + return memberFound + } + } + // A bare association is a different mistake, with its own rule and fix + // (MDL-XPATH01) — not "names nothing". + if sc.associations[member] != "" || sc.ambiguousAssc[member] { + return memberFound + } + if assocs == nil { + assocs = buildAssociationIndex(ctx) + } + if assocs[member] != "" { + return memberFound + } + return memberMissing + } + + var errs []string + for _, r := range retrieves { + if r.entity == "" || r.stored == "" { + continue + } + for _, bad := range unresolvableXPathStepsWith(ctx, m, r.stored, r.entity, resolve) { + errs = append(errs, fmt.Sprintf( + "retrieve from %s: the constraint names %q, which is neither an attribute nor an association of it — mxbuild rejects the constraint (CE0161 \"Error(s) in XPath constraint\")%s", + r.entity, bad.name, systemMemberSpellingHint(bad.name))) + } + } + return errs +} + +// isAutoSystemMemberType reports the attribute types that are not attributes in +// the model: the entity stores a flag, and XPath names the member in lower +// camel case (createdDate) whatever the declaration called it. +func isAutoSystemMemberType(k ast.DataTypeKind) bool { + switch k { + case ast.TypeAutoCreatedDate, ast.TypeAutoChangedDate, ast.TypeAutoOwner, ast.TypeAutoChangedBy: + return true + } + return false +} + +// systemMemberSpellingHint names the XPath spelling of a system member written +// the way describe prints the attribute (`CreatedDate`). +func systemMemberSpellingHint(name string) string { + for member := range xpathImplicitMembers { + if member != "id" && member != name && strings.EqualFold(member, name) { + return fmt.Sprintf(". In XPath the system member is spelled `%s`", member) + } + } + return "" +} diff --git a/mdl/executor/validate_widget_member_refs.go b/mdl/executor/validate_widget_member_refs.go index 2bc8a3d9c8..aa3cc491b2 100644 --- a/mdl/executor/validate_widget_member_refs.go +++ b/mdl/executor/validate_widget_member_refs.go @@ -215,6 +215,14 @@ type badStep struct { // the consequence is a corrupted constraint; here it is only a false error, but // a false error still blocks a script that builds. func unresolvableXPathSteps(ctx *ExecContext, m xpathrefs.Model, constraint, entityQN string) []badStep { + return unresolvableXPathStepsWith(ctx, m, constraint, entityQN, nil) +} + +// unresolvableXPathStepsWith is unresolvableXPathSteps with the bare-member +// lookup replaced — the microflow retrieve check passes one that also knows the +// entities the script declares. nil is resolveMemberOnEntity. +func unresolvableXPathStepsWith(ctx *ExecContext, m xpathrefs.Model, constraint, entityQN string, + resolveBare func(entityQN, member string) memberResolution) []badStep { groups := visitor.SplitXPathPredicateGroups(constraint) if len(groups) == 0 { groups = []string{constraint} @@ -225,7 +233,7 @@ func unresolvableXPathSteps(ctx *ExecContext, m xpathrefs.Model, constraint, ent if !ok || expr == nil { continue } - v := &xpathMemberVisitor{ctx: ctx, model: m} + v := &xpathMemberVisitor{ctx: ctx, model: m, resolveBare: resolveBare} v.walk(expr, entityQN) out = append(out, v.bad...) } @@ -235,10 +243,11 @@ func unresolvableXPathSteps(ctx *ExecContext, m xpathrefs.Model, constraint, ent // xpathMemberVisitor walks a parsed constraint carrying the entity each step is // evaluated against, the same traversal xpathrefs' walker performs for renames. type xpathMemberVisitor struct { - ctx *ExecContext - model xpathrefs.Model - bad []badStep - seen map[string]bool + ctx *ExecContext + model xpathrefs.Model + resolveBare func(entityQN, member string) memberResolution // nil: resolveMemberOnEntity + bad []badStep + seen map[string]bool } func (v *xpathMemberVisitor) add(s badStep) { @@ -290,7 +299,18 @@ func (v *xpathMemberVisitor) noteBare(name, cur string) { if cur == "" || name == "" { return } - if resolveMemberOnEntity(v.ctx, cur, name) == memberMissing { + // The members every entity has in XPath without declaring them. An entity + // stores createdDate as a flag, not an attribute, so the attribute lookup + // would call `[createdDate > …]` — which builds clean, measured on 11.13.0 — + // missing. + if xpathImplicitMembers[name] { + return + } + resolve := v.resolveBare + if resolve == nil { + resolve = func(e, m string) memberResolution { return resolveMemberOnEntity(v.ctx, e, m) } + } + if resolve(cur, name) == memberMissing { v.add(badStep{name: name, kind: "attribute"}) } } @@ -356,6 +376,12 @@ func (v *xpathMemberVisitor) walkPath(steps []ast.XPathStep, cur string) { } } +// xpathImplicitMembers are the bare names XPath resolves on every entity +// without the entity declaring an attribute of that name. +var xpathImplicitMembers = map[string]bool{ + "id": true, "createdDate": true, "changedDate": true, "owner": true, "changedBy": true, +} + // execXPathModel answers xpathrefs.Model from the connected project. type execXPathModel struct{ ctx *ExecContext } diff --git a/mdl/executor/validate_widget_unbuildable.go b/mdl/executor/validate_widget_unbuildable.go new file mode 100644 index 0000000000..75c0f46b12 --- /dev/null +++ b/mdl/executor/validate_widget_unbuildable.go @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// A widget keyword the grammar accepts and the page builder has no writer for. +// +// `referenceselector` parses, so `check --references` reported "Check passed!", +// and `exec` then refused the page — after the statements before it had been +// written — with "unsupported widget type: referenceselector — … refresh the +// project's widget definitions: 'mxcli widget init'" (ako/mxcli#563). That hint +// does not apply: the classic reference selector is a built-in Forms widget, not +// a pluggable one, so there is no .mpk and `widget init` cannot help. +// +// formsWidgetsWithoutWriter is the one answer to "can mxcli build this +// keyword?" for these: the builder's fall-through reports with it, and the +// validator refuses with it, so check and exec say the same thing. +// TestFormsWidgetsWithoutWriterAreUnbuilt keeps it honest against the +// builder's dispatch: a keyword that gains a builder case must leave the set. +var formsWidgetsWithoutWriter = map[string]struct { + storedType string + replacement string +}{ + "referenceselector": { + storedType: "Forms$ReferenceSelector", + replacement: "use `combobox` over the association instead — it needs its own `datasource:` " + + "(CE0642 without one) and a caption attribute", + }, +} + +// unbuildableWidgetMessage is the refusal for a keyword mxcli has no writer +// for, or "" when it has one (or the keyword is not one of these). +func unbuildableWidgetMessage(widgetType string) string { + u, ok := formsWidgetsWithoutWriter[strings.ToLower(widgetType)] + if !ok { + return "" + } + return fmt.Sprintf("`%s` is the built-in Forms widget %s, which mxcli cannot write — it is not a "+ + "pluggable widget, so `mxcli widget init` does not help; %s", + strings.ToLower(widgetType), u.storedType, u.replacement) +} + +// validateUnbuildableWidgetKind refuses (MDL-WIDGET38) a widget keyword the page +// builder would refuse. A project's own widget definition of that MDL name wins, +// exactly as it does in the builder, whose registry lookup comes first. +func validateUnbuildableWidgetKind(w *ast.WidgetV3, registry *WidgetRegistry, locationPrefix string) []linter.Violation { + if w == nil || w.TypeIsGeneric { + return nil + } + msg := unbuildableWidgetMessage(w.Type) + if msg == "" { + return nil + } + if registry != nil { + if _, ok := registry.Get(strings.ToUpper(w.Type)); ok { + return nil + } + } + return []linter.Violation{{ + RuleID: "MDL-WIDGET38", + Severity: linter.SeverityError, + Message: fmt.Sprintf("%s: widget `%s`: %s", locationPrefix, w.Name, msg), + Suggestion: "replace the widget; exec refuses the page otherwise, after the statements before it were written", + }} +} diff --git a/mdl/executor/validate_widget_unbuildable_test.go b/mdl/executor/validate_widget_unbuildable_test.go new file mode 100644 index 0000000000..b11fd00de0 --- /dev/null +++ b/mdl/executor/validate_widget_unbuildable_test.go @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "go/ast" + "go/parser" + "go/token" + "os" + "path/filepath" + "strconv" + "strings" + "testing" +) + +// TestFormsWidgetsWithoutWriterAreUnbuilt keeps formsWidgetsWithoutWriter — the +// one answer check and exec share about which keywords the page builder cannot +// write (ako/mxcli#563) — in step with the builder. Each entry must be a widget +// keyword the grammar accepts, and must NOT be a case of buildWidgetV3's +// dispatch: a keyword that gains a builder has to leave the set, or check goes +// on refusing a widget exec can now write. +func TestFormsWidgetsWithoutWriterAreUnbuilt(t *testing.T) { + cases := builderSwitchCases(t) + if !cases["textbox"] || !cases["dataview"] { + t.Fatalf("could not read buildWidgetV3's dispatch (got %d cases) — the guard would pass vacuously", len(cases)) + } + grammar, err := os.ReadFile(filepath.Join("..", "grammar", "domains", "MDLPage.g4")) + if err != nil { + t.Fatal(err) + } + rule := string(grammar) + if i := strings.Index(rule, "\nwidgetTypeV3"); i >= 0 { + rule = rule[i:] + if j := strings.Index(rule, ";"); j >= 0 { + rule = rule[:j] + } + } + for kw := range formsWidgetsWithoutWriter { + if cases[kw] { + t.Errorf("%q has a case in buildWidgetV3 now — drop it from formsWidgetsWithoutWriter", kw) + } + if !strings.Contains(rule, strings.ToUpper(kw)) { + t.Errorf("%q is not a widgetTypeV3 keyword — formsWidgetsWithoutWriter lists only what parses", kw) + } + } +} + +// builderSwitchCases returns the string cases of the `switch strings.ToLower( +// w.Type)` in buildWidgetV3. +func builderSwitchCases(t *testing.T) map[string]bool { + t.Helper() + fset := token.NewFileSet() + f, err := parser.ParseFile(fset, "cmd_pages_builder_v3.go", nil, 0) + if err != nil { + t.Fatal(err) + } + out := map[string]bool{} + for _, d := range f.Decls { + fn, ok := d.(*ast.FuncDecl) + if !ok || fn.Name.Name != "buildWidgetV3" { + continue + } + ast.Inspect(fn.Body, func(n ast.Node) bool { + cc, ok := n.(*ast.CaseClause) + if !ok { + return true + } + for _, e := range cc.List { + if lit, ok := e.(*ast.BasicLit); ok && lit.Kind == token.STRING { + if s, err := strconv.Unquote(lit.Value); err == nil { + out[s] = true + } + } + } + return true + }) + } + return out +} diff --git a/mdl/executor/validate_widgets.go b/mdl/executor/validate_widgets.go index 57709807f6..0efbae82c5 100644 --- a/mdl/executor/validate_widgets.go +++ b/mdl/executor/validate_widgets.go @@ -112,7 +112,8 @@ func ValidateWidgetPropertiesForStatement(stmt ast.Statement, registry *WidgetRe return nil } if label, widgets, ok := documentWidgets(stmt); ok { - return validateWidgetTree(widgets, registry, label) + out := validateWidgetTree(widgets, registry, label) + return append(out, validatePageVariableBindings(widgets, documentVariables(stmt), label)...) } switch s := stmt.(type) { case *ast.AlterPageStmt: @@ -184,6 +185,8 @@ func validateWidgetTreeIn(widgets []*ast.WidgetV3, registry *WidgetRegistry, loc // A keyword whose stored $Type Mendix no longer has. Unlike MDL-WIDGET25 // this needs no project: the type is unknown to every Mendix version. out = append(out, validateRetiredWidgetKind(w, locationPrefix)...) + // A keyword the page builder has no writer for (ako/mxcli#563). + out = append(out, validateUnbuildableWidgetKind(w, registry, locationPrefix)...) out = append(out, validatePluggableWidgetProperties(w, registry, locationPrefix)...) // A repeatable property written as a property value — `attributes: // [(…)]` — which used to check clean, exec, and vanish (#999). Runs for @@ -279,14 +282,14 @@ func validateDatasourceXPathAssociationEmpty(w *ast.WidgetV3, locationPrefix str return nil } var out []linter.Violation - for _, assoc := range xpathAssociationEmptyMatches(ds.Where) { + for _, h := range xpathAssociationEmptyMatches(ds.Where) { out = append(out, linter.Violation{ RuleID: "MDL047", Severity: linter.SeverityError, Message: fmt.Sprintf( - "%s: widget `%s` datasource constraint tests association `%s = empty`, which Mendix XPath does not support (CE0161 \"Error(s) in XPath constraint\") — `= empty` works on attributes, not associations", - locationPrefix, w.Name, assoc), - Suggestion: fmt.Sprintf("Test for the absence of the associated object with negation: `[not(%s/)]`.", assoc), + "%s: widget `%s` datasource constraint tests association `%s`, which Mendix XPath does not support (CE0161 \"Error(s) in XPath constraint\") — `empty` compares attributes, not associations", + locationPrefix, w.Name, h.test()), + Suggestion: h.fix(), }) } return out diff --git a/mdl/exprcheck/adapters/adapter_scope.go b/mdl/exprcheck/adapters/adapter_scope.go index 87e32511d9..facdc59ba0 100644 --- a/mdl/exprcheck/adapters/adapter_scope.go +++ b/mdl/exprcheck/adapters/adapter_scope.go @@ -3,6 +3,7 @@ package adapters import ( + "reflect" "strings" "github.com/mendixlabs/mxcli/mdl/ast" @@ -239,7 +240,27 @@ func StatementErrorHandling(stmt ast.MicroflowStatement) *ast.ErrorHandlingClaus case *ast.AggregateListStmt: return s.ErrorHandling } - return nil + return reflectedErrorHandling(stmt) +} + +var errorHandlingClauseType = reflect.TypeOf(&ast.ErrorHandlingClause{}) + +// reflectedErrorHandling reads the clause off any other statement that carries +// one in an `ErrorHandling` field. The switch above is a list, and a list +// drifts: `call workflow`, the REST statements and the other workflow +// statements all take an onErrorClause and were missing from it, so the rules +// that ask for the clause — MDL076 for one (mendixlabs/mxcli#175) — could not +// see it, and their handler bodies were never walked. +func reflectedErrorHandling(stmt ast.MicroflowStatement) *ast.ErrorHandlingClause { + v := reflect.ValueOf(stmt) + if v.Kind() != reflect.Ptr || v.IsNil() || v.Elem().Kind() != reflect.Struct { + return nil + } + f := v.Elem().FieldByName("ErrorHandling") + if !f.IsValid() || f.Type() != errorHandlingClauseType || f.IsNil() { + return nil + } + return f.Interface().(*ast.ErrorHandlingClause) } // errorHandlerBody returns a statement's custom ON ERROR body, or nil when it diff --git a/mdl/grammar/MDLParser.g4 b/mdl/grammar/MDLParser.g4 index 3fe55a28b0..a9efa5df4c 100644 --- a/mdl/grammar/MDLParser.g4 +++ b/mdl/grammar/MDLParser.g4 @@ -346,7 +346,7 @@ alterStylingAssignment * set (Title: 'Edit order'); -- the document itself * insert after txtName { textbox txtNew (Label: 'New', Attribute: Attr) } * insert into ctnMain { … } - * replace footer1 with { footer f1 { … } } + * replace dvMain.footer with { footer { … } } * drop txtOld, dgOrders.Total; * } * ``` diff --git a/mdl/grammar/domains/MDLMicroflow.g4 b/mdl/grammar/domains/MDLMicroflow.g4 index a754ab6256..c43ed60cf3 100644 --- a/mdl/grammar/domains/MDLMicroflow.g4 +++ b/mdl/grammar/domains/MDLMicroflow.g4 @@ -684,14 +684,19 @@ openWorkflowStatement : OPEN WORKFLOW VARIABLE onErrorClause? ; -// LOCK WORKFLOW $Wf; or LOCK WORKFLOW ALL; +// LOCK WORKFLOW $WfDef [PAUSE ALL]; or LOCK WORKFLOW Module.Workflow [PAUSE ALL]; +// PAUSE ALL is Studio Pro's "Pause instances" (PauseAllWorkflows). A lock always +// names its workflow: the bare `LOCK WORKFLOW ALL` still parses, and check and +// exec refuse it — there is no model for it, it built as CE1825 +// (mendixlabs/mxcli#870). ALL is before qualifiedName so `all` stays that form. lockWorkflowStatement - : LOCK WORKFLOW (VARIABLE | ALL) onErrorClause? + : LOCK WORKFLOW (VARIABLE | ALL | qualifiedName) (PAUSE ALL)? onErrorClause? ; -// UNLOCK WORKFLOW $Wf; or UNLOCK WORKFLOW ALL; +// UNLOCK WORKFLOW $WfDef [UNPAUSE ALL]; — UNPAUSE ALL is "Unpause instances" +// (ResumeAllPausedWorkflows). Same rule for the bare ALL. unlockWorkflowStatement - : UNLOCK WORKFLOW (VARIABLE | ALL) onErrorClause? + : UNLOCK WORKFLOW (VARIABLE | ALL | qualifiedName) (UNPAUSE ALL)? onErrorClause? ; callArgumentList diff --git a/mdl/grammar/domains/MDLSettings.g4 b/mdl/grammar/domains/MDLSettings.g4 index b3e5ff3dce..57bbd77f0c 100644 --- a/mdl/grammar/domains/MDLSettings.g4 +++ b/mdl/grammar/domains/MDLSettings.g4 @@ -569,7 +569,11 @@ annotationParams ; annotationParam - : annotationParamName COLON (annotationValue | annotationParenValue) // Named parameter + // annotationParenValue FIRST: `(to: top)` is also an expression — `:` is + // Mendix's division operator — and ANTLR takes the first alternative that + // matches, so with annotationValue first `@anchor(true: (to: top))` parsed + // as `true: to ÷ top` and set no anchor (mendixlabs/mxcli#992). + : annotationParamName COLON (annotationParenValue | annotationValue) // Named parameter | annotationValue // Positional parameter ; diff --git a/mdl/roundtrip/flow_modify_notes_test.go b/mdl/roundtrip/flow_modify_notes_test.go index 9aa090b60e..bd44fdd167 100644 --- a/mdl/roundtrip/flow_modify_notes_test.go +++ b/mdl/roundtrip/flow_modify_notes_test.go @@ -185,7 +185,7 @@ end; create or modify nanoflow MyFirstModule.Rerun_RollbackNf ($E: MyFirstModule.RerunRbThing) begin change $E (Name = 'one'); - commit $E on error rollback; + delete $E on error rollback; end; ` if err := h.exec(flows); err != nil { diff --git a/mdl/roundtrip/flow_splice_parity_test.go b/mdl/roundtrip/flow_splice_parity_test.go index dbdfd9f0ff..19f5a5ed54 100644 --- a/mdl/roundtrip/flow_splice_parity_test.go +++ b/mdl/roundtrip/flow_splice_parity_test.go @@ -5,11 +5,15 @@ package roundtrip import ( + "bytes" "strings" "testing" + "github.com/mendixlabs/mxcli/mdl/backend" + modelsdkbackend "github.com/mendixlabs/mxcli/mdl/backend/modelsdk" "github.com/mendixlabs/mxcli/mdl/executor" "github.com/mendixlabs/mxcli/mdl/langver" + "github.com/mendixlabs/mxcli/mdl/scriptdiff" "github.com/mendixlabs/mxcli/mdl/visitor" ) @@ -103,8 +107,8 @@ func TestPedAppSpliceParity_AuthoredSpellings(t *testing.T) { t.Run(name, func(t *testing.T) { script := header + spliceParityFlow // diff first: it must reach exec's verdict, and it writes nothing. - if out := h.diff(script); !strings.Contains(out, "0 new, 0 modified, 1 unchanged") || - strings.Contains(out, "refused") { + if out := h.diff(script); !strings.Contains(out, "exec would write nothing") || + strings.Contains(out, "Refused") { t.Errorf("diff of the unchanged source under %s:\n%s", name, out) } if err := h.exec(script); err != nil { @@ -127,7 +131,7 @@ func TestPedAppSpliceParity_AuthoredSpellings(t *testing.T) { // spliced in — the comparison is not blind to the where clause. t.Run("an edited constraint writes", func(t *testing.T) { edited := strings.Replace(spliceParityFlow, "MonthKey >= $Context/FromKey", "MonthKey > $Context/FromKey", 1) - if out := h.diff("mdl 1;\n" + edited); !strings.Contains(out, "0 new, 1 modified, 0 unchanged") { + if out := h.diff("mdl 1;\n" + edited); !strings.Contains(out, "0 new, 1 modified, 0 removed") { t.Errorf("diff of an edited constraint:\n%s", out) } if err := h.exec("mdl 1;\n" + edited); err != nil { @@ -149,7 +153,7 @@ func TestPedAppSpliceParity_AuthoredSpellings(t *testing.T) { inLoop := strings.Replace(spliceParityFlow, "$Total + $R/MonthKey", "$Total + $R/MonthKey + 1", 1) before := h.snapshot() out := h.diff("mdl 1;\n" + inLoop) - if !strings.Contains(out, "Refused: Microflow MyFirstModule.SpliceParity") || !strings.Contains(out, "1 refused") { + if !strings.Contains(out, "Refused: exec would stop at this error, having written nothing") || !strings.Contains(out, "cannot be spliced") { t.Errorf("diff of a change exec refuses:\n%s", out) } err := h.exec("mdl 1;\n" + inLoop) @@ -160,7 +164,7 @@ func TestPedAppSpliceParity_AuthoredSpellings(t *testing.T) { t.Errorf("a refused statement wrote %d unit(s)", len(changed)) } // Under mdl 0 the same change is rebuilt, and diff shows the change. - if out := h.diff(inLoop); !strings.Contains(out, "0 new, 1 modified, 0 unchanged") { + if out := h.diff(inLoop); !strings.Contains(out, "0 new, 1 modified, 0 removed") { t.Errorf("diff of the same change under mdl 0:\n%s", out) } }) @@ -184,11 +188,12 @@ func TestPedAppSpliceParity_DiffReportsTheWriteExecMakes(t *testing.T) { for _, header := range []string{"", "mdl 1;\n"} { name := map[string]string{"": "mdl 0", "mdl 1;\n": "mdl 1"}[header] // Control: the same statement in the stored folder is unchanged. - if out := h.diff(header + spliceParityFlow); !strings.Contains(out, "0 new, 0 modified, 1 unchanged") { + if out := h.diff(header + spliceParityFlow); !strings.Contains(out, "exec would write nothing") { t.Errorf("diff of the unchanged source under %s:\n%s", name, out) } - if out := h.diff(header + moved); !strings.Contains(out, "0 new, 1 modified, 0 unchanged") || - !strings.Contains(out, "moved to folder 'Moved'") { + // The folder is new, and the flow is moved into it. + if out := h.diff(header + moved); !strings.Contains(out, "1 new, 1 modified, 0 removed") || + !strings.Contains(out, "moved to 'MyFirstModule/Moved'") { t.Errorf("diff of a statement exec moves, under %s:\n%s", name, out) } } @@ -276,16 +281,21 @@ func runFlowSpliceParity(t *testing.T, fx fixture) { t.Logf("%d flows checked under mdl 0 and mdl 1", flows) } -// diff runs `mxcli diff` on a script and returns what it printed. +// diff runs `mxcli diff` on a script and returns what it printed: the script +// executed on a scratch copy of the working copy, compared unit by unit. func (h *harness) diff(script string) string { h.t.Helper() - h.out.Reset() prog, errs := visitor.Build(script) if len(errs) > 0 { h.t.Fatalf("parse: %v", errs[0]) } - if err := h.exe.DiffProgram(prog, executor.DiffOptions{}); err != nil { + rep, err := scriptdiff.Run(h.mpr, prog, scriptdiff.Options{ + NewBackend: func() backend.FullBackend { return modelsdkbackend.New() }, + }) + if err != nil { h.t.Fatalf("diff: %v", err) } - return h.out.String() + var out bytes.Buffer + rep.Write(&out, executor.DiffOptions{}, false) + return out.String() } diff --git a/mdl/roundtrip/flow_verdict_agreement_test.go b/mdl/roundtrip/flow_verdict_agreement_test.go index 1d276fe366..8e0a038399 100644 --- a/mdl/roundtrip/flow_verdict_agreement_test.go +++ b/mdl/roundtrip/flow_verdict_agreement_test.go @@ -23,7 +23,7 @@ import ( type verdicts struct { checkError bool // check -p: an MDL-V1-REBUILD error checkWarning bool // check -p: the MDL-V1-REBUILD warning - diffRefused bool // diff: "Refused:" + diffRefused bool // diff: "Refused:" with the splice's refusal execRefused bool // exec: the splice refusal, nothing written execRebuilt bool // exec: the MDL-V1-REBUILD warning, the flow rebuilt checkMessage string @@ -51,7 +51,10 @@ func (h *harness) verdictsOf(script string) verdicts { v.checkWarning = true } } - v.diffRefused = strings.Contains(h.diff(script), "Refused:") + // diff runs exec on a scratch copy, so it reports whatever exec stops + // at; the refusal compared here is the splice's. + out := h.diff(script) + v.diffRefused = strings.Contains(out, "Refused:") && strings.Contains(out, "cannot be spliced into the stored flow") before := h.snapshot() err := h.exec(script) changed := before.diff(h.snapshot()) diff --git a/mdl/roundtrip/pluggable_replace_keeps_test.go b/mdl/roundtrip/pluggable_replace_keeps_test.go new file mode 100644 index 0000000000..e09c847bd6 --- /dev/null +++ b/mdl/roundtrip/pluggable_replace_keeps_test.go @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package roundtrip + +import ( + "bytes" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// mendixlabs/mxcli#1247: `alter page … replace with { combobox … }` +// rebuilt the widget from its template, so every property MDL has no word for +// came back at the template's value. On TestApp's Studio Pro-authored +// Rules.BusinessRule_NewEdit, adding a sort to comboBox1's options turned its +// stored Editable "Never" into "Always" and an expression property's +// PrimitiveValue "" into "false". The replace now changes what the statement +// changes and nothing else, and running it again writes nothing. +func TestReplacePluggableKeepsWhatTheStatementDoesNotState(t *testing.T) { + h := newFixtureHarness(t, testApp) + defer h.close() + + const replace = `mdl 1; +alter page Rules.BusinessRule_NewEdit { + replace comboBox1 with { + combobox comboBox1 ( + Label: 'Rule category', + Attribute: Rules.BusinessRule_RuleCategory, + DataSource: database from Rules.RuleCategory sort by Name asc, + CaptionAttribute: Name + ) + } +};` + before := h.pageUnit(t, "BusinessRule_NewEdit") + storedType, storedObj := pluggableParts(t, before, "comboBox1") + // Control: the stored widget carries a value the template does not, so the + // assertions below can fail. + if got := pluggableField(t, before, "comboBox1", "Editable"); got != "Never" { + t.Fatalf("control: comboBox1's stored Editable is %q, the fixture changed", got) + } + + if err := h.exec(replace); err != nil { + t.Fatalf("exec: %v", err) + } + after := h.pageUnit(t, "BusinessRule_NewEdit") + typ, obj := pluggableParts(t, after, "comboBox1") + if got := pluggableField(t, after, "comboBox1", "Editable"); got != "Never" { + t.Errorf("Editable = %q after the replace, want the stored \"Never\"", got) + } + if !bytes.Equal(typ, storedType) { + t.Errorf("the stored Type was replaced by the template's") + } + diff := bsonDiff(storedObj, obj) + if len(diff) == 0 { + t.Fatal("the replace changed nothing — the sort was not written") + } + for _, d := range diff { + if !strings.Contains(d, "SortItems") { + t.Errorf("a property the statement does not state changed: %s", d) + } + } + + // Twice-exec: the same statement again writes nothing. + again := h.snapshot() + if err := h.exec(replace); err != nil { + t.Fatalf("second exec: %v", err) + } + if changed := again.diff(h.snapshot()); len(changed) > 0 { + t.Errorf("running the replace a second time wrote:\n %s", strings.Join(changed, "\n ")) + } +} + +// pluggableField returns a string field of the pluggable widget named name. +func pluggableField(t *testing.T, unit []byte, name, field string) string { + t.Helper() + var doc bson.D + if err := bson.Unmarshal(unit, &doc); err != nil { + t.Fatal(err) + } + var got string + var walk func(any) bool + walk = func(v any) bool { + switch x := v.(type) { + case bson.D: + isCW, n := false, "" + for _, e := range x { + switch e.Key { + case "$Type": + isCW = e.Value == "CustomWidgets$CustomWidget" + case "Name": + n, _ = e.Value.(string) + } + } + if isCW && n == name { + for _, e := range x { + if e.Key == field { + got, _ = e.Value.(string) + } + } + return true + } + for _, e := range x { + if walk(e.Value) { + return true + } + } + case bson.A: + for _, e := range x { + if walk(e) { + return true + } + } + } + return false + } + if !walk(doc) { + t.Fatalf("no pluggable widget %s", name) + } + return got +} diff --git a/mdl/roundtrip/pluggable_replace_nested_test.go b/mdl/roundtrip/pluggable_replace_nested_test.go new file mode 100644 index 0000000000..ab97728e44 --- /dev/null +++ b/mdl/roundtrip/pluggable_replace_nested_test.go @@ -0,0 +1,82 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package roundtrip + +import ( + "strings" + "testing" +) + +// mendixlabs/mxcli#1247, the nested case: a DataGrid 2 stores its columns as +// an object list, and each column's filter is a pluggable widget of its own. +// Restating TestApp's Studio Pro-authored dataGrid2_1 with one column caption +// changed fell back to the template rebuild — the filter widgets' type +// pointers aim into their OWN Type, which the graft could not re-aim — and +// silently turned itemSelectionMethod "rowClick" into "checkbox", reset +// onClickTrigger, and rebuilt every column's unmapped properties. Only the +// caption the statement changes may change, and running it again writes +// nothing. +func TestReplaceDataGridKeepsWhatTheStatementDoesNotState(t *testing.T) { + h := newFixtureHarness(t, testApp) + defer h.close() + + described, err := h.describe("page Rules.BusinessRule_Overview") + if err != nil { + t.Fatalf("describe: %v", err) + } + lines := strings.Split(described, "\n") + start, end := -1, -1 + for i, l := range lines { + if start < 0 && strings.Contains(l, "datagrid dataGrid2_1") { + start = i + continue + } + if start >= 0 { + indent := lines[start][:len(lines[start])-len(strings.TrimLeft(lines[start], " "))] + if l == indent+"}" { + end = i + break + } + } + } + if start < 0 || end < 0 { + t.Fatalf("dataGrid2_1 not found in the description:\n%s", described) + } + grid := strings.Join(lines[start:end+1], "\n") + const from, to = "Caption: 'Name'", "Caption: 'Rule name'" + if !strings.Contains(grid, from) { + t.Fatalf("control: the fixture's first column caption changed:\n%s", grid) + } + replace := "mdl 1;\nalter page Rules.BusinessRule_Overview {\n replace dataGrid2_1 with {\n" + + strings.Replace(grid, from, to, 1) + "\n }\n};" + + before := h.pageUnit(t, "BusinessRule_Overview") + storedType, storedObj := pluggableParts(t, before, "dataGrid2_1") + if err := h.exec(replace); err != nil { + t.Fatalf("exec: %v", err) + } + after := h.pageUnit(t, "BusinessRule_Overview") + typ, obj := pluggableParts(t, after, "dataGrid2_1") + if string(typ) != string(storedType) { + t.Errorf("the stored Type was replaced by the template's") + } + diff := bsonDiff(storedObj, obj) + if len(diff) == 0 { + t.Fatal("the replace changed nothing — the caption was not written") + } + for _, d := range diff { + if !strings.Contains(d, "Rule name") { + t.Errorf("a property the statement does not state changed: %s", d) + } + } + + again := h.snapshot() + if err := h.exec(replace); err != nil { + t.Fatalf("second exec: %v", err) + } + if changed := again.diff(h.snapshot()); len(changed) > 0 { + t.Errorf("running the replace a second time wrote:\n %s", strings.Join(changed, "\n ")) + } +} diff --git a/mdl/roundtrip/testapp_allowlist_test.go b/mdl/roundtrip/testapp_allowlist_test.go index f9330a2964..15ea36da41 100644 --- a/mdl/roundtrip/testapp_allowlist_test.go +++ b/mdl/roundtrip/testapp_allowlist_test.go @@ -184,7 +184,6 @@ var testAppKnownFailures = map[string]knownFailure{ "page WorkflowCommons.WorkflowUserTaskView_View": {laws: []law{lawGetPut}, issue: "#721 #826", why: "executes since describe prints snippet call Params (#826); the page Appearance keeps 1 of 3 DesignProperties (#721 C)"}, "page WorkflowCommons.Workflow_Dashboard": {laws: []law{lawGetPut}, issue: "#721", why: "page: breaks getput on TestApp, measured when it joined the harness (#743); not yet triaged into #721's classes"}, "page WorkflowCommons.Workflow_JumpTo_Options": {laws: []law{lawGetPut}, issue: "#721", why: "page: breaks getput on TestApp, measured when it joined the harness (#743); not yet triaged into #721's classes"}, - "published rest service Services.OrdersRestApi": {laws: []law{lawGetPut}, issue: "#721", why: "published rest service: breaks getput on TestApp, measured when it joined the harness (#743); not yet triaged into #721's classes"}, "rest client Clients.OrdersRestClient": {laws: []law{lawParse}, issue: "#721", why: "rest client: breaks parse on TestApp, measured when it joined the harness (#743); not yet triaged into #721's classes"}, "rest client Mappings.RestClient": {laws: []law{lawParse}, issue: "#721", why: "rest client: breaks parse on TestApp, measured when it joined the harness (#743); not yet triaged into #721's classes"}, "rule Rules.Rule1": {laws: []law{lawGetPut}, issue: "#721", why: "rule: breaks getput on TestApp, measured when it joined the harness (#743); not yet triaged into #721's classes"}, diff --git a/mdl/scriptdiff/copy.go b/mdl/scriptdiff/copy.go new file mode 100644 index 0000000000..a2403260f5 --- /dev/null +++ b/mdl/scriptdiff/copy.go @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: Apache-2.0 + +package scriptdiff + +import ( + "fmt" + "io" + "os" + "path/filepath" +) + +// skippedDirs are directories of a project folder the scratch copy leaves out: +// version-control metadata, build output and package caches. Exec reads none of +// them and writes none of them, and together they are most of a project folder's +// size once it has been built or deployed. +var skippedDirs = map[string]bool{ + ".git": true, + ".svn": true, + "deployment": true, + "releases": true, + "node_modules": true, + ".mendix-cache": true, + ".mendix-cache2": true, +} + +// copyProject copies the project folder src into dst, which must not exist. +// +// Everything is a real copy rather than a link: exec writes the .mpr, the unit +// files and, for some statements, files next to the project (java and +// javascript sources, theme files), and a hard link or a symlinked directory +// would carry such a write back into the project being diffed. A symlink in the +// project is copied as a symlink, which is the one exception, and the same one +// `cp -a` makes. +// +// The folder dst is created in is left out too: it is the scratch folder, and +// it sits inside the project when TMPDIR does, where copying it would copy the +// copy into itself until the path is too long. +func copyProject(src, dst string) error { + dstParent, err := os.Stat(filepath.Dir(dst)) + if err != nil { + return err + } + return filepath.Walk(src, func(p string, info os.FileInfo, err error) error { + if err != nil { + return err + } + rel, err := filepath.Rel(src, p) + if err != nil { + return err + } + target := filepath.Join(dst, rel) + switch mode := info.Mode(); { + case mode&os.ModeSymlink != 0: + link, err := os.Readlink(p) + if err != nil { + return err + } + return os.Symlink(link, target) + case info.IsDir(): + if rel != "." && (skippedDirs[info.Name()] || os.SameFile(info, dstParent)) { + return filepath.SkipDir + } + return os.MkdirAll(target, 0o755) + case mode.IsRegular(): + return copyFile(p, target, mode.Perm()) + default: + return nil // sockets, devices: nothing exec reads + } + }) +} + +func copyFile(src, dst string, perm os.FileMode) error { + in, err := os.Open(src) + if err != nil { + return err + } + defer in.Close() + out, err := os.OpenFile(dst, os.O_CREATE|os.O_EXCL|os.O_WRONLY, perm|0o200) + if err != nil { + return err + } + if _, err := io.Copy(out, in); err != nil { + out.Close() + return fmt.Errorf("copy %s: %w", src, err) + } + return out.Close() +} diff --git a/mdl/scriptdiff/helpers_test.go b/mdl/scriptdiff/helpers_test.go new file mode 100644 index 0000000000..9459d6e07f --- /dev/null +++ b/mdl/scriptdiff/helpers_test.go @@ -0,0 +1,168 @@ +// SPDX-License-Identifier: Apache-2.0 + +package scriptdiff + +import ( + "bytes" + "errors" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + modelsdkbackend "github.com/mendixlabs/mxcli/mdl/backend/modelsdk" + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +func fileEngine() backend.FullBackend { return modelsdkbackend.New() } + +// pedAppCopy copies the committed PedApp fixture (Studio Pro-authored) into a +// temporary folder and returns its .mpr. +func pedAppCopy(t testing.TB) string { + t.Helper() + src := filepath.Join("..", "..", "testdata", "pedapp") + if _, err := os.Stat(filepath.Join(src, "PedApp.mpr")); err != nil { + t.Skipf("PedApp fixture not found: %v", err) + } + dst := filepath.Join(t.TempDir(), "PedApp") + if err := copyProject(src, dst); err != nil { + t.Fatal(err) + } + return filepath.Join(dst, "PedApp.mpr") +} + +func parse(t testing.TB, script string) *ast.Program { + t.Helper() + prog, errs := visitor.Build(script) + if len(errs) > 0 { + t.Fatalf("parse: %v\n%s", errs[0], script) + } + return prog +} + +// execScript runs a script the way `mxcli exec` does after its pre-flight: a +// fresh executor, connected to the project, running the program. +func execScript(t testing.TB, mpr, scriptDir string, prog *ast.Program, continueOnError bool) (string, error) { + t.Helper() + var out bytes.Buffer + x := executor.New(&out) + x.SetBackendFactory(fileEngine) + if scriptDir != "" { + x.SetScriptDir(scriptDir) + } + defer x.Close() + if err := x.Execute(&ast.ConnectStmt{Path: mpr}); err != nil { + t.Fatalf("connect: %v", err) + } + var err error + if continueOnError { + var fails bytes.Buffer + _, err = x.ExecuteProgramContinueOnError(prog, &fails) + out.WriteString(fails.String()) + } else { + err = x.ExecuteProgram(prog) + } + if errors.Is(err, executor.ErrExit) { + err = nil + } + _ = x.Execute(&ast.DisconnectStmt{}) + return out.String(), err +} + +func snap(t testing.TB, mpr string) *Snapshot { + t.Helper() + s, err := TakeSnapshot(mpr) + if err != nil { + t.Fatal(err) + } + return s +} + +// writeSet is a run's writes as comparable lines: " ", +// and "file ". +func writeSet(units []UnitChange, files []FileChange) []string { + var out []string + for _, u := range units { + k := string(u.Kind) + if u.Kind == Modified && !u.Rewritten { + k = "moved" + } + out = append(out, fmt.Sprintf("%s %s %s %s", k, u.ID, u.Type, u.Name)) + } + for _, f := range files { + out = append(out, fmt.Sprintf("file %s %s", f.Kind, f.Path)) + } + sort.Strings(out) + return out +} + +// symmetricDiff lists the lines only in a (prefixed "diff only: ") and only in +// b ("exec only: "). +func symmetricDiff(a, b []string) []string { + in := func(s []string) map[string]bool { + m := map[string]bool{} + for _, x := range s { + m[x] = true + } + return m + } + ma, mb := in(a), in(b) + var out []string + for _, x := range a { + if !mb[x] { + out = append(out, "diff only: "+x) + } + } + for _, x := range b { + if !ma[x] { + out = append(out, "exec only: "+x) + } + } + return out +} + +// diffThenExec runs diff on the project, checks it wrote nothing, then runs +// exec on it, and returns what each said is written. New units get fresh IDs +// in each run, so an added unit is compared by type and name only. +func diffThenExec(t testing.TB, mpr, scriptDir string, prog *ast.Program, continueOnError bool) (rep *Report, diffSet, execSet []string, execOut string, execErr error) { + t.Helper() + before := snap(t, mpr) + rep, err := Run(mpr, prog, Options{NewBackend: fileEngine, ScriptDir: scriptDir, ContinueOnError: continueOnError}) + if err != nil { + t.Fatalf("diff: %v", err) + } + if u, f := before.Compare(snap(t, mpr)); len(u)+len(f) > 0 { + t.Fatalf("diff wrote to the project it diffs:\n %s", strings.Join(writeSet(u, f), "\n ")) + } + execOut, execErr = execScript(t, mpr, scriptDir, prog, continueOnError) + u, f := before.Compare(snap(t, mpr)) + return rep, comparable(writeSet(rep.Units, rep.Files)), comparable(writeSet(u, f)), execOut, execErr +} + +// comparable drops the unit ID of an added unit, which each run mints afresh. +func comparable(lines []string) []string { + out := make([]string, len(lines)) + for i, l := range lines { + if strings.HasPrefix(l, "added ") { + parts := strings.SplitN(l, " ", 3) + l = "added " + parts[2] + } + out[i] = l + } + sort.Strings(out) + return out +} + +// parseOK parses a script, returning its first error instead of failing. +func parseOK(script string) (*ast.Program, error) { + prog, errs := visitor.Build(script) + if len(errs) > 0 { + return nil, errs[0] + } + return prog, nil +} diff --git a/mdl/scriptdiff/outside_test.go b/mdl/scriptdiff/outside_test.go new file mode 100644 index 0000000000..42314b6f0c --- /dev/null +++ b/mdl/scriptdiff/outside_test.go @@ -0,0 +1,109 @@ +// SPDX-License-Identifier: Apache-2.0 + +package scriptdiff + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// A headerless (mdl 0) script may CONNECT, and exec follows it. diff runs the +// script for real, so a connect it followed would write to the project being +// diffed — while reporting that exec writes nothing, because the copy it +// compares saw no write. A connect to the project itself is followed on the +// copy instead. +func TestDiffFollowsAConnectToTheProjectOntoTheCopy(t *testing.T) { + mpr := pedAppCopy(t) + prog := parse(t, "connect local '"+mpr+"';\n"+ + "create persistent entity MyFirstModule.DiffConnectEscape (Name: String(50));\n") + + // diffThenExec fails the test if diff writes to the project. + rep, diffSet, execSet, out, err := diffThenExec(t, mpr, "", prog, false) + if err != nil { + t.Fatalf("exec: %v\n%s", err, out) + } + if len(execSet) == 0 { + t.Fatal("exec wrote nothing, so this is no test of diff") + } + if d := symmetricDiff(diffSet, execSet); len(d) > 0 { + t.Errorf("diff disagrees with exec:\n %s\n%s", strings.Join(d, "\n "), render(rep)) + } +} + +// A connect to any other project is refused: diff has no copy of it. +func TestDiffRefusesAConnectToAnotherProject(t *testing.T) { + mpr := pedAppCopy(t) + other := pedAppCopy(t) + before := snap(t, other) + prog := parse(t, "connect local '"+other+"';\n"+ + "create persistent entity MyFirstModule.DiffConnectEscape (Name: String(50));\n") + _, err := Run(mpr, prog, Options{NewBackend: fileEngine}) + if err == nil || !strings.Contains(err.Error(), "outside the scratch copy") { + t.Errorf("diff of a connect to another project: err = %v, want a refusal", err) + } + if u, f := before.Compare(snap(t, other)); len(u)+len(f) > 0 { + t.Fatalf("diff wrote to the other project:\n %s", strings.Join(writeSet(u, f), "\n ")) + } +} + +// The same holds for a connect inside a script the script executes. +func TestDiffRefusesAConnectInANestedScript(t *testing.T) { + mpr := pedAppCopy(t) + other := pedAppCopy(t) + before := snap(t, other) + dir := t.TempDir() + nested := "connect local '" + other + "';\n" + + "create persistent entity MyFirstModule.DiffConnectEscape (Name: String(50));\n" + if err := os.WriteFile(filepath.Join(dir, "nested.mdl"), []byte(nested), 0o644); err != nil { + t.Fatal(err) + } + prog := parse(t, "execute script 'nested.mdl';\n") + _, err := Run(mpr, prog, Options{NewBackend: fileEngine, ScriptDir: dir}) + if err == nil || !strings.Contains(err.Error(), "outside the scratch copy") { + t.Errorf("diff of a nested connect: err = %v, want a refusal", err) + } + if u, f := before.Compare(snap(t, other)); len(u)+len(f) > 0 { + t.Fatalf("diff wrote to the other project:\n %s", strings.Join(writeSet(u, f), "\n ")) + } +} + +// SQL queries and IMPORT act on databases, not on the project: diff cannot +// run them on a copy, and refuses rather than running them for real. +func TestDiffRefusesStatementsThatActOnADatabase(t *testing.T) { + for name, stmt := range map[string]ast.Statement{ + "sql query": &ast.SQLQueryStmt{Alias: "db", Query: "delete from customers"}, + "import": &ast.ImportStmt{SourceAlias: "db", Query: "select 1", TargetEntity: "MyFirstModule.Customer"}, + } { + t.Run(name, func(t *testing.T) { + mpr := pedAppCopy(t) + prog := &ast.Program{Statements: []ast.Statement{stmt}} + _, err := Run(mpr, prog, Options{NewBackend: fileEngine}) + if err == nil || !strings.Contains(err.Error(), "outside the scratch copy") { + t.Errorf("err = %v, want a refusal", err) + } + }) + } +} + +// The scratch folder may sit inside the project folder (TMPDIR=./tmp in the +// project); the copy must not copy itself into itself. +func TestDiffWithTheScratchFolderInsideTheProject(t *testing.T) { + mpr := pedAppCopy(t) + tmp := filepath.Join(filepath.Dir(mpr), "tmp") + if err := os.Mkdir(tmp, 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("TMPDIR", tmp) + prog := parse(t, defaultSpellings) + rep, err := Run(mpr, prog, Options{NewBackend: fileEngine}) + if err != nil { + t.Fatalf("diff: %v", err) + } + if got := render(rep); !strings.Contains(got, "Summary: 1 new, 0 modified, 0 removed") { + t.Errorf("diff:\n%s", got) + } +} diff --git a/mdl/scriptdiff/property_test.go b/mdl/scriptdiff/property_test.go new file mode 100644 index 0000000000..30d9132fc0 --- /dev/null +++ b/mdl/scriptdiff/property_test.go @@ -0,0 +1,143 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package scriptdiff + +import ( + "bufio" + "os" + "path/filepath" + "regexp" + "strings" + "testing" +) + +// The property `mxcli diff` exists for (ako/mxcli#907): the documents diff +// reports changed are exactly the units exec writes. For every script, diff +// runs first and must not touch the project; exec then runs on the same +// project; the two sets of writes must be equal. Each script then runs a +// second time, which is the twice-exec rule seen from diff: whatever exec 2 +// writes (usually nothing) is what diff 2 reports. +// +// Every statement runs, as `exec --continue-on-error` runs them, so a script +// with one failing statement still compares everything else it does. +// +// By default a handful of mdl-examples doctype scripts covering the document +// kinds diff used not to compare run on the PedApp fixture, each on its own +// copy. MXCLI_DIFF_ALL=1 runs every doctype script. MXCLI_DIFF_LEGS names a +// file of legs, one per line — "\t\t