hal: Add halcompupdate to migrate .comp files to the new HAL API

[grandixximo]

Problem

The HAL API break is in (#4565): pins and parameters are read and written through typed getters and setters, and HAL is 64-bit only. halcompile still accepts the legacy declaration types (float, bit, s32, u32, s64, u64, signed, unsigned) with a warning, but an out-of-tree .comp that writes a pin or parameter with plain C assignment (out = x;, count++;, *out_ptr = x;) no longer compiles. That is nearly every component that has an output.

Solution

A new tool, halcompupdate, rewrites legacy .comp files to the new API:

Area	Conversion
Declaration types	float->real, bit->bool, s32->si32, u32->ui32, s64->sint, u64->uint, signed->si32, unsigned->ui32 (port is left alone, no new-style replacement yet)
Writes to out/io pins and params	x = e -> x_set(e), x += e -> x_set(x + (e)), x++ -> x_set(x + 1), arrays x(i) = e -> x_set(i, e), chained a = b = e -> a_set(b_set(e)), *x_ptr = e -> x_set(e), *x_ptr reads -> (x)
#define macros	bodies are rewritten too, with macro parameters correctly shadowing same-named pins
Legacy C types	double/real_t -> rtapi_real; hal_bit_t -> volatile rtapi_bool etc., the volatile qualifier is kept so the conversion is semantics-identical, and each conversion is reported with its line number for review (--no-c-types to skip)
Reads	unchanged, the bare name (or name(idx)) works in both APIs

s32 and u32 become si32 and ui32, which keep the component's 32-bit behaviour. Moving on to the 64-bit sint and uint depends on whether the code relies on the 32-bit range or on wrap-around, so the tool leaves that to the author and notes each such declaration with its line.

Pin and parameter names are never renamed, because they are part of the HAL configuration interface, and existing .hal files keep working. A name that spells a type word is noted with the spelling the in-tree renames use (out_s32 -> out_sint), and the rename is left to the author.

Usage:

halcompupdate mycomponent.comp        # show a diff of the required changes
halcompupdate -i mycomponent.comp     # rewrite in place (keeps a .bak)
halcompupdate --check *.comp          # exit 1 if migration is needed

halcompile's own warning for a legacy declaration type now ends with "halcompupdate(1) converts a .comp."

Safety

• In-place rewriting is atomic (temp file + os.replace(), original mode preserved). Backups are created with O_EXCL|O_NOFOLLOW and a unique-name fallback, so a pre-existing .bak (or a planted symlink) is never clobbered.
• Constructs that cannot be converted safely are left unchanged with a warning instead of producing broken or silently different code: taking the address of a pin/param (&x), direct use of the legacy hal_pin_*_new/hal_param_*_new creation API, postfix ++/-- whose value is used (the setter yields the new value, postfix must yield the old), array indices with side effects, pointers to legacy HAL types, and hal_*_t types in variable declarations (the grammar allows only a single-word type there, so the volatile qualifier cannot be preserved).
• Writes to pins/params in EXTRA_SETUP are converted but warned about: the setter references are initialized only after extra_setup() runs.
• Every file ends with a summary line: how many mechanical changes were applied, how many constructs were left for manual review, how many pins and parameters were kept 32-bit, and how many names spell a type word.

Validation

• The 120 in-tree components that still used the legacy types before their hand conversion, taken from the tree of 2026-07-01, converted with the tool and compiled with today's halcompile from a plain directory (the out-of-tree case): 101 compile cleanly. Of the other 19, 5 (spindle, raster, mesa_pktgyro_test, homecomp, tpcomp) need in-tree headers or TOPDIR to build at all, and 14 use exactly what the tool leaves to the author with a warning: the legacy creation API, hal_*_t in a variable declaration, or a pointer into HAL memory.
• Regression test tests/halcompile/update-api covers declarations, writes, unsafe constructs, EXTRA_SETUP writes, volatile preservation, the 32-bit notes, name notes and the summary output.

Docs

Migration section in docs/src/hal/comp.adoc (type table and _set() accessor documentation), halcompupdate(1) manpage, and a SEE ALSO link in halcompile(1).

[grandixximo]

@BsAtHome If this is too ugly I'll drop it, this does most of your careful hand conversion, to alleviate the porting churn for comps out of tree, let me know how much you hate it ;-)
Keeping as draft for now....

[BsAtHome]

The old hal types should never be used anywhere anymore. All places I've see they were used wrongly as variable types and that fails the smell-test miserably because they are volatile.
For example, hal_bit_t must be rtapi_bool when used in as a variable. hal_bit_t * becomes hal_bool_t when dealing with pins. And hal_bit_t becomes hal_bool_t when dealing with parameters, BUT, when you deal with parameters, you need to adapt the code because parameters were used as variables.

For components (.comp files), that only declare their pins and params through halcompile means, will not see a difference (except for the _set() suffix for writing). However, those constructs that add their own pins/params need more care in the conversion.

[grandixximo]

Thanks for the review. Aligned the C-type conversion with that:

• hal_bit_t (and the other hal_t) used as variable types now convert to plain rtapi_bool/rtapi. The volatile is dropped, it was only there for direct HAL memory access.
• hal_bit_t * and friends are left unchanged with a warning. Pointers into HAL memory become hal_bool_t refs and need the manual hal_get/hal_set treatment, the tool does not pretend to do that part.
• Comps that only declare pins/params through halcompile get exactly the type rename plus set() treatment automatically. Comps using hal_pinnew/hal_paramnew directly are detected, their hal*_t types are kept intact, and a warning points at the manual part.

[BsAtHome]

FWIW, I don't think we should even attempt to fix the component files automatically.

The only risk-free change is to fixup pins/params to use the new types. Everything else is intrinsically a huge risk in an auto conversion that needs to consider the context it is used in. That is not something you can do automatically. You could do considerable damage.

IMO, the risks outweigh the benefits.

[grandixximo]

FWIW, I don't think we should even attempt to fix the component files automatically.

Partially I agree, but I don't like what the lack of an attempt says...

IMO, the risks outweigh the benefits.

The tool is strictly opt-in. It does not pop up and ask you to convert like the INI converter does, and doesn't convert automatically on build, it sits there until someone runs it, and its default mode only prints a diff for review.

The only risk-free change is to fixup pins/params to use the new types

Types and writes are inseparable. Converting pin out float x -> real while leaving x = 1; in the body breaks the comp immediately (assignment to a getter macro), while doing nothing keeps it compiling until the API break. So the only coherent options are mechanical conversion of both, or nothing but warnings, and IMO only the first one helps out-of-tree comps mentioned by @rene-dev at the meeting.

The tool already stays inside the risk-free zone you describe, it converts only the mechanical subset (type renames and direct writes.

The writable namespace is exactly the declared pins/params, no aliasing, no pointers beyond _ptr) and defers everything that needs context with a warning, unchanged: &pin, raw hal_pin_*_new, postfix ++/-- whose value is used, side-effect indices. Validated by converting all 124 in-tree comps and building each from a plain directory, which is the out-of-tree scenario: 119 compile cleanly. The other 5 (spindle, raster, mesa_pktgyro_test, homecomp, tpcomp) depend on in-tree-only headers or the TOPDIR skeleton and fail the same way before and after conversion, so those failures are unrelated to the tool.

[grandixximo]

@BsAtHome This needs a decision now that #4472 is merged.

I've rebased and added a commit pointing the hal_*_t deprecation warnings at halcompupdate(1), so merging or closing this PR settles the question in one action.

My position is unchanged: the tool is strictly opt-in, diff-only by default, converts only the mechanical subset, and is verified against all 124 in-tree comps. Not perfect, but for comps that only declare pins/params through halcompile it beats hand-editing every write. And "only fix the types" is not a middle ground, renaming types without converting the writes breaks the comp immediately.

If you still think the risks outweigh the benefits even in this form, I'll close it. But I'd like to hear why, and what would have to change for it to be acceptable.

[BsAtHome]

I don't think it is about the risk, at least not for us. That would be a user problem ;-)

The real question is whether it is worth the bother. Is the added value greater than the user-effort required to do a manual conversion?
It may also give you a false sense of security when conversion succeeds. I think we can assume that the quality of in-tree components is generally better than out-of-tree components. Then we should ask ourselves, how do these get converted?

I'm still quite ambivalent. Yes, it does help the user. No, it may cause hard problems. Do the advantages outweigh the (potential) problems?

[grandixximo]

On "worth the bother": a manual conversion is not just the type renames, it is every write to every pin and param, each one a chance to typo a setter or miss an array-index form. For a comp with a handful of writes that is trivial. For the long tail of comps with dozens of writes it is exactly the kind of mechanical, error-prone editing a script does better than a human. The in-tree comps got expert hand conversion; out-of-tree authors do not get that attention.

On false sense of security: the tool now ends every file with a summary, "N mechanical changes, M constructs left for manual review; review the diff and test before use". Default mode prints only a diff, nothing is written unless asked, --in-place keeps a .bak. Everything the tool cannot prove safe (pointers, volatile qualifiers, raw creation API, side-effect indices, EXTRA_SETUP writes) is left unchanged with a specific warning. I do not claim it makes conversion safe; it makes the safe part cheap and the unsafe part loud.

And the alternative to the tool is not careful manual conversion of out-of-tree comps, it is no conversion at all: they simply fail to build after the API break. For those authors an imperfect opt-in helper beats nothing.

[BsAtHome]

Okay, fine then. Let me take another look and see if I can warm to the idea of auto conversion ;-)

[BsAtHome]

There is one more thing that I noticed when doing conversion. There are several pins/params that have s32 or u32 in their name. This is mostly not appropriate when we kill off the 32-bit version. At least all the names that will be moving to sint and uint should not have that name.
Currently, my WIP tree is 64-bit only and (mostly) functional. For some, mux_generic.c and demux_generic.c, I've added a pin name alias. The .comp components will get a new line in halcompile alias pinname aliasname;. Aliases are already supported by HAL. It should function properly when building nets as hal_link searches for names and aliases. Need to checkup on all paths, also in HAL query.

[grandixximo]

One thing I found while reading mux_generic.c on your WIP branch: both alias format strings use the demux-gen. prefix, lines 226 and 278, while every real pin in that file uses mux-gen.. Looks like it came across from demux_generic.c.

On the alias line in halcompile, some questions before I form an opinion:

Would the directive cover params as well as pins, or pins only? hal_param_alias() exists, so the asymmetry would be a choice rather than a limitation.

Which name becomes canonical? In mux_generic.c the pin is created with the old type name and the new one is the alias, so show pin keeps printing in-s32-00 and in-sint-00 is secondary. Is that the intended direction, given the goal is to get the type out of the name?

And what happens to halcmd save all? It calls save_aliases(), so aliases created by a component appear in a user's saved config as explicit alias pin ... lines they never wrote, plus a permanent entry under show alias. Is that acceptable, or does a comp-declared alias need to be distinguishable from a user-declared one?

I ask because there is no precedent for a component shipping aliases; so far they have been purely a user-side tool. For halcompupdate none of this changes anything: it does not rename and I do not intend it to, an alias makes a rename non-breaking but not mechanical. Would you want it to at least point the names out, a gentle warning naming pins and params whose name mentions a type that no longer exists, leaving the decision to the author?

[BsAtHome]

Yeah, there is some copy/paste in there... :-)
All unreviewed hacks so far.

Both pins and params should be aliased when using halcompile. Working on it in my tree (pins done).

The current format:

# Previously:
pin in sint name-s32;

# Now:
pin in sint name-sint;
alias name-sint name-s32;

That generates the same "visible" pin list and moves the alias name into oldname. The hal_pin_alias() does something like:

• unlink pin from list
• oldname = pinname
• pinname = aliasname
• relink pin into list

It is not always easy to get rid of the type name. See for example hal/components/reset.comp.

The question is whether it should be:

1. prefix.name-s32 + prefix.name-sint
2. prefix.name-sint + prefix.name-s32

The first keeps the same names visible as everybody was used to. The second makes the new name visible while still allowing the old name. Either way works. Not sure which is the best (probably the second).

What happens to halcmd... Well, I renamed the types, all of them. So there are already a lot of changes coming. Saving the list will already change the line.

The user alias question is a good one. No opinion as of yet. The real question is how hard we are going to break every config out there. Probably best to break it completely. There also is this issue of the conv-*-*.comp converters. They have the type in the filename and therefore the component name. That sucks too. So that would argue for making a complete break and drop the aliases in the stock components.

And, yes, halcompupdate should probably warm if it detects type names in the pin/param name.

[BsAtHome]

You are right.

Removed the in-tree component aliases. Still implemented it in halcompile for integrators if they wish that much.

The conv-X components rename will break some substantial portion of user's configs. Therefore, the aliases are a patch that would hurt more than you get out of it. Once you need to redo the config, then you should redo it thoroughly.

That makes me wonder whether a similar tool as this will be helping to rewrite .hal files for the user. Could even get the file list from an INI file... Want to hack a bit more? ;-)

[grandixximo]

Yes, happy to hack on that. Here is the shape I would propose before I write any of it, because it deliberately does not work like the INI updater.

The trigger is the error, not the tool. No dialog, no prompt on startup, nothing runs automatically. A user starts the config that worked yesterday, hits the failure at the exact line that uses a name we removed, and the error message itself names the replacement and mentions the converter. Users who never touch a renamed name never hear about it.

This is safe here in a way the INI updater is not: the old names are gone, so every affected config fails deterministically. There is no silent-wrong-behaviour class to worry about, and the mapping is not a guess: conv_s32_float was removed, conv_sint_real exists, one entry, no context needed.

Pieces:

A generated oldname => newname table (component names, pin names, param names) in hal_lib, produced from the same source as the type conversion so it cannot drift.

One lookup on three existing error paths, only after the call has already failed, so nothing costs anything when the config is fine:

• halcmd_commands.cc:985, insmod for %s failed, the loadrt conv_s32_float case, which is the first line most configs will trip on
• halcmd_commands.cc:408, Pin '%s' does not exist, for net and linkps
• halcmd_commands.cc:673, the generic setp/setparam path

Output would read roughly:

foo.hal:42: Pin 'conv-s32-float.in' does not exist
  This name was removed in 2.10~pre2; it is now 'conv-sint-real.in'.
  'halnameupdate foo.hal' converts the names in that file.

And the converter itself, shaped exactly like halcompupdate: diff by default, -i rewrites in place keeping a .bak, --check for CI. It rewrites only names in the table and leaves everything else alone. The primary input is a HAL file, because that is what the error message just handed the user; taking an INI is a convenience on top, for converting a whole config in one go:

halnameupdate myfile.hal        # diff
halnameupdate -i myfile.hal     # rewrite in place, keeps a .bak
halnameupdate --check *.hal     # CI mode
halnameupdate mymachine.ini     # every HAL file the INI lists

The name is not precious to me, so if you would rather it were called something else, suggest away.

Two limitations I want to state up front rather than discover later. Not every failure goes through halcmd: the Python hal module, custom GUIs, halui bindings in the INI, and pyvcp/qtvcp widget pin names in XML fail elsewhere. That is an argument for the table living in hal_lib where any consumer can query it, and for the INI mode also scanning the GUI files the INI references, but the coverage will not be total. And this does put a legacy table in HAL, so I would write the expiry into the commit when it lands, say dropped two releases after the break, rather than letting it become permanent.

Does this shape look right to you?

[BsAtHome]

Embedding lists with things that changed in an executable may not be the best of ways to "fix" a situation. Reminds me of eternal backward compatibility code. You also say, you cannot catch al instances anyway, so the argument for "safe conversion" is not really applicable because you cannot fix everything automatically. Having a table of names in hal_lib is even worse and reminds me of the "wonderful" world of (shattered) windows.

We decided we were going to break people's configs. We can help them overcome the version change. We should not try to fix all problems. Making the conversion easier is all that we can do and should not try to burden our own code with more legacy.

Going through the .hal files is probably the best estimate to fix most of the user's pain. If people are using pins from other code, then that cannot be fixed automatically without full semantic analysis. Just like halcompupdate, you will not be doing this automatically. However, there will be more configurations that need a .hal file update than there will be local out-of-tree components.

You really don't know the pin names for sure without going through all the logic. Components can be loaded with a names= argument (several configs have this). Then the pin names will be different too. This is good for conversion because components with names= keep the pin names constant over the change of component name. However, there is also the possibility that you now have (a tiny chance for) false positives. The only really safe automatic conversion is to track the loadrt and fix pin names that match the loadrt statement (also with count=).
BTW, you cannot rely on reading one .hal file only. In some stock configs it was seen that the loadrt and the pin usage are split over files (had a count=3). That means only an ini-file search and convert .hal files in sequence with remembering what is there will be required if you want to play it safe.

I'm not sure we need a CI check. We do all conversion beforehand and all sims should be tested. There are some that even today need updating. However, a dry-run mode that just tells you what would be done is nice. That would be nice for a single .hal file as well as ini-file batch mode. I think you called it "diff" ;-)

halfileupdate myfile.hal        # dry-run diff single file
halfileupdate mymachine.ini     # dry-run diff every HAL file the INI lists
halfileupdate -i myfile.hal     # rewrite in place, keeps a .bak
halfileupdate -i mymachie.ini   # rewrite in place every HAL file the INI lists, keeps a .bak

My color is halfileupdate or updatehalfile, depending light or dark mode :-)

We may be somewhat fortunate in that there will be an INI-file version update too (if we get that code working). We can attach the halfileupdate to the ini update.

[grandixximo]

@BsAtHome Rebased onto the break. With #4565 in, halcompile still takes the old declaration types with a warning, but a comp that writes a pin with plain assignment no longer compiles, which is most out-of-tree comps. That is the part this tool converts.

Two changes since you last looked:

• s32 and u32 still become si32 and ui32, so the behaviour is unchanged. Whether a comp can move to sint depends on its code, so each one gets a note with its line instead of being widened.
• Your halcompile warning for an old declaration type now ends with "halcompupdate(1) converts a .comp." This is the halcompile pointer I mentioned after dropping the hal.h one.

I checked it on the 120 in-tree comps as they were before your conversion: converted and compiled with today's halcompile, 101 build, and the other 19 hit exactly what the tool leaves to the author with a warning, or need in-tree headers.

[BsAtHome]

I think this needs to include a strong note that they need to do a 64-bit clean analysis and move the [su]i32 types to proper [su]int types. The note should include that a code analysis is required to prevent inadvertent truncation in intermediate storage and calculation.

[grandixximo]

Done

[grandixximo]

@BsAtHome A question on the docs: hal.h says the si32/ui32 getters and setters are only there for compatibility and may be removed once the remaining code is updated, but also that there's a case for keeping them. Are you still heading towards removing si32 and ui32? If so, I'll describe them in comp.adoc as due for removal rather than kept for compatibility.

[BsAtHome]

That is a hard question. I'm undecided whether we leave it to the compiler or that we do this by name.
If we remove the [su]i32 version and replace them unconditionally with [su]int, then the compiler may choose a different compile path because the word sizes change. That could potentially change the semantics.

The setters can, in principle, be removed. The compiler will do the word size extension. However, there is a possibility of sign mismatch because the setters are "functions" and the arguments are first cast and then sign-extended. Using the 64-bit setter would probably do the sign-extension before the cast. This needs to be tested.

[grandixximo]

Ok, will keep it on a softer

they may be retired once components are 64-bit clean.

We will see how thing go...