Feature Request: All Rules are "Top-Level" Functions

[rneswold]

This is a great project and I've used it in a handful of Rust applications. Thanks!

There is one feature from the OCaml tools (ocamllex and ocamlyacc) have that I found convenient. In the code that ocamlyacc generates, all the grammar rules create functions you can call (if you need to parse a subset of the grammar.)

For instance, I'm working on a project that uses a string to describe data acquisition. There's a "name" field, optional range specification, option field names, and a field for the event on which to sample. All of that is straightforward to implement. However, we also have an API which would like just the event portion and another API that just wants the device name.

We could have a struct with a bunch of Option fields, but I'd rather be able to have Event as a parameter to a function.

If this feature would be too disruptive, I wonder how others would solve this with the current tools.

[ltratt]

Right now, grmtools doesn't generate functions in this way. I guess it's possible to do so and to call "into the middle of" the LR statetable. I haven't thought about that before and it might need a bit of thought about exactly what it means.

In the interim, I think there is a (horrible) hack one can do: you can duplicate the grammar (including in a build.rs file), change the %start line and output to a different file(s).

[ratmice]

Comes to mind that with the horrible hack, is seems undoubtedly likely to produce the 'unused rule' and 'unused token' warnings/errors. You'll likely need to set at least warnings_are_errors and more than likely want to set show_warnings to false entirely.

[ltratt]

@ratmice Definitely! It would be good to do something nicer here, though an interesting question is what "unused" means if you have multiple start rules.

[rneswold]

What if I created a grammar for each portion and then a grammar that called the subsets? I'd have to know where the previous parsing ended to feed the next one. I'd be nicer to have it all in one module, but this might be doable...

[rneswold]

Comes to mind that with the horrible hack, is seems undoubtedly likely to produce the 'unused rule' and 'unused token' warnings/errors.

I would delete the unused rules in the "horrible hack". However, I was hoping to use the same lex file, so the "unused token" warnings would be a problem.

[ratmice]

@ratmice Definitely! It would be good to do something nicer here, though an interesting question is what "unused" means if you have multiple start rules.

Interesting question, my inclination would be to define unused as unreachable from any start rule.

I'm assuming this is considering some sort of feature that lifts the (current) restriction that there is a single start rule,
and making some sort of parser entry point for each start rule?

So for the purposes of checking unused rules the start rule would be treated as though each start rule were a production of an implicit rule, as in the following.

%start start1 start2
^: start1 | start2

I don't think it would be hard to to modify the unused_symbols function that does these checks to work in that way at least.

[rneswold]

So for the purposes of checking unused rules the start rule would be treated as though each start rule were a production of an implicit rule, as in the following.

%start start1 start2
^: start1 | start2

I don't think it would be hard to to modify the unused_symbols function that does these checks to work in that way at least.

This is nice because I really don't need every rule to be top-level. Out of the 1/2 dozen sections of my string, I really only need 3 field parsers. But it might be too complicated to have some rules be top-level callable and others purely internal.

Also, each of these start targets is probably a different type (in my use, that's definitely the case.)

[ratmice]

Also, each of these start targets is probably a different type (in my use, that's definitely the case.)

Ahh, yeah there are definitely complexities with this multiple start rules idea, that is one I hadn't considered.
Another is that IIRC the start rule is given index 0 by default, rather than something like rules.len() which might be a more expandable location. But at least I think it seems like a reasonable interpretation of unused.

IIRC anyways, though I couldn't remember exactly where to look off-hand to verify this 0 index rule in the moment.

[ajuvercr]

Is there a way to import .y files in other .y files?
By 'inversing' the dependencies, it is probably prettier then copying the file many times.

Nvm I saw #110

[ltratt]

Composition in a general sense is a very hard problem. This issue (relative to my memory of #110) is more limited: it's asking to subset an existing grammar. My intuition is that subsets are always OK -- we just don't happen to support taking advantage of that right now.

For example, if we generated Rust code directly, specifically one Rust function per rule, I suspect this would fall out of the hat. Doing so isn't rocket science (I think it's what e.g. LALRPOP does), but someone has to put in the hard yards.

[rneswold]

I think what I'll do is use an enumeration. Something like:

enum DAQSpec {
    FullSpec { device_name: String,
               field: String,
               range: Option<Range<usize>>,
               event: Event },
    DeviceSpec(String),
    EventSpec(Event)
}

The grammar can recognize when a subset is specified and return the smaller-scoped enum values. Then I can make some simple wrapper functions:

fn parse_event(spec: String) -> Option<Event> {
    if let DAQSpec::EventSpec(ev) = parsing the input {
        Some(ev)
    } else {
        None
    }
}

Thanks for the discussion!

[ratmice]

I don't recall it being mentioned here, and I hadn't run across it before,
but there is an interesting mechanism for this discussed in the bison faq.

https://www.gnu.org/software/bison/manual/html_node/Multiple-start_002dsymbols.html

Which is declaring for each rule we want to act like a start rule declare a synthetic %token
that token gets passed in as the initial token of each entry point's production.

Example from the manual (sic):

%token START_FOO START_BAR;
%start start;
start:
  START_FOO foo
| START_BAR bar

This can undoubtedly be done right now using a manual lexer

I suppose it seems like if one iterates over the token_map
partitioning it into two token maps (one for the synthetic start rule tokens), and one for the actual tokens.

Then bind the actual tokens to the lexer (the old way) using rule_ids_map
Maybe even calling CTTokenMapBuilder on the partitioned set of
synthetic start rule tokens.

Potentially using the manual lexer just to produce the first token/entry point, which proceeds the actual lrlex stack of tokens.
If I understand correctly at some point lalrpop may switched to this method for multiple pub tokens, away from cloning the entire parser and substituting the start rule (This is where I heard of this). Because this ends up sharing the parser tables between the parsers.

Anyhow there are some good (or perhaps silly) questions here:

1. Is it indeed possible to do with the manual lexer (Having tested this one slightly, it does appear to work).
2. Is it also possible to utilize the rule mapping method, splitting the tokens maps to have a half manual/half lrlex built lexer
3. Would it be possible to provide some syntax sugar in the code generator to automate the manipulation of the token/rule mapping
   generating say an entry point that automatically provides the initial token.

I haven't tried the second yet, but one thing about it I don't see how to workaround is when combining the lexemes from the manual lex producing the synthetic tokens, and the real lrlex, because of the method signatures it seems like we have to reallocate a vec with an additional element. It might be nice if there was a way to avoid that like an entry point where you passed the first token separately from the vec?