Conversation

Looks ok I think! Tbh I do prefer separating out the lexing stage, from parsing, even though in PEG you can sometimes get away with smooshing it all together. Not sure if pest supports separate lexers though... 🤔

3:25 AM · Oct 4, 2018Twitter Web Client

This Tweet was deleted by the Tweet author. Learn more

CAD97

@CAD97_

Oct 5, 2018

Pest doesn't support a separate lexing phase. This isn't an innate restriction of PEG (any grammar just works on an alphabet, after all; it doesn't matter if that's Unicode or some token set), but I don't think pest intends to support non-&str parsing.

Show replies

Replying to

and

Food for thought: A lexerless parser with a discrete set of terminal productions is isomorphic to a pull-lexer parser where you don't tokenize the whole thing first. It's for this reason that I'm less vocal about preferring a separate lexer now. What I want requires pull lexing

Brendan Zabarauskas

@brendanzab

Oct 5, 2018

Yeah, I know they are the same, I just prefer the separation of concerns… I'm cool with both phases being pegs though.