What's your favorite approach to lexical analysis (i.e. turning a stream of bytes/characters into a stream of tokens)?
14
3
15
Conversation
I was raised in the school of parser combinators, which, among other things, means "No lexer, only parse."
2
6
I used to skip lexing too. In my experience, parsing with tokens is a lot nicer than parsing with characters, so I tend not to skip it anymore.
2
5
That said, parser combinators make it easy to do both in a single step, and it's not hard to keep the "token rules" separate from the "parser rules" conceptually. So it's good :)
3
3
IIRC some parser combinator libs let you parse custom streams too? So you can feed a token string->token parser to a token->ast parser?
Totally! Might be speaking through me, but I thought that was the standard way of implementing lexers when you're already using parser combinators!
1