What's your favorite approach to lexical analysis (i.e. turning a stream of bytes/characters into a stream of tokens)?
14
3
15
Conversation
My approach is usually ... just to do it by hand.
But depending on the language I also like to throw in the smallest amount of parsing by matching parentheses/brackets/braces at the same time. It's not very elegant but it makes the output immediately useful.
1
1
8
By "just to do it by hand", I mean it's usually a big loop with a switch statement with a getc-equivalent, manually keeping track of line/column numbers ... it's great. 😿
2
1
8
Replying to
All you need to keep track of is the byte offset. Iine+column number can be figured out when rendering the diagnostic.
Replying to
Is that better? Need to think about it.
Could also just store the column and keep track of newlines some other way.
1
It's nice because you only need to keep track of the byte offset through the rest of the compiler, which can cut down on the size of AST nodes. You do however need to then keep a list of the line-start positions for each file for later on.
1
1
Show replies