Tangentially related, but I wonder what's the right way to do things when the language has some amount of whitespace sensitivity.
Like indentation, or spaces required around binops, or newlines terminate statements (but are also allowed in the middle of expressions)...
Conversation
The last time I tried that, having a separate tokenizer and parser but the former having a "newline" token for the last bit above, the latter ended up very hacky.
1
3
What I do in Kitten is have a lexing pass that generates tokens annotated with source spans, followed by a bracketing pass that desugars them into explicitly delimited blocks using those spans—the concept of an “indent/dedent token” seems obvious but doesn’t work well in practice
1
5
(Because you’re attempting to deduce the bracketing as if it were already present, in addition to the basic mistakes of mixing concerns)
1
2
Hmm. I mentioned the other things just to avoid being overly specific, but my overly specific problem was with the "newlines terminate statements (but are also allowed elsewhere)" thing -- does your advice translate to that case?
1
Yeah—the specifics depend on what you want to do in particular, but for example I allow newlines within parentheses/square braces, or outside a layout context, and they don’t cause statement terminators to be inserted there
1
2
Here’s the latest code for my implementation of this technique, if you’re curious: github.com/evincarofautum
1
1
Yeah, I believe that this is what Gluon and Haskell do too.
1
I've been doing some work in fsharp recently and .. whoo boy I did not expect to have so many feelings against whitespace sensitivity boil up all of a sudden. But for real: it does not go well in expression languages where you're doing a lot of nesting & continuation lines.
1
6
I’m almost considering not doing whitespace sensitivity in Pikelet. 🤔
3
1
I do find it quite pretty in Haskell though.