The last time I tried that, having a separate tokenizer and parser but the former having a "newline" token for the last bit above, the latter ended up very hacky.
Conversation
What I do in Kitten is have a lexing pass that generates tokens annotated with source spans, followed by a bracketing pass that desugars them into explicitly delimited blocks using those spans—the concept of an “indent/dedent token” seems obvious but doesn’t work well in practice
1
5
(Because you’re attempting to deduce the bracketing as if it were already present, in addition to the basic mistakes of mixing concerns)
1
2
Hmm. I mentioned the other things just to avoid being overly specific, but my overly specific problem was with the "newlines terminate statements (but are also allowed elsewhere)" thing -- does your advice translate to that case?
1
Yeah—the specifics depend on what you want to do in particular, but for example I allow newlines within parentheses/square braces, or outside a layout context, and they don’t cause statement terminators to be inserted there
1
2
Here’s the latest code for my implementation of this technique, if you’re curious: github.com/evincarofautum
1
1
Yeah, I believe that this is what Gluon and Haskell do too.
1
I've been doing some work in fsharp recently and .. whoo boy I did not expect to have so many feelings against whitespace sensitivity boil up all of a sudden. But for real: it does not go well in expression languages where you're doing a lot of nesting & continuation lines.
1
6
I’m almost considering not doing whitespace sensitivity in Pikelet. 🤔
3
1
It's weird because it did not bother me in haskell (nor python) but it's like fingernails on blackboard in fsharp. Might just be the years I've spent doing ocaml which is not. But also I think there's much more rightward drift with pervasive keyword args and module decl nesting.
3
2
Weirdly the braces in Rust bother me a lot more because it is so rightward drifty. 🤔
It wasn't supposed to be! My early syntax guideline was "there's always more vertical space in the buffer", which I still mostly believe. But the FP expression language squad showed up and I lost. IMO the only half-decent way to do an expression language is sexps.
3
6
I dunno, I'm so conflicted because I also love the expression syntaxness of it. 😂 Super interesting to know that history though.
1