Conversation

This Tweet was deleted by the Tweet author. Learn more

Apr 12, 2019

I've wondered about using a parser on a stream of characters whose result is a stream of tokens to implement lexing, then running that stream of tokens through the regular parser. Maybe cause I'm too lazy to write my own regex stuff for lexing.

Replying to

and

Yeah, this is just how I see it. A tokeniser is a parser from chars to tokens. And you can implement a parser from tokens to something else from that. Or maybe everything is just little compiler phases. I dunno!

11:53 AM · Apr 12, 2019Twitter Web Client

This Tweet is from a suspended account. Learn more

Brendan Zabarauskas

@brendanzab

Apr 12, 2019

bahaha I am now doing type-directed literal parsing in my elaborator 🤪

This Tweet is from a suspended account. Learn more

Paul Bone

@Paul_Bone

Apr 12, 2019

They're the same. I just feel like we didn't need to create the word "lexing" when tokenisation is already a word. And yeah, I know what a lexicon is.

This Tweet is from a suspended account. Learn more

Brendan Zabarauskas

@brendanzab

Apr 12, 2019

My tokens have a location associated with them: github.com/brendanzab/rus