Brendan Zabarauskas on Twitter: "@AnnieTheObscure @Paul_Bone @afsofia_ My tokens have a location associated with them: https://t.co/OpQ8QRaAi8" / Twitter

This Tweet was deleted by the Tweet author. Learn more

Apr 12, 2019

I've wondered about using a parser on a stream of characters whose result is a stream of tokens to implement lexing, then running that stream of tokens through the regular parser. Maybe cause I'm too lazy to write my own regex stuff for lexing.

4

1

Brendan Zabarauskas

@brendanzab

Apr 12, 2019

Yeah, this is just how I see it. A tokeniser is a parser from chars to tokens. And you can implement a parser from tokens to something else from that. Or maybe everything is just little compiler phases. I dunno!

3

1

This Tweet is from a suspended account. Learn more

Replying to

and

My tokens have a location associated with them: github.com/brendanzab/rus

1:13 PM · Apr 12, 2019Twitter Web Client

1

Like

Conversation