Discussion
Continuing my exploration of reimplementing GLSL for a risc32cfimv, the next step is to implement a lexer.
For starters: What would I implement it in?
If I'm saying that it'd be stored in a separate page of RAM without access to the Lua interpreter, that would suggest that we need this compiled to RISC-V Assembly. We could hand write this Assembly, or I've designed a stack-based mid-level language.
Or I suppose we could save the results from compiling some form of type-checked Lua!
1/3?
Lets say we use type-checked Lua (possibly in a special mode), since that aids new parsing techniques. And allows us to reuse the scanner I designed for parsing Lua, inlining those routines. Wrapped in a routine to track line number & filename, which can be set via the #line directive.
Getting back to lexing... GLSL has similar lexing rules to C, except we don't have strings. Too high-level for GLSL!
That is we split the file into numbers, identifiers & keywords, symbols, & punctuation.
2/3!
We ignore whitespace, single-line comments (preceded by "//"), multi-line comments (surrounded by slashes & asterisks).
After iterating over the sign, base, digits, decimal point, & exponent we could embed tonumber which I've already incorporated into the Lua parser.
As for identifiers, those are sequences of letters & underscores. Or, apart from the 1st char, digits. Consult a hashmap if we're seeing a keyword instead.
Output as an array of tokens with value, filename, & linenumber.
3/3!
A space for Bonfire maintainers and contributors to communicate
