Skip to content

Overhaul Tokenization

recite1670 requested to merge api/tokenizer-chars into master

Overhauls the tokenization system to use IEnumerable<char> as input and IEnumerable<Token> as output. This reduces the type requirements (previously string input) and gives greater control over the tokens (previously the entire input was interpolated, now the marked tokens are).

These changes also include ClosureTokenzier<TContext> and with it closure interfaces (and default implementations). These closure interfaces made it easy to implement various delimiters and wrappers (now the standard Bash set of \ , "", and '' are valid), as well as also implement #4 (closed) as a part of the overhaul.

Merge request reports

Loading