Overhaul Tokenization
Overhauls the tokenization system to use IEnumerable<char>
as input and IEnumerable<Token>
as output. This reduces the type requirements (previously string
input) and gives greater control over the tokens (previously the entire input was interpolated, now the marked tokens are).
These changes also include ClosureTokenzier<TContext>
and with it closure interfaces (and default implementations). These closure interfaces made it easy to implement various delimiters and wrappers (now the standard Bash set of \
, ""
, and ''
are valid), as well as also implement #4 (closed) as a part of the overhaul.