Fields

buffer: [:0]const u8,
index: usize,
pending_invalid_token: ?Token,

Functions

fn dump(self: *Tokenizer, token: *const Token) void

For debugging purposes

fn findTagAtCurrentIndex(self: *Tokenizer, tag: Token.Tag) Token

This is a workaround to the fact that the tokenizer can queue up ’pending_inval…

This is a workaround to the fact that the tokenizer can queue up ‘pending_invalid_token’s when parsing literals, which means that we need to scan from the start of the current line to find a matching tag - just in case it was an invalid character generated during literal tokenization. Ideally this processing of this would be pushed to the AST parser or another later stage, both to give more useful error messages with that extra context and in order to be able to remove this workaround.

fn init(buffer: [:0]const u8) Tokenizer

No documentation provided.

fn next(self: *Tokenizer) Token

No documentation provided.