fn TokenIterator(comptime T: type, comptime delimiter_type: DelimiterType) type

Parameters

T: type,
delimiter_type: DelimiterType,

Fields

buffer: []const T,
delimiter: switch (delimiter_type) {
            .sequence, .any => []const T,
            .scalar => T,
        },
index: usize,

Functions

fn next(self: *Self) ?[]const T

Returns a slice of the current token, or null if tokenization is complete, and …

Returns a slice of the current token, or null if tokenization is complete, and advances to the next token.

fn peek(self: *Self) ?[]const T

Returns a slice of the current token, or null if tokenization is complete. Does…

Returns a slice of the current token, or null if tokenization is complete. Does not advance to the next token.

fn reset(self: *Self) void

Resets the iterator to the initial token.

fn rest(self: Self) []const T

Returns a slice of the remaining bytes. Does not affect iterator state.