Function leo_parser::tokenizer::tokenize_iter

source ยท 
pub(crate) fn tokenize_iter(
    input: &str,
    lo: BytePos,
) -> impl '_ + Iterator<Item = Result<SpannedToken>>
Expand description

Yields spanned tokens from the given source code text.

The lo byte position determines where spans will start.