Module tokenizer

The tokenizer to convert Leo code text into tokens.

This module contains the tokenize() function, which breaks down string text into tokens, optionally separated by whitespace.

Re-exports

pub use self::token::KEYWORD_TOKENS;

Modules

lexer 🔒

token 🔒

Functions

tokenize 🔒

Creates a new vector of spanned tokens from a given file path and source code text.

tokenize_iter 🔒

Yields spanned tokens from the given source code text.