Module leo_parser::tokenizer
source ยท Expand description
The tokenizer to convert Leo code text into tokens.
This module contains the tokenize()
function, which breaks down string text into tokens,
optionally separated by whitespace.
Re-exportsยง
pub use self::token::KEYWORD_TOKENS;
Modulesยง
Functionsยง
- tokenize ๐Creates a new vector of spanned tokens from a given file path and source code text.
- tokenize_iter ๐Yields spanned tokens from the given source code text.