Module tokenizer

The tokenizer to convert Leo code text into tokens.

This module contains the tokenize() function, which breaks down string text into tokens, optionally separated by whitespace.

Re-exports

• pub use self::token::KEYWORD_TOKENS;

Modules

• lexer 🔒
• token 🔒

Functions

• tokenize 🔒
  Creates a new vector of spanned tokens from a given file path and source code text.
• tokenize_iter 🔒
  Yields spanned tokens from the given source code text.