Function tokenize

Source
pub(crate) fn tokenize(input: &str, start_pos: u32) -> Result<Vec<SpannedToken>>
Expand description

Creates a new vector of spanned tokens from a given file path and source code text.