leo_parser::tokenizer

Module token

Source

Enums§

  • Delimiter
    Describes delimiters of a token sequence.
  • Token
    Represents all valid Leo syntax tokens.

Constants§

  • KEYWORD_TOKENS
    Represents all valid Leo keyword tokens. This also includes the boolean literals true and false, unlike the ABNF grammar, which classifies them as literals and not keywords. But for the purposes of our lexer implementation, it is fine to include the boolean literals in this list.