leo_parser::tokenizer

Module lexer

Source

Structs§

Functions§

  • eat_identifier 🔒
    Eat an identifier, that is, a string matching ‘[a-zA-Z][a-zA-Z\d_]*’, if any.
  • ensure_no_bidi_override 🔒
    Ensure that string contains no Unicode Bidirectional Override code points.
  • is_bidi_override 🔒
    Checks if a char is a Unicode Bidirectional Override code point