Enum leo_parser::tokenizer::token::Token

pub enum Token {
    CommentLine(String),
    CommentBlock(String),
    WhiteSpace,
    True,
    False,
    Integer(String),
    AddressLit(String),
    StaticString(String),
    Identifier(Symbol),
    Not,
    And,
    AndAssign,
    Or,
    OrAssign,
    BitAnd,
    BitAndAssign,
    BitOr,
    BitOrAssign,
    BitXor,
    BitXorAssign,
    Eq,
    NotEq,
    Lt,
    LtEq,
    Gt,
    GtEq,
    Add,
    AddAssign,
    Sub,
    SubAssign,
    Mul,
    MulAssign,
    Div,
    DivAssign,
    Pow,
    PowAssign,
    Rem,
    RemAssign,
    Shl,
    ShlAssign,
    Shr,
    ShrAssign,
    Assign,
    LeftParen,
    RightParen,
    LeftSquare,
    RightSquare,
    LeftCurly,
    RightCurly,
    Comma,
    Dot,
    DotDot,
    Semicolon,
    Colon,
    DoubleColon,
    Question,
    Arrow,
    BigArrow,
    Underscore,
    At,
    Address,
    Bool,
    Field,
    Group,
    I8,
    I16,
    I32,
    I64,
    I128,
    Record,
    Scalar,
    Signature,
    String,
    Struct,
    U8,
    U16,
    U32,
    U64,
    U128,
    Aleo,
    As,
    Assert,
    AssertEq,
    AssertNeq,
    Async,
    Block,
    Console,
    Const,
    Constant,
    Else,
    Fn,
    For,
    Function,
    Future,
    If,
    Import,
    In,
    Inline,
    Let,
    Mapping,
    Network,
    Private,
    Program,
    Public,
    Return,
    SelfLower,
    Transition,
    Eof,
    Leo,
}

Represents all valid Leo syntax tokens.

The notion of ‘token’ here is a bit more general than in the ABNF grammar: since it includes comments and whitespace, it corresponds to the notion of ‘lexeme’ in the ABNF grammar. There are also a few other differences, noted in comments below.

Variants

CommentLine(String)

CommentBlock(String)

WhiteSpace

True

False

Integer(String)

AddressLit(String)

StaticString(String)

Identifier(Symbol)

Not

And

AndAssign

Or

OrAssign

BitAnd

BitAndAssign

BitOr

BitOrAssign

BitXor

BitXorAssign

Eq

NotEq

Lt

LtEq

Gt

GtEq

Add

AddAssign

Sub

SubAssign

Mul

MulAssign

Div

DivAssign

Pow

PowAssign

Rem

RemAssign

Shl

ShlAssign

Shr

ShrAssign

Assign

LeftParen

RightParen

LeftSquare

RightSquare

LeftCurly

RightCurly

Comma

Dot

DotDot

Semicolon

Colon

DoubleColon

Question

Arrow

BigArrow

Underscore

At

Address

Bool

Field

Group

I8

I16

I32

I64

I128

Record

Scalar

Signature

String

Struct

U8

U16

U32

U64

U128

Aleo

As

Assert

AssertEq

AssertNeq

Async

Block

Console

Const

Constant

Else

Fn

For

Function

Future

If

Import

In

Inline

Let

Mapping

Network

Private

Program

Public

Return

SelfLower

Transition

Eof

Leo

Implementations

impl Token

pub fn is_keyword(&self) -> bool

Returns true if the self token equals a Leo keyword.

pub fn keyword_to_symbol(&self) -> Option<Symbol>

Converts self to the corresponding Symbol if it is_keyword.

impl Token

fn eat_integer( input: &mut Peekable<impl Iterator<Item = char>>, ) -> Result<(usize, Token)>

Returns a tuple: [(integer length, integer token)] if an integer can be eaten. An integer can be eaten if its characters are at the front of the given input string. If there is no input, this function returns an error. If there is input but no integer, this function returns the tuple consisting of length 0 and a dummy integer token that contains an empty string. However, this function is always called when the next character is a digit. This function eats a sequence of one or more digits and underscores (starting from a digit, as explained above, given when it is called), which corresponds to a numeral in the ABNF grammar.

pub(crate) fn eat(input: &str) -> Result<(usize, Token)>

Returns a tuple: [(token length, token)] if the next token can be eaten, otherwise returns an error. The next token can be eaten if the characters at the front of the given input string can be scanned into a token.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for Token

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Display for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for Token

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Eq for Token

impl StructuralPartialEq for Token