Provides a sequence of chars

Every token detector will return a MatchResult if they find a match.

A standard token detector base class. Standard detectors have a DetectorType

String token that a tokenizer could detect it

Provides more function on the tokens

String tokenizer builder which builds a tokenizer with token detectors

Interface to string source reader

Interface to string source scanner

Interface to string tokenizer

Every token detector must implement this interface

Context of finding a token