re2c

re2c is a software tool that will allow you to write very fast and very flexible scanners. Unlike any other such tool, re2c focuses on generating high efficient code for regular expression matching.

re2c is a preprocessor that generates C-based recognizers from regular expressions. The input to re2c consists of C/C++ source interleaved with comments of the form /*!re2c ... */ which contain scanner specifications.

In the output these comments are replaced with code that, when executed, will find the next input token and then execute some user-supplied token-specific code.

As a result this allows a much broader range of use than any traditional lexer offers. And Last but not least re2c generates warning free code that is equal to hand-written code in terms of size, speed and quality.

re2c does not provide a default action: the generated code assumes that the input will consist of a sequence of tokens. Typically this can be dealt with by adding a rule such as the one for unexpected characters in the example above.

The user must arrange for a sentinel token to appear at the end of input (and provide a rule for matching it): re2c does not provide an expression. If the source is from a null-byte terminated string, a rule matching a null character will suffice.

If the source is from a file then you could pad the input with a newline (or some other character that cannot appear within another token); upon recognizing such a character check to see if it is the sentinel and act accordingly.

And you can also use YYFILL(n) to end the scanner in case not enough characters are available which is nothing else then e detection of end of data/file.

re2c does not provide start conditions: use a separate scanner specification for each start condition (as illustrated in the above example).