is because tokenization depends on context. Even in these cases, syntactical analysis is often seen as approximating this ideal model.

The levels generally correspond to levels in the Chomsky hierarchy. Words are in a regular language, specified in the lexical grammar, which is a Type-3 grammar, generally given as regular expressions. Phrases are in a context-free language