I made a basic nodejs sample:
fun main() {
val input = "fn foo = (n) -> { n }"
val tokens = DefaultTokenizer(Tokens).tokenize(input).toList()
println(tokens)
}
This results in the following tokens:
Development
[fn@1 for "fn" at 0 (1:1), whitespace@2 for " " at 2 (1:3), identifier@3 for "foo" at 3 (1:4), whitespace@4 for " " at 6 (1:7), equal@5 for "=" at 7 (1:8), whitespace@6 for " " at 8 (1:9), leftParenthesis@7 for "(" at 9 (1:10), identifier@8 for "n" at 10 (1:11), rightParenthesis@9 for ")" at 11 (1:12), whitespace@10 for " " at 12 (1:13), rightArrow@11 for "->" at 13 (1:14), whitespace@12 for " " at 15 (1:16), leftBrace@13 for "{" at 16 (1:17), whitespace@14 for " " at 17 (1:18), identifier@15 for "n" at 18 (1:19), whitespace@16 for " " at 19 (1:20), rightBrace@17 for "}" at 20 (1:21)]
Production
[whitespace@1 for "f" at 0 (1:1), whitespace@2 for "n" at 1 (1:2), whitespace@3 for " " at 2 (1:3), whitespace@4 for "f" at 3 (1:4), whitespace@5 for "o" at 4 (1:5), whitespace@6 for "o" at 5 (1:6), whitespace@7 for " " at 6 (1:7), whitespace@8 for "=" at 7 (1:8), whitespace@9 for " " at 8 (1:9), whitespace@10 for "(" at 9 (1:10), whitespace@11 for "n" at 10 (1:11), whitespace@12 for ")" at 11 (1:12), whitespace@13 for " " at 12 (1:13), whitespace@14 for "-" at 13 (1:14), whitespace@15 for ">" at 14 (1:15), whitespace@16 for " " at 15 (1:16), whitespace@17 for "{" at 16 (1:17), whites