Add initial tokenizer for Python

[certik]

One can use it using lpython --show-tokens somefile. Currently the tokenizer is a Fortran tokenizer, so we need to adapt it to become a Python tokenizer.

Towards #298.

[Thirumalai-Shaktivel]

LGTM! Let's merge this once we fix the CI

[Thirumalai-Shaktivel]

Hi @certik, I updated the tokenizer to recognise Comments, DocString, Symbols and also made the CI pass.
Can you please review this??

[Thirumalai-Shaktivel]

I tested this with many Python files. I got some errors, which I fixed and will push here.
For the rest of the files, --show-tokens provides output without any errors!
Now I think we can move to the parser (Bison).

[Thirumalai-Shaktivel]

There is a conflict between String and Docstring in the tokenizer, we need to fix that.

[certik]

Can you get the CI passing again please?

[certik]

I left a fix above, after the update I think this is good to go in.

After it is merged, we should do three more steps before moving to the parser:

• Emit indent and dedent tokens properly (i.e., handle indentation in the tokenizer)
• Access the tokens from CPython somehow (figure out how to do that), and then compare against it, that we are getting exactly those tokens and nothing more or less.
• Remove all the Fortran stuff from the tokenizer

It doesn't have to be perfect, it won't be perfect, but I would definitely do the above three things. Then we can move to the parser and start parsing it. And then we'll iterate on the design of the tokenizer as needed.

I posted this comment at #298 (comment).

[Thirumalai-Shaktivel]

Perfect, I will work on indent and dedent and send a PR.
Thanks for the review!

[certik]

Thanks! I think this looks good, now we just have to iteratively improve it, the TODO list is written up at: #298 (comment)