Deciding the future of `commonmark.py`

[astrojuanlu]

Hi everyone, from Read the Docs we would like to open up the question about the future of commonmark.py. The landscape of Markdown in Python and Sphinx has changed a lot over the past 12 months, and in particular markdown-it-py by @chrisjsewell et al achieves a level of extensibility that is beyond what's possible with commonmark.py (see #106 (comment)). Similarly, MyST-Parser provides a broader feature set than what recommonmark offers.

As a result of the latter, and with the hope of optimizing efforts, a couple of months ago we decided to deprecate recommonmark in favor of MyST-Parser. Since commonmark.py has a similar status, we would also like to deprecate it - but given that the project has a richer history and it's also a lower level package, we would like to first ask the current maintainers their opinion.

@nikolas and @lu-zero are the ones that have been merging code to commonmark.py in recent years. There are a bunch of open bugs and pull requests that haven't received much attention, and the last release is from October 2019 (_{ah, the good old times without global pandemics}). Nobody from Read the Docs plans to step in and address them, so we would like to offer two possibilities:

• We find a new GitHub organization for the project, where the same or new maintainers can give it some love, or
• We add a similar deprecation note than the one we placed on recommonmark README, and archive the repository here.

If there is no response or consensus in a reasonable time frame, the latter option wins, since anyone can fork the repository anyway. But we would like to give others the chance to speak up so the complete repository, with all the issues and pull request history, can be cleanly transferred.

Thoughts?

cc @ericholscher

[nikolas]

I'm all for transferring commonmark.py to Jazzband so it can receive better regular maintenance and attention!

[nikolas]

@astrojuanlu I've submitted a proposal to move this package to jazzband here: jazzband/help#221

[astrojuanlu]

Thanks a lot @nikolas ! I subscribed there 👍🏽

[lu-zero]

I'm fine with moving it or favor markdown-it-py.

[ericholscher]

Yea, Jazzband seems like a good home if we want to keep it maintained in some fashion 👍

[tejasvi]

I would add that no other commonmark compliant parser in python provides source level map for the ast which is required for efficient documentation manipulation.

[astrojuanlu]

I would add that no other commonmark compliant parser in python provides source level map for the ast which is required for efficient documentation manipulation.

Summonning @chrisjsewell on this

[chrisjsewell]

no other commonmark compliant parser in python provides source level map ... Summonning @chrisjsewell on this

yep markdown-it/markdown-it-py does

If you want to use commonmark.js over markdown-it that obviously that's absolutely fine.
But I looked through all the markdown parsers for myst-parser, and well there is a reason markdown-it has 10 times the GitHub stars, and is used by VS Code etc; it's just a lot better, in terms of parsing logic, performance, extensibility, source mapping, etc

[tejasvi]

@chrisjsewell Does it provide map from tokens to locations in source? AFAIK original JS markdown-it does not provide them markdown-it/markdown-it#336 In JS world, only commonmark.js (block granularity) and remark (inline granularity) provide such info.

Edit: Oops, my memory failed me. Markdown-it does provide line position but not character position. Somehow I couldn't find them the list of token generated. In light, of that my original comment is moot.

Edit2: Thanks @chrisjswell. For me markdown-it-py is the way forward and you saved the day when I was finishing up ast generation implementation via NodeJS over rpc. (after brief adventure with JS2Py which ended up being too slow)

[chrisjsewell]

Does it provide map from tokens to locations in source?

Yes it does, in the map attribute; it provides block level mapping to source lines: https://markdown-it.github.io/
But then you have already discussed this in markdown-it/markdown-it#336

No doubt, inline level (column) mapping would be great, but personally I think it would be a lot better to create an adapted version of markdown-it rather than commonmark.js, if you ever want to consider extending the syntax past commonmark in any way.

[chrisjsewell]

https://ui.toast.com/weekly-pick/en_20200402 looks interesting though, will certainly keep an eye on it 😄