Markdown Conversion

[fokoenecke]

Hi! Thanks for this great library. After looking at some of the XML docs, i really see the pain in creating it ;)

I am developing a webapplication taking user input in a form and generating a docx file from it. Amongst others some fields are formatted in markdown. I am planning to take the markdown fields, convert them to XML (with pandoc or python markdown) and put it into the document via your low level API.

Is there a better/easier way to do this or any plans for implementing markdown directly into python-docx?

greatings

[scanny]

I think it's an interesting idea. I believe that ReportLab has something analogous for adding content to a PDF document. There they use HTML as I recall.

It would be important for it to be fairly well separated from the main API, and I'm thinking it would be good for it to have something like a plug-in architecture since there's quite a variety of formats that folks might want. reStructuredText and HTML both come to mind in addition to Markdown.

I'm inclined to think something like this could be done with a function call as a start, something like:

append_markdown_content(document, markdown_text)

that would interpret markdown_text into Word objects and add them to document in the order they were encountered.

I don't think this will be a feature that will come soon, given we still have a good deal of core library work to do. So a separate module that implemented append_markdown_content() might be a useful adjunct for a year or two and perhaps beyond.

[fokoenecke]

Today i took a look at python-markdown. They parse the markdown code with regular expressions and convert it into HTML by using the python xml library. They also provide an Extension API for writing own processors, handling the markdown elements.

I think I'll give it a try and write an extension that generates OOXML instead of HTML based on your low level API.

[scanny]

That would be great! Even an early prototype I'm sure would flush out a lot of learning about what would be possible and how much work it might take. I'll look forward to hearing what you come up with :)

[waylan]

@fokoenecke I just stumbled across this and am the lead dev for Python-Markdown. If there is anything I can do to help, let me know. BTW, while it is not documented, you can swap out Python-Markdown's serializer for your own. So a library that converted ElementTree HTML objects to docx.Document objects would be were I would start. You could plug that into Python-Markdown as a serializer... or after parsing any HTML with lxml or [html5lib](both output ElementTree Objects), use it to convert any HTML document (not just markdown) to docx.

In fact, the latter is probably preferable. For example, Markdown permits raw HTML. The Python-Markdown parser runs a pre-processor that just stores the raw HTML and inserts placeholders in the document that pass through the parser. A post-processor (run after the serializer) then swaps out the placeholders and reinserts the raw HTML. For complete markdown support, you would need to run through the entire markdown parser and then pass the HTML string into an HTML parser before converting to docx. Of course, that requires getting an HTML parser working first. Swapping out Python-Markdown's serializer might be an easier place to start - even if it doesn't give you the ability to support raw HTML.

[scanny]

Hmm, I think @waylan's idea for making the 'translator' based on HTML is an idea worth exploring seriously. There are a lot of simplified markup languages out there, but they all have HTML in common. Plus it can be a handy way to build up some rich text in any program, even if you're doing it "manually". If there were an HTML translator then any way we ended up with it would work. Great call @waylan :)

[fokoenecke]

@waylan thanks for your reply! My idea was to build my own processors for python-markdown that generate a ElementTree of docx objects directly. But I see that it would be far more flexible parsing HTML and converting those objects to docx.

In fact, I tried that before on a higher level by rendering a HTML page with mako and converting it via pypandoc to docx. I have not been satisfied by the results and the lack of control this solution provides.

I guess I have to take a deeper look at python-docx to find the best place to insert the generated doxc. This dictates if i can use text generated by a serializer or if I have to get the results as a branch of an ElementTree.

As soon as I make a plan I will inform both of you and ask for advice ;)

[fokoenecke]

I decided to go with waylans plan and hand a html string to pydocx. That string will be parsed to an ElementTree by lxml. Then I am going to translate the different tags to pydocx functions.

A very early prototype for this already worked, at least for my requirements.

[scanny]

Glad to hear it worked out @fokoenecke :)

I'll close this for now then since there's no immediate prospect of development on it. Happy if someone wants to reopen it in future.

[fokoenecke]

oh, I hope you didn't get me wrong on that. I am far from done now and I'd love to contribute what i did afterwards. I just don't know, if anyone can use it, because I won't do a full-flectched HTML conversion, but only the stuff needed to get from markdown to docx. I will tell you, when I am satisfied and the code is tested, so you can decide, if it makes sense.

I opened a branch for it here: https://github.com/fokoenecke/python-docx/tree/html

[scanny]

Ah, ok, no problem, opening it back up :)

[fokoenecke]

Just a little update on what i did so far:
I created an API function that takes an HTML string and a docx container to add it to. The function hands the HTML to a builder class, that recusively iterates through the HTML elements and tries to find a fitting OOXML object to append to the given container. This works by mapping the HTML tags to various dispatcher objects.

By now my code can barely handle the HTML I get from the markdown converter. It's far from covering full-stack HTML. But it does all the things I need it to. I can append small markdown snippets to the document and paragraphs pretty well.

If no one else is interested in this, it's not likely that I will change much more on it. I will fix problems that arise from unlikely markdown combinations and add missing features as they appear in python-docx.