Removable unsafe entries

[stevemk14ebr]

The options for unsafe does not allow the removal of unsafe entries. I'd like to customize this array but I cannot since .concat is used so passing an empty array does not remove any rules:

mdast-util-to-markdown/lib/configure.js

Line 18 in f3df741

base[key] = base[key].concat(extension[key] || [])

I want to remove these rules in particular:

mdast-util-to-markdown/lib/unsafe.js

Lines 92 to 94 in f3df741

	{atBreak: true, character: '_'},
	{before: '[^A-Za-z]', character: '_', inConstruct: 'phrasing'},
	{character: '_', after: '[^A-Za-z]', inConstruct: 'phrasing'},

[wooorm]

Why do you want to remove those three?

[stevemk14ebr]

Jira nonsense again:

function horizontalRule(node) {
    return { type: 'text', value: '----' }
}

emits:

\----

I'd have to duplicate the whole stringify crate if this isn't customizable :/

[wooorm]

Why use a text node and not return a string?

mdast-util-to-markdown/lib/handle/thematic-break.js

Line 7 in f3df741

function thematicBreak(node, parent, context) {

[stevemk14ebr]

thanks for the pointer. I was doing these changes in a custom remarkPlugin before which was before the stringify stage so things were escaped. I'm now also adding custom handlers in the stringify to handle the horizontalRule type where i return string exactly.

stringify handler:

[RemarkStringify, {
            handlers: {
                horizontalRule: (node, parent, context) => {
                    return node.value;
                },
            }
        }]

remark-jira

function horizontalRule(node) {
    return { type: 'horizontalRule', value: '----' }
}

[wooorm]

I think your particular use case is better solved by remarkjs/remark#720 (comment) (the other issue).
I might be open to revisit this in the future, but there are quite good reasons for these escape patterns to exist: to make sure markdown can roundtrip

[stevemk14ebr]

agreed

[about-code]

I sometimes face a similar situation like @stevemk14ebr. That's my use case:

In short glossarify-md is a tool which you feed Markdown in to get some enhanced Markdown out. As such it might be embedded into some Markdown toolchain. The tool itself is fine with CommonMark and GFM syntax to do its magic. Yet I would like the tool to be as tolerant as possible towards all the other Markdown syntax extensions that exist out there and which users might use in their input files.

Currently, there are two things which can happen when parsing non-standard Markdown:

1. some syntactic elements get parsed incorrectly into a CommonMark node type due to an overlap/collision of the syntax extension with CommonMark (see Frontmatters with trailing --- dashes, for example, which overlap with "Headings")
2. some elements get parsed into an mdAst text node due to not being recognized as syntax at all (see comment above)

In terms of 1 there's hardly anything - even the best parser can do apart from offering a great plug-in API. However, often the second happens. The mere fact that a text node is being created - rather than throwing an error - is quite nice. For a markdown-to-markdown tool only the escaping when stringifying the text node can be an unfortunate mutation of the user's inputs and break the user's MD toolchain.

Said this, I consider it absolutely the right choice for mdast-util-to-string to operate on the premise that mdast-util-to-markdown has created a semantically sound AST where a text node means regular text. Therefore by default it should definitely escape text content and maintain semantic interoperability, that is, make sure any subsequent parser of the stringified output reads it as regular text again. The issue in the face of an MD pipeline and syntax extensions is that mdast-util-to-markdown's interpretation as regular text isn't semantically correct in that context even though it is the only possible for it.

As a consequence the premise of mdast-util-to-string doesn't hold true anymore as well. A text node might not hold a value which represents regular text but unknown markdown syntax which should not be escaped. There could be a another parser in the pipeline capable of understanding the syntax.

The reason why I do not push for reopening this currently is, because there are no user-reported issues for glossarify-md ATM and it is possible for them to enjoy the remark-plugin ecosystem. Still, the space of MD syntax extensions is so broad that I fear even the remark ecosystem doesn't provide a plug-in for each. That's why I would be glad if I could point users to a remark setting by which they could disable escaping. Maybe I would even make glossarify-md to disable it by default.

[ChristianMurphy]

The issue in the face of an MD pipeline and syntax extensions is that mdast-util-to-markdown's interpretation as regular text isn't semantically correct in that context even though it is the only possible for it.

How so?
Syntax extensions will have their own node types and stringifiers, and don't influence the interpretation of text.

[about-code]

Syntax extensions will have their own node types and stringifiers, and don't influence the interpretation of text

That's only the case if there's a corresponding remark plug-in with support for the syntax. The plug-in would create the node type and provide the stringifiers, wouldn't it? In case of glossarify-md it means (in some cases) that users have to install and configure a plug-in just to get out unmodified what they've passed in. Sometimes this weren't necessary if escaping could just be disabled and unknown syntax would be simply considered text (as it is, obviously if there's no plug-in which creates the node types) but then written as text again - yet unescaped text, though.

[about-code]

Here's an example of what I had to do specifically to support vuepress' [[toc]] instruction in glossarify-md to workaround escaping: https://github.com/about-code/glossarify-md/blob/c99d5c67fc022311109d84b6857f498d30982cd6/lib/ast/with/toc-instruction.js

This is actually similar to writing a particular plug-in but also a case where I couldn't find one that fits exactly, so had to write it (partly) myself. This isn't an approach that scales to the multitude of markdown flavours out there, I am afraid.

[ChristianMurphy]

That's only the case if there's a corresponding remark plug-in with support for the syntax. The plug-in would create the node type and provide the stringifiers, wouldn't it?

Yes, syntax extensions generally need both a parser and stringifier.

In case of glossarify-md it means (in some cases) that users have to install and configure a plug-in just to get out unmodified what they've passed in

Right, remark is an AST, not a CST.
Structure (as parsed) is preserved, exact formatting is not (more context in #3)

[about-code]

Thanks for the quick reponse and pointer on #3. Even though it is not a CST (wasn't aware there's a difference like AST/CST) is there anything that prevents making escaping an opt-out feature?

[ChristianMurphy]

Disabling escapes may (or may not) preserve custom markdown syntax, but it will break valid commonmark/gfm markdown in places where escapes should happen.
It trades one problem for another, without solving the core issue you seem to be getting at, which is preserving exact format.