Provide option to incorporate directory of Markdown files into generated HTML documentation

[mattt]

As suggested by @MaxDesiatov in #57:

It would be great if there was an option not just to include README.md, but a given directory hierarchy of multiple markdown files that provide extended documentation: tutorials, guidelines, license, code of conduct etc.

Scoping the feature request a bit, I think a good first step would be to add a new, optional parameter to the generate command to specify a directory of Markdown files (Documentation by default) to be rendered alongside the generated symbol reference docs.

For example, given the following project structure:

.
├── Documentation
│   ├── Advanced-Usage.md
│   ├── FAQ.md
│   └── Getting-Started.md
└── Sources

swift doc generate --format html would produce three "Guide" pages for "Advanced Usage", "FAQ", and "Getting Started".

My only hesitation here is that it gets us into the business of static site generation more generally (a corollary of Zawinski's Law. To that end, I'm unsure on whether to should support any form of YAML front matter...

Edit: Actually, you know what? I'm putting my foot down: No YAML front matter! I know of a better way that doesn't add another dependency or threaten to go down a slippery slope of supporting TOML / JSON / whatever front matter — I randomly came across it in this Hugo issue thread:

@bep: This is just me thinking out loud during Christmas. I kind of like the simplicity of it. The idea is to use SGML Processing Instructions as a way to pass parameters to the new and shiny render hooks in Hugo (images and links).

So instead of this:

---
title: Getting Started
---

Reprehenderit culpa qui enim elit sunt laborum do occaecat nostrud exercitation enim ex...

You do something like this (final spelling TBD):

Reprehenderit culpa qui enim elit sunt laborum do occaecat nostrud exercitation enim ex...

<?swiftdoc title="Getting Started" ?>

Processing Instructions aren't rendered by default, and would be otherwise invisible. And unlike front matter, it can appear anywhere in the document.

If we scope this feature to Markdown files with optional PI for customization, I think that strikes a good balance for a "batteries included" output, with the option to further customize using a dedicated static site generator based on the CommonMark output.

[MaxDesiatov]

Another option could be to explicitly state that swift-doc focuses only on documentation generation and static site generators should handle the rest if needed, which could help in avoiding feature creep. If that's the case, it would be great to have some integration between swift-doc and say Publish and maybe an ability to reuse themes across both.

[mattt]

@MaxDesiatov Please see my edit, which came in after your comment.

I fully acknowledge the role for dedicated static site generators, and am committed to tailoring the CommonMark / SQLite output to support that kind of extensibility. That said, I think we can be smart about offering 80% of SSG functionality without blowing up the scope of our project.

[kean]

I agree with @MaxDesiatov, hosting guides alongside the API reference and using the same design language - which I really love, and which I would like to be standard for all frameworks - would be a major improvement in user experience.

I understand this probably blows up the scope significantly. But what I'm personally looking for is a comprehensive solution offering an experience similar or better to what we have at developer.apple.com. This includes guides with auto-generated links for symbols (belonging to the current module and ideally also to native frameworks) and proper semantic code highlighting which a simple static site generator can't do.

What I would also like to see is auto-generated "See Also" section:

[kean]

Hey, I hacked together a quick prototype using SwiftDoc and some of its dependencies. I hosted the end result here.

Some of the issues I encountered:

• The approach with <base href> completely breaks anchors (e.g. #deduplication). They now all redirect to the root URL instead of opening an anchor in the current document.
• There are limitations with the current approach of linking code elements. Running a highlighter and then attempting to add links only gets you so far. For example, it doesn't work for nested types, e.g. ImageProcessors.Resize, or methods though methods linking is not yet supported anyway.
• There are some major limitations in Markup
• There are some issues with code samples formatting
• SwiftDoc should probably not add underlines in code samples, it makes them unreadable. Instead, the user should just expect clicking on elements to work. There should be other way to communicate it.

I also started wondering what the user experience for these kinds of guides should be. Ideally, I would probably like it to be similar to that of Xcode Playgrounds, except that instead of jump-to-definition, clicking on an element would open an API reference. If SwiftDoc could also check whether code samples in your documentation compile, great! It sounds a lot like a pipe-dream, to be honest. But maybe this is something that could be implemented using the existing tools.

[Lukas-Stuehrk]

I worked on a small proof of concept to continue the discussion: master...Lukas-Stuehrk:AdditionalDocumentation

Problems solved:

• You can provide additional guides with a new flag--documentation path/to/the/directory.
• Providing a title: I simply take the title from the first level 1 heading in the document. If the document does not provide such a heading, I take the filename. I somehow like this approach. No processing instructions or any other mechanism needed.

Things to be discussed

• I simply added a new section in the homepage and the sidebar and called it "Additional Guides" (very inspired from jazzy).
• It's possible to overwrite a generated page (e.g. for a type) with a non-generated guide. I'm not sure if this is a cool feature or by accident because of my implementation approach.
• I flatten the file structure of the additional documents and do not preserve the tree structure of their folder. This might be very surprising, especially if you link between documents. This is rather easy to fix, I simply did not want to spend too much time on the routing, especially because @mattt already has a WIP pull request where the routing implementation is changed.
• This adds again a lot of functionality into swift-doc (the executable) and not to SwiftDoc (the library). Currently we can't add tests to swift-doc, except for the end-to-end tests. This starts to not feel right anymore, I think there are some parts that could be tested.

The ugly parts:

• The HTML and Markdown generation is....let's say it's not very sophisticated.

Please let me know what you think. I'm happy to work on this and to bring it into a mergeable state.