Standardise source encoding and line endings

[rjmholt]

The source code for PSES seems to have inconsistent BOM usage and line endings. It's not a huge issue, but making a change recently caused VSCode to redo some line endings and generated a large number of false changes, which mainly just makes git diffs impossible to review.

We should pick a standard of BOM usage (as in, UTF-8 with BOM or UTF-8 without BOM) and of line endings (\n vs \r\n). Ideally, a pragmatic choice will be one that works cross platform and which editors and tools are least likely to change back/interfere with.

There are a couple of points of information worth noting here:

• git can handle line-ending conversion nicely, but is usually only configured to do so on Windows.
• VSCode allows "files.encoding" to take "utf8" and "utf8bom" as values, and I'm assuming it honours the encoding.
• VSCode also has a "files.eol" setting which can be set to "\n" or "\r\n", but apparently it ignores this setting if it finds a CRLF in the file and just reverts to CRLF.

I'm not sure how good other editors are at handling encoding issues like this; to fix the problem I had, I ended up needing to use vim to prevent VSCode from automatically changing line endings. I'm hoping that Atom and SublimeText will also honour encoding settings.

[rkeithhill]

This likely from its origin as a VS only code-base to a combo of VS and VSCode editing. I still primarily use Visual Studio 2017 for editing PSES. I suggest we add a .gitattributes file to specify the file line-endings for the repo. We could use the one from the PS repo.

As for file encoding, it seems that VS wants to use utf8 w/bom whereas VSCode defaults to utf8 no bom. We could switch to utf8 no bom to be consistent as long as that doesn't mess with VS. Hopefully VS preserves the orginal encoding. I don't want to have to fight VS to get it to save w/no bom.

RE EOL, why don't we add a .editorconfig to the PSES root. We can specify EOL style and that should be honored by both VS and VSCode. Although for the latter, we should add a .vscode\extensions.json file with the following contents:

{
    // See http://go.microsoft.com/fwlink/?LinkId=827846
    // for the documentation about the extensions.json format
    "recommendations": [
        "ms-vscode.csharp",
        "ms-vscode.PowerShell",
        "EditorConfig.EditorConfig",
        "DavidAnson.vscode-markdownlint"
    ]
}

It is the EditorConfig extension that will make VSCode use the .editorconfig file.

[rjmholt]

Yeah, I'd be in favour of any choices that minimise the possibility of different editors changing it back and forth and making noise. So if VS is less tolerant and CRLF will lead to more stability, so be it.

Same with BOM -- I know PowerShell is trying to standardise on BOM-less, but I'd be happiest with whatever causes the fewest unintentional changes.

I imagine .gitattributes is the best first measure, since it can hook into translating the formats relatively transparently.

Do we know how tolerant Visual Studio is of no-BOM and LF?

[rjmholt]

Looks like .editorconfig is needed to use LF manageably in VS. BOM still looks trickier. The VS Code EditorConfig plugin also doesn't seem to support charset yet to manage encodings.

Given that, plus the fact we are still mostly supporting and being developed on Windows, I'm thinking the best encodings to use are CRLF and UTF-8 with BOM (despite my moral convictions). We can always change it down the track if we feel that cross-platform development has more traction and Windows tools support it better.

[rkeithhill]

RE EOL, the .gitattributes submitted in this PR is the same as the one used by the PowerShell repo and it uses auto. So on Windows, EOL will be CRLF and on Linux it will be LF. Not sure what it will be on macOS but it should be the native EOL.

[rjmholt]

@rkeithhill that sounds perfect. I think it should be LF on macOS (gone are the days of CR on classic Mac).

[rjmholt]

Re BOM, the other possibility is possibly to ignore it:
https://stackoverflow.com/questions/27223985/git-ignore-bom-prevent-git-diff-from-showing-byte-order-mark-changes

[rkeithhill]

RE encoding, it looks to me that VS thinks the files it created are encoded with codepage 1252. VSCode thinks the same files are UTF8 no bom? Probably best to avoid any mass conversions at this point. However, the PS1/PSM1/PSD1 files in the PSES repo could be settled upon as UTF8 no BOM since we (well I) mostly edit those in VSCode anyway. :-)

[rkeithhill]

Closed by PR #667

[rjmholt]

codepage 1252

😩 CP1252 is such a thorn in our sides!

[rjmholt]

My personal thought is that if we have files in different encodings in the repo, and if there are files using different line endings in the repo, we should make them all the same rather than having a heterogenously encoded code base.

Checking code in and out in a way that preserves things makes sense too, but doesn't prevent the canonical source code from being weird.