Non-ascii module names break `cabal sdist`

[hasufell]

cabal-install uses the tar package, which just recently got proper unicode support.

The following module breaks cabal sdist: https://github.com/haskell/haskell-language-server/blob/master/plugins/hls-semantic-tokens-plugin/test/testdata/TModula%F0%90%90%80bA.hs

It will truncate the module name to TModula:

$ tar -tvf dist-newstyle/sdist/haskell-language-server-2.7.0.0.tar.gz | grep plugins/hls-semantic-tokens-plugin/test/testdata/            Thu 11:32
drwxr-xr-x 0/0               0 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/
-rw-r--r-- 0/0             722 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/T1.hs
-rw-r--r-- 0/0              59 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TClass.hs
-rw-r--r-- 0/0             277 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TClassImportedDeriving.hs
-rw-r--r-- 0/0             295 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TDataFamily.hs
-rw-r--r-- 0/0              57 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TDataType.hs
-rw-r--r-- 0/0              75 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TDatatypeImported.hs
-rw-r--r-- 0/0              84 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TDoc.hs
-rw-r--r-- 0/0              76 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TFunction.hs
-rw-r--r-- 0/0              57 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TFunctionLet.hs
-rw-r--r-- 0/0              77 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TFunctionLocal.hs
-rw-r--r-- 0/0             125 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TFunctionUnderTypeSynonym.hs
-rw-r--r-- 0/0             236 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TGADT.hs
-rw-r--r-- 0/0             104 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TInstanceClassMethodBind.hs
-rw-r--r-- 0/0              52 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TInstanceClassMethodUse.hs
-rw-r--r-- 0/0              63 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TModula
-rw-r--r-- 0/0             123 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TModuleB.hs
-rw-r--r-- 0/0              74 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TNoneFunctionWithConstraint.hs
-rw-r--r-- 0/0             306 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TOperator.hs
-rw-r--r-- 0/0              50 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TPatternMatch.hs
-rw-r--r-- 0/0              82 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TPatternSynonym.hs
-rw-r--r-- 0/0              57 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TPatternbind.hs
-rw-r--r-- 0/0             142 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TQualifiedName.hs
-rw-r--r-- 0/0              56 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TRecord.hs
-rw-r--r-- 0/0             122 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TRecordDuplicateRecordFields.hs
-rw-r--r-- 0/0             113 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TTypefamily.hs
-rw-r--r-- 0/0              49 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TUnicodeSyntax.hs
-rw-r--r-- 0/0              73 2001-09-09 09:46 haskell-language-server-2.7.0.0/plugins/hls-semantic-tokens-plugin/test/testdata/TValBind.hs

@fendor @soulomoon

[michaelpj]

Checking that I understand:

• This is a bug in cabal sdist that is now hopefully fixed? Or does cabal need to update to use newer tar?
• The consequences in the short term are that someone won't be able to successfully run the semantic tokens tests from a HLS sdist

So I guess we could remove the character for the next release or just live with the consequences until cabal catches up.

[hasufell]

This is a bug in cabal sdist that is now hopefully fixed? Or does cabal need to update to use newer tar?

Not sure. It depends on the tar version used.

So I guess we could remove the character for the next release or just live with the consequences until cabal catches up.

Even if cabal catches up, you'd have to ensure everyone updates to the cabal version that has proper unicode support, which is not realistic.

Additionnally, hackage rejects packages with non-ascii filenames: https://github.com/haskell/hackage-server/blob/8e3655845b2f152e3cd7dcfae522f6fdafba8950/src/Distribution/Server/Packages/Unpack.hs#L445-L448

Please don't use them.

[michaelpj]

Additionnally, hackage rejects packages with non-ascii filenames

That's a good reason for sure.

[soulomoon]

Additionnally, hackage rejects packages with non-ascii filenames

Okay, guess we would have to remove the non-ascii in filename.

[hasufell]

Well, nothing is uploaded to hackage yet and it isn't released in ghcup proper either.

[fendor]

@hasufell It seems cabal sdist ignores truncates the module. Looks like it doesn't crash hackage if we upload it like that?

Any way, do you want the PRs #4103, #4105 and #4106 backported to the branch wip/2.7.0.0?

[hasufell]

#4103 and #4105

[fendor]

Ok, will do once #4105 is merged and ping you again.