Doc export file paths all start with `ile://`

[rhys-vdw]

How are you using the lua-language-server?

Command Line

Which OS are you using?

Windows WSL

What is the issue affecting?

Other

Expected Behaviour

Paths in exported doc JSON should start with file:/// as in the example:

"file": "file:///c%3A/Users/Me/Documents/Array.lua",

Actual Behaviour

Missing the leading f:

"file": "ile:///home/rhys/spring/./cont/base/springcontent/LuaHandler/handler.lua",

Reproduction steps

rhys@RhysDesktop:~/spring$ lua-language-server --doc . --doc_out_path docs/
root uri = file:///home/rhys/spring/.
Documentation exported:91/2488

docs/doc.json
docs/doc.md

Additional Notes

rhys@RhysDesktop:~/spring$ lua-language-server --version
<Unknown>

rhys@RhysDesktop:~/spring$ lua-language-server --help
Content-Length: 55

{"jsonrpc":"2.0","method":"$/hello","params":["world"]}

How do you get a version?

Log File

No response

[rhys-vdw]

Ah...

rhys@RhysDesktop:~/spring$ brew info lua-language-server
==> lua-language-server: stable 3.13.0 (bottled), HEAD

[tomlau10]

This may be a related issue: #2964, which points out that the file uri format seems changed since a recent refactor PR #2821 by a contributor.

I strongly suspect that there maybe some hidden bugs in that refactor PR. 🤔

[rhys-vdw]

We are currently working on a full documentation export for a large RTS engine, so it should be a good test case. Is there a prior release version we can try rolling back to to compare doc export before this PR was merged?

[tomlau10]

By looking at the release page, #2821 is released in v3.10.6, so I guess you may try the prior release v3.10.5 🤔

[rhys-vdw]

I can't work out how to install a historic version with brew.

rhys@RhysDesktop:~/spring/doc/site$ brew install lua-language-server@3.10.5
Warning: No available formula with the name "lua-language-server@3.10.5". Did you mean lua-language-server?
==> Searching for similarly named formulae...
==> Formulae
lua-language-server

To install lua-language-server, run:
  brew install lua-language-server

rhys@RhysDesktop:~/spring/doc/site$ brew search lua-language-server
==> Formulae
lua-language-server                    vala-language-server                   yaml-language-server                   sql-language-server                    bash-language-server

Is this possible?

[rhys-vdw]

Okay, I just installed directly from the GitHub releases. I can confirm that this version does not have the above issue, and also fixes #2977. It seems this PR may have introduced a number of issues.

[tomlau10]

I just took some time to debug into this, found the logic that causing this issue, but have yet come up with a good solution 😕

• the file path of each file is transformed by this export.getLocalPath()
  

  lua-language-server/script/cli/doc/export.lua

  Lines 57 to 65 in 7f76050

  	function export.getLocalPath(uri)
  	--remove uri root (and prefix)
  	local local_file_uri = uri
  	local i, j = local_file_uri:find(DOC)
  	if not j then
  	return '[FORIEGN]'..uri
  	end
  	return local_file_uri:sub( j + 1 )
  	end

• the uri here is the file:///... string
• the DOC global here is the --doc=xxx value that passed in via CLI

---

I believe this logic intends to remove the path prefix, such that each paths in the output are relative to the supplied directory. For example:

• /data/test/a.lua
• /data/test/b.lua
• by using --doc=/data/test
  => the expected paths in output md/json will just be /a.lua, /b.lua

‼️ However when using --doc=., since . means match anything in a standard string.find()
=> this just match the 1st char
=> j = 1 (the end index of the match)
=> so the last statement return local_file_uri:sub( j + 1 ) gives ile://... 🤦‍♂️

Now you may think that we can simply pass plaintext=true in string.find
=> local_file_uri:find(DOC, 1, true)
But that's not enough ‼️
Because DOC can just be ., and there can be cases that the project folder is located in some directory with name aaa.bbb
=> this will still matches the . in the middle

---

Workaround 1

• use absolute path when invoking from cli, such as

lua-language-server --doc=$PWD --doc_out_path docs/

=> each file path will now be /a.lua, /b.lua, ...

• if you don't want to / at the begining, then use $PWD/

lua-language-server --doc=$PWD/ --doc_out_path docs/

=> each file path will now be a.lua, b.lua, ...

Workaround 2

Judging from the diff in #2821, the original handling of file name is just to use guide.getUri()
And that #2821 actually provides a Lua.docScriptPath config for you to override the functions in that export module.
So you can have the following:

• in .luarc.json

   ...
   "docScriptPath": "export.lua"

• export.lua

function export.getLocalPath(uri)
    return uri
end

• finally trigger through cli using absolute path with /

lua-language-server --doc=$PWD/ --doc_out_path docs/

=> now you should have the classic uri result

[AndreasMatthias]

Now you may think that we can simply pass plaintext=true in string.find => local_file_uri:find(DOC, 1, true) But that's not enough ‼️ Because DOC can just be ., and there can be cases that the project folder is located in some directory with name aaa.bbb => this will still matches the . in the middle

How about converting all paths into canonical form before using string:find()?

lua-language-server/script/cli/doc/export.lua

Lines 57 to 65 in 7f76050

	function export.getLocalPath(uri)
	--remove uri root (and prefix)
	local local_file_uri = uri
	local i, j = local_file_uri:find(DOC)
	if not j then
	return '[FORIEGN]'..uri
	end
	return local_file_uri:sub( j + 1 )
	end

local fileuri = require 'file-uri'
local fs = require 'bee.filesystem'

function export.getLocalPath(uri)
    local file_canonical = tostring(fs.canonical(fileuri.decode(uri)))
    local doc_canonical = tostring(fs.canonical(DOC))
    local _, j = file_canonical:find(doc_canonical .. '/', 1, true)
    if not j then
        return '[FOREIGN] ' .. uri
    end
    return file_canonical:sub(j + 1)
end

[rhys-vdw]

Surely there is a library for dealing with URIs/paths that can be used. Is there no equivalent to Node.js's path.relative? e.g. path.relative(DOC, uri).

[tomlau10]

How about converting all paths into canonical form before using string:find()?

I just tested and your snippet seems working well 👍

Just a few comments about the suggested change:

• file-uri is required as local furi as seen in all other places in the codebase

local furi = require 'file-uri'

• the fs.path class object has a :string() method, codebase is using this instead of doing a tostring()

    local file_canonical = fs.canonical(fileuri.decode(uri)):string()
    local doc_canonical = fs.canonical(DOC):string()

maybe you can create a PR for it 😄 @AndreasMatthias

---

Is there no equivalent to Node.js's path.relative? e.g. path.relative(DOC, uri).

Seems not 😕 according to the api listed in the meta file: https://github.com/LuaLS/lua-language-server/blob/115a518ad995bfc6ae8dabf216451055e1633898/script/meta/bee/filesystem.lua

[AndreasMatthias]

This is a summary of how different commits are writing file names into doc.json.

I was puzzled to see that a trailing slash (backslash) changes the output on Windows. E.g. --doc=./ doesn't work well on Window. See the tables below. Interestingly this issue (Is it an issue?) is not related to #2821.

The tables below show file names of two different file types: a project file (marked with P in the second column) and a foreign (non-project) file (marked with F), i.e.:

On Linux:

• project file: /project/subdir/project.lua
• foreign file: /foreign/subdir/foreign.lua

On Windows:

• project file F:\project\subdir\project.lua
• foreign file: C:\foreign\subdir\foreign.lua

Question: Should non-project files be marked with a [FOREIGN] tag in doc.json? This was new to me. Sorry, I didn't follow the development closely.

Original ba8f90e (tag: 3.10.5)

This is the original behavior. It was the release just before the changes in #2821.

Linux

DOC arg	M	Path in doc.json	Comments
--doc=.	P	file:///project/subdir/project.lua
	F	file:///foreign/subdir/foreign.lua
--doc=./	P	file:///project/subdir/project.lua
	F	file:///foreign/subdir/foreign.lua
--doc=/project	P	file:///project/subdir/project.lua
	F	file:///foreign/subdir/foreign.lua
--doc=/project/	P	file:///project/subdir/project.lua
	F	file:///foreign/subdir/foreign.lua

Windows

DOC arg	M	Path in doc.json	Comments
--doc=.	P	file:///f%3A/project/subdir/project.lua
	F	file:///c%3A/foreign/subdir/foreign.lua
--doc=.\	P		project files missing
	F	file:///c%3A/foreign/subdir/foreign.lua
--doc=F:\project	P	file:///f%3A/project/subdir/project.lua
	F	file:///c%3A/foreign/subdir/foreign.lua
--doc=F:\project\	P		project files missing
	F	file:///c%3A/foreign/subdir/foreign.lua

Current master 115a518

Linux

DOC arg	M	Path in doc.json	Comments
--doc=.	P	ile://project/subdir/project.lua	f missing
	F	ile://foreign/subdir/foreign.lua	f missing, no [FOREIGN] tag
--doc=./	P	//project/subdir/project.lua	// is wrong
	F	//foreign/subdir/foreign.lua	// is wrong, no [FOREIGN] tag
--doc=/project	P	/subdir/project.lua	/ is wrong (rel not abs path)
	F	[FOREIGN] file:///foreign/subdir/foreign.lua
--doc=/project/	P	subdir/project.lua
	F	[FOREIGN] file:///foreign/subdir/foreign.lua

Windows

DOC arg	M	Path in doc.json	Comments
--doc=.	P	ile:///f%3A/project/subdir/project.lua	f missing
	F	ile:///c%3A/foreign/subdir/foreign.lua	f missing, no [FOREIGN] tag
--doc=.\	P		project files missing
	F	//c%3A/foreign/subdir/foreign.lua	// is wrong
--doc=F:\project	P	[FOREIGN] file:///f%3A/project/subdir/project.lua	[FOREIGN] tag is wrong
	F	[FOREIGN] file:///c%3A/foreign/subdir/foreign.lua
--doc=F:\project\	P		project files missing
	F	[FOREIGN] file:///c%3A/foreign/subdir/foreign.lua

My patch

Linux

DOC arg	M	Path in doc.json	Comments
--doc=.	P	subdir/project.lua
	F	[FOREIGN] file:///foreign/subdir/foreign.lua
--doc=./	P	subdir/project.lua
	F	[FOREIGN] file:///foreign/subdir/foreign.lua
--doc=/project	P	subdir/project.lua
	F	[FOREIGN] file:///foreign/subdir/foreign.lua
--doc=/project/	P	subdir/project.lua
	F	[FOREIGN] file:///foreign/subdir/foreign.lua

Windows

DOC arg	M	Path in doc.json	Comments
--doc=.	P	subdir/project.lua
	F	[FOREIGN] file:///c%3A/foreign/subdir/foreign.lua
--doc=.\	P		project files missing
	F	[FOREIGN] file:///c%3A/foreign/subdir/foreign.lua
--doc=F:\project	P	subdir/project.lua
	F	[FOREIGN] file:///c%3A/foreign/subdir/foreign.lua
--doc=F:\project\	P		project files missing
	F	[FOREIGN] file:///c%3A/foreign/subdir/foreign.lua

[rhys-vdw]

I also noticed that providing relative paths does strange things, for example --doc=../../src would retain the ../.. in the output file field. I don't have an example on me, but obviously this is wrong.

[rhys-vdw]

Is there no equivalent to Node.js's path.relative? e.g. path.relative(DOC, uri).

Seems not 😕 according to the api listed in the meta file: https://github.com/LuaLS/lua-language-server/blob/115a518ad995bfc6ae8dabf216451055e1633898/script/meta/bee/filesystem.lua

— @tomlau10

I don't really know how this all works, but it looks like such a function exists within bee.lua as path.relative_path. Perhaps it exists but is not documented? Or the dependency could be upgraded to include it.