Investigate supporting filesystem overlay in rustc

[Xanewok]

Rationale

This would allow us to support in-memory/patched (mainly for IDEs, maybe would be useful for build systems?) versions of source code when invoking rustc as a separate process.

While it's possible to provide a FileLoader trait implementation when running rustc, it requires the user to run the compiler in-process.

Use cases

1. We would like to stop the RLS from depending on Clippy the library, which breaks often due to relying on rustc internals and thus is responsible for most of the RLS disappearances from nightly builds (Decouple nightly RLS from Clippy #59761).
   I can't think of how we could solve this, short of ugly build-time hacks that would somehow detect if Clippy is compilable and if so, conditionally enable it via a feature (or provide a custom Clippy shim with IPC VFS support but that'd still include the lib and thus this comes back to the original stability problem).
   EDIT: Done in Update RLS #61670.

2. Due to the current RLS' architecture, we'd like to run the compiler as a separate process to compute the analysis in parallel - right now we compile all the primary crates in the workspace in a linear fashion in order to support easy VFS access to in-memory files and to reduce save-analysis serialization overhead.
   This particular case could be side-stepped by implementing IPC for our VFS implementation that could be relayed to using our FileLoader trait and rustc shim (rls-rustc).

Pros

• Uniform way to provide patched versions of the file at a process level

Cons

• This only helps dev tools use cases
• Has to be implemented and maintained
• We're not sure yet as to what the target architecture is for a rustc-as-a-language-server and in the long run this might not be necessary

I'd be interested in hearing your thoughts and whether we should ultimately support it.

cc'ing people that might be interested @nikomatsakis @matklad (anyone that I'm missing?)

[oblitum]

@Ixrec at rust-lang/rfcs#2912 (comment)

How hard would it be for an $EDITOR plugin to use both RLS and RA, and know to call one or the other for the various tasks one of them is currently better at?

It's not hard to configure it yourself, see rust-lang/rust-analyzer#4185 (comment) for example. The drawback is that it's heavy on memory.

[programmerjake]

it would be useful to somehow expose the virtual filesystem to proc macros and build.rs scripts so they also see up-to-date file contents.

Also, if the file content sharing could be implemented using shared memory, it would probably resolve a lot of the memory usage issues since all processes could use the exact same memory pages. maybe use memfd on linux.

[matklad]

it would probably resolve a lot of the memory

Raw file text occupies a negligible amount of memory

[nikomatsakis]

This came up again in the rust-analyzer RFC discussion, since some folks were complaining about the high cost of having to save in-progress edits over networked file systems.

[Mark-Simulacrum]

I've personally been coming around to the idea of something like this. I think we would want any design we come up with to work well with extensions to proc macros that permit file loading. I guess build scripts aren't feasible to patch at all to work with a filesystem overlay.

I would also like it if whatever we did worked as more than just an overlay, i.e., you could specify a mode where the files do not exist on disk at all (and are entirely given to rustc via some other protocol).

[leo60228]

Has there been any progress on this?

[bjorn3]

Not that I know of.

[leo60228]

Just to add a point of data as to why I want this (well, more specifically support for rustc errors in RA without saving, but they're closely related): I use vim as closer to a traditional editor instead of as an IDE, and using autosave makes "quit without saving" meaningless.

[ivan-aksamentov]

As I mentioned in intellij-rust/intellij-rust#8079, another side effect of automatic saving is triggering file watchers. This makes the "hot reload" dev workflows (automatic rebuild/test/restart on file changes) somewhat broken.