Leaking 10GB memory per minute with wrong type alias def `type Foo = Foo` for some ghc versions

[anka-213]

Opening this file in vs code

import Control.Monad.State
import Control.Applicative
import Data.List

data T = T [T] deriving (Show)
type Parser = Parser

ch :: Char -> Parser Char
ch c = mfilter (== c) $ StateT uncons

parens :: Parser T
parens = T <$ ch '(' <*> many parens <* ch ')'

parse :: Parser a -> String -> Maybe (a, String)
parse = runStateT

causes HLS to consume a lot of ram quickly. This is the result after just a few minutes of having the file open.

Log

[client] run command: "/Users/anka/Library/Application Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-0.3.0-darwin-8.8.3 --lsp"
[client] debug command: "/Users/anka/Library/Application Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-0.3.0-darwin-8.8.3 --lsp"
[client] server cwd: /Users/anka/projekt/haskell/bug
haskell-language-server version: 0.3.0.0 (GHC: 8.8.3) (PATH: /Users/anka/Library/Application Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-0.3.0-darwin-8.8.3) (GIT hash: d36bb9929fdd0df76f86d3635067400272f68497)
Starting (haskell-language-server)LSP server...
  with arguments: LspArguments {argLSP = True, argsCwd = Nothing, argFiles = [], argsShakeProfiling = Nothing, argsTesting = False, argsExamplePlugin = False, argsDebugOn = False, argsLogFile = Nothing, argsThreads = 0, argsProjectGhcVersion = False}
  with plugins: [PluginId "brittany",PluginId "eval",PluginId "floskell",PluginId "fourmolu",PluginId "ghcide",PluginId "ormolu",PluginId "pragmas",PluginId "retrie",PluginId "stylish-haskell"]
  in directory: /Users/anka/projekt/haskell/bug
If you are seeing this in a terminal, you probably should have run ghcide WITHOUT the --lsp option!
 Started LSP server in 0.00s
2020-08-25 22:14:15.349258 [ThreadId 31] - Opened text document: file:///Users/anka/projekt/haskell/bug/ParserInTweet.hs
2020-08-25 22:14:15.350402 [ThreadId 90] - Data.HashMap.Internal.(!): key not found
CallStack (from HasCallStack):
  error, called at ./Data/HashMap/Internal.hs:753:16 in nrdrd-cntnrs-0.2.12.0-a2bf77ff:Data.HashMap.Internal
  !, called at src/Development/IDE/Import/DependencyInformation.hs:107:40 in ghcide-0.2.0-inplace:Development.IDE.Import.DependencyInformation
2020-08-25 22:14:15.350857 [ThreadId 87] - Consulting the cradle for "/Users/anka/projekt/haskell/bug/ParserInTweet.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/anka/projekt/haskell/bug", cradleOptsProg = CradleAction: Default}
2020-08-25 22:14:15.496314 [ThreadId 87] - Using interface files cache dir: /Users/anka/.cache/ghcide/main-da39a3ee5e6b4b0d3255bfef95601890afd80709
2020-08-25 22:14:15.496422 [ThreadId 87] - Making new HscEnv[main]
2020-08-25 22:14:17.522717 [ThreadId 202] - Plugin.makeCodeLens (ideLogger)

---

Regardless of if this specific bug is fixed, we should probably add some limit, so HLS won't consume all available memory on the system when problems arise.

[anka-213]

I can reproduce it consistently on my machine, so if there is a good way for me to debug it locally, let me know.

[ndmitchell]

Thanks for the report, that seems brutal!

I just tried the simplest reproduction - create a new buffer, paste that in, toggle to Haskell mode. It uses 20.4Mb for me, with no visible change over 5 minutes. I was using 8.6.5 in VS Code. Therefore, I imagine there's some other factor involved.

Did you have that file saved? With a .cabal project? With a .stack project? Or just the raw file? Did you edit it somehow? Did RAM increase even when you were doing nothing? If you kept having to do something, what was it? Is this just the first file you tried, and thus all files are likely buggy? Or has it worked with other files?

As for setting a limit, certainly easy to do with the GHC heap settings. But I don't have any idea how to chose a good limit. For a big project I might want to dedicate 100Gb to the IDE. For my home machine, that would kill everything. There's no good solution I can think of...

[pepeiborra]

Could you share a full project including a hie.yaml file?
Have you tested with ghcide HEAD?

[pepeiborra]

I can reproduce it consistently on my machine, so if there is a good way for me to debug it locally, let me know.

We have an automated benchmark suite in the ghcide repo which loads a project and runs experiments. The first step would be to set this up as a benchmark that reproduces the leak that you are seeing.

[anka-213]

I can reproduce without having it in a project. Just a bare HS file.

Ram usage increases constantly, even when I do nothing. (was over 200GB when I had forgotten about it for a while)

I think it might be caused by some impurity, where some remnants of an older version still affects the current version, maybe?

[anka-213]

I'm not entirely sure how I would reproduce it in terms of the benchmark suite's commands

[pepeiborra]

I have no idea and I have very little time. Without a reproducer that I can run myself, I am unable to help

[googleson78]

just noting that there is something missing from the example:

type Parser = Parser

Isn't valid.

[anka-213]

I have no idea and I have very little time. Without a reproducer that I can run myself, I am unable to help
I understand. I'll see what I can do for reproduction.

I have not yet tested in ghcide HEAD. But the problem is still there when selecting ghcide as a backend.

[anka-213]

just noting that there is something missing from the example:

type Parser = Parser

Isn't valid.

Oh, right! That's probably the issue!

I was using the retrie plugin to "fold" Parser (possibly in another file) and it seems like it replaced it in such a way to cause an infinite loop in the type alias.

[anka-213]

Maybe it is already fixed on a newer version of ghcide, which is why people had trouble reproducing. This is my version:

 $ "/Users/anka/Library/Application Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-0.3.0-darwin-8.8.3" --version
haskell-language-server version: 0.3.0.0 (GHC: 8.8.3) (PATH: /Users/anka/Library/Application Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-0.3.0-darwin-8.8.3) (GIT hash: d36bb9929fdd0df76f86d3635067400272f68497)`

If so, the issue that needs fixing is to place some memory limit so the process can't consume arbitrarily much memory if a similar issue arises.

[pepeiborra]

Retrie folds can be very memory hungry. I submitted a related PR a few days ago: #321

To be sure, try to repro with the retrie command line tool