Retry tests on SQLError

[pepeiborra]

The PR does a few things:

• All the ghcide tests now run in-process using runServerWithHandles
• Cleaned up a few cli flags
• The tests now send logs to the LSP stream. This seems the only reasonable thing to do, but I'm open to other ideas
• Use the logger consistently in places that were printing directly to stderr
• The tests now retry once on SQLError, after a 1s sleep. This should hopefully help with Non-deterministic CI test failures #1430
• The ghcide binary is no longer tested. This is fine - the ghcide README states that the ghcide binary is unsupported
• Running tests in parallel fails in a new way even though I have removed all the calls to setCurrentDirectory:

  "Finishing build session(exception: Lint checking error - current directory has changed:\n  When:    When running\n  Wanted:  /private/tmp/extra-dir-95351899043\n  Got:     /private/tmp/extra-dir-95351899044\n)"

This turned out to be due to haskell/lsp#326

[berberman]

It seems that ghcide-tests: <file descriptor: 14>: hPutBuf: resource vanished (Broken pipe) and lsp:Got EOF, exiting 1 ... are messing up the test report:

ghcide
  open close:                                OK (0.04s)
  get
    definition
      field in record definition:            ghcide-tests: <file descriptor: 14>: hPutBuf: resource vanished (Broken pipe)
OK (0.68s)
      field in record construction    #1102: OK (0.39s)
      field name used as accessor:           OK (0.39s)
      top-level name:                        OK (0.36s)
      data constructor record         #1029: OK (0.40s)
      data constructor plain:                OK (0.36s)
      type constructor                #1028: lsp:Got EOF, exiting 1 ...
ghcide-tests: <file descriptor: 59>: hPutBuf: resource vanished (Broken pipe)

OK (0.41s)
      type constructor external   #717,1028: FAIL (expected: known broken) (0.33s)
        test/exe/Main.hs:3306:
        Expecting to fail to find in external file (expected failure)
      value external package           #717: FAIL (expected: known broken) (0.40s)
        test/exe/Main.hs:3306:
        Expecting to fail to find in external file (expected failure)
      plain parameter:                       OK (0.37s)
      pattern match name:                    OK (0.42s)
      top-level operator               #713: OK (0.43s)
      parameter operator:                    OK (0.54s)
      name in backticks:                     OK (0.42s)
      class in instance declaration   #1027: OK (0.34s)
      class in signature              #1027: OK (0.43s)
      external class in signature #717,1027: FAIL (expected: known broken) (0.37s)
        test/exe/Main.hs:3306:
        Expecting to fail to find in external file (expected failure)
      do-notation   bind              #1073: OK (0.43s)
      do-notation lookup:                    OK (0.35s)
      listcomp   bind                 #1073: OK (0.40s)
      listcomp lookup:                       ghcide-tests: <file descriptor: 51>: hPutBuf: resource vanished (Broken pipe)
lsp:Got EOF, exiting 1 ...
OK (0.41s)
      top-level fn 1st clause:               OK (0.39s)
      top-level fn 2nd clause         #1030: OK (0.38s)
      top-level fn on space           #1002: OK (0.34s)
      top-level signature              #767: FAIL (expected: known broken) (0.41s)
        test/exe/Main.hs:3313:
        expected: Range {_start = Position {_line = 46, _character = 0}, _end = Position {_line = 46, _character = 5}}
         but got: Range {_start = Position {_line = 50, _character = 0}, _end = Position {_line = 50, _character = 5}} (expected failure)
      inner     signature              #767: FAIL (expected: known broken) (0.39s)
        test/exe/Main.hs:3313:
        expected: Range {_start = Position {_line = 49, _character = 2}, _end = Position {_line = 49, _character = 7}}
         but got: Range {_start = Position {_line = 53, _character = 2}, _end = Position {_line = 53, _character = 7}} (expected failure)
      Imported symbol:                       OK (0.34s)
      Imported symbol (reexported):          ghcide-tests: <file descriptor: 82>: hPutBuf: resource vanished (Broken pipe)
lsp:Got EOF, exiting 1 ...

OK (0.42s)
    hover
      field in record definition:            OK (0.39s)
      field in record construction    #1102: OK (0.40s)
      field name used as accessor:           OK (0.34s)
      top-level name:                        OK (0.43s)
      data constructor record         #1029: OK (0.34s)
      data constructor plain:                OK (0.49s)
      type constructor                #1028: OK (0.34s)
      type constructor external   #717,1028: OK (0.40s)
      value external package           #717: OK (0.34s)
      plain parameter:                       OK (0.35s)
      pattern match name:                    OK (0.39s)
      top-level operator               #713: OK (0.38s)
      parameter operator:                    OK (0.36s)
      name in backticks:                     lsp:Got EOF, exiting 1 ...

ghcide-tests: <file descriptor: 61>: hPutBuf: resource vanished (Broken pipe)
OK (0.40s)
      class in instance declaration   #1027: OK (0.32s)
      class in signature              #1027: OK (0.41s)
      external class in signature #717,1027: OK (0.43s)
      do-notation   bind              #1073: OK (0.36s)
      do-notation lookup:                    OK (0.41s)
      listcomp   bind                 #1073: OK (0.36s)
      listcomp lookup:                       OK (0.32s)
      top-level fn 1st clause:               OK (0.40s)
      top-level fn 2nd clause         #1030: OK (0.32s)
      top-level fn on space           #1002: OK (0.39s)
      documentation                   #1129: OK (0.40s)
      kind of Either                  #1017: OK (0.32s)
      kind of Int                     #1017: OK (0.39s)

EDIT: this problem also arose in #1628, but silenceStderr helped conceal it.

[berberman]

I suspect the test was aborted because of OOM: https://github.com/haskell/haskell-language-server/pull/1752/checks?check_run_id=2376999816

Actually I observed that the memory usage of tactics plugin test suite increased significantly in #1628, where I couldn't run full test on my machine. But it got improved as I tried to recompile all packages with enable-profiling flags 🤔

[pepeiborra]

I will need some time to investigate if the test suite OOMs after this change, probably won't happen until next weekend

UPDATE: It's currently using 85G and still running, so definitely something going on here. Massive space leak?

[pepeiborra]

It seems that ghcide-tests: <file descriptor: 14>: hPutBuf: resource vanished (Broken pipe) and lsp:Got EOF, exiting 1 ... are messing up the test report:

I think those lines denote tests that failed because of the SQLError and the retry is kicking in. That is good.

Even if the test report is being a little dirty, I prefer not to silence the output since that will make test debugging much harder. Imho the silencing should be done by the test framework, and then the full output of test failures included at the end of the test report. I'm not a tasty expert so I don't know whether it supports this feature, either directly or via some plugin.

[pepeiborra]

I'll rebase this when the new his-graph implementation lands

[pepeiborra]

This seems to be less of a problem these days, so closing