References Use db for findDef

save source file location to db
Find source for boot files

Use DynFlags from HieDb instead of unsafeGlobalDynFlags

Return multiple definitions

don't typecheck files on load

Add support for persistent stale values
Add persistent hie file rule

docs wip

better typedef

defs for deps

update hiedb

Fix for files with errors

Fix build

integrate hiedb commands and set dynflags on boot

workspace symbol tweaks, cabal.project

Write ifaces on save

use real mtime for saved files

safe indexing

bump hiedb

Proper refs for FOIs

hlint

Update exe/Main.hs

Co-authored-by: Pepe Iborra <pepeiborra@me.com>

Review comments

update hiedb

Update src/Development/IDE/Core/Shake.hs

Co-authored-by: Pepe Iborra <pepeiborra@me.com>

Update src/Development/IDE/Core/Rules.hs

Co-authored-by: Pepe Iborra <pepeiborra@me.com>

Update src/Development/IDE/Core/Rules.hs

Co-authored-by: Pepe Iborra <pepeiborra@me.com>

Update src/Development/IDE/Spans/AtPoint.hs

Co-authored-by: Pepe Iborra <pepeiborra@me.com>

Update src/Development/IDE/Core/Rules.hs

Co-authored-by: Pepe Iborra <pepeiborra@me.com>

Apply suggestions from code review

Co-authored-by: Pepe Iborra <pepeiborra@me.com>

more careful re-indexing

update for hiedb-0.1.0.0

Remove cached-deps stuff for now

explicit showSDoc

docs in AtPoint

add doc comment about database consistency

add TODO for better position mapping from diff

Make HLS compile with (GHC 8.10.2) and ghcide+hiedb.

Hacks to tactics plugic, class plugin, cabal file, and main.

HLS unit tests for the find references functionality.

Mark failing test.

Hlint fromRight hint.

Hlint nubOrd. Update ignore annotation to point to correct function.

This block of code that uses nub used to be part of main. But it got split off
into a new function called runIde, and the hlint annotation was not updated to
match.

Update tests to document that we now have references and workspace symbol providers.

Fix line number in broken test.

We have this code:
20: ddd :: Num a => a -> a -> a
21: ddd vv ww = vv +! ww

The intention was to ask for the type definition of symbol "a" in line 20, and
then assert that no type definitions were found.

The reality is that the test was asking for the definition of the symbol at (20,
15) in 0-based indexing, which is the "!" in "+!". Until recently, ghcide could
not find type definitions for "+!", so no type definitions were
found and the test passed. But now, ghcide can find type definitions for "+!",
and this test has begun to fail.

The solution is to change (20, 15) to (19, 15), so that we ask for the type
definitions of the symbol "a", which will not be found.

Update testcase.

Getting a type definition can produce more than one result. E.g. the type of the
symbol "pid" in:

1: data Parameter a = Parameter a
2: f :: forall a. Parameter a -> Parameter a
3: f pid = pid

is (Parameter a), and the definition of this type is two part: the definition of
Parameter on line 1, and the definition of a on line 2.

Kludge compilation for <=8.8.3.

FunTy takes a third argument in 8.10.1 and on.

Indexing improvments, persistent rules, diff mapping

fix position mapping for persistent

position mapping improvements

local type references

Move hiedb initialization stuff to session-loader

Document indexHieFile

Update HLS

formatting

hiedb from hackage

Unboxed vector for position mapping

tests

rebase fixes

Author: wz1000

ghcide/exe/Arguments.hs

@@ -1,41 +1,52 @@
 -- Copyright (c) 2019 The DAML Authors. All rights reserved.
 -- SPDX-License-Identifier: Apache-2.0
 
-module Arguments(Arguments(..), getArguments) where
+module Arguments(Arguments, Arguments'(..), getArguments, IdeCmd(..)) where
 
 import Options.Applicative
+import HieDb.Run
 
+type Arguments = Arguments' IdeCmd
 
-data Arguments = Arguments
+data IdeCmd = Typecheck [FilePath] | DbCmd Command | LSP
+
+data Arguments' a = Arguments
     {argLSP :: Bool
     ,argsCwd :: Maybe FilePath
-    ,argFiles :: [FilePath]
     ,argsVersion :: Bool
     ,argsShakeProfiling :: Maybe FilePath
     ,argsOTMemoryProfiling :: Bool
     ,argsTesting :: Bool
     ,argsDisableKick :: Bool
     ,argsThreads :: Int
     ,argsVerbose :: Bool
+    ,argFilesOrCmd :: a
     }
 
 getArguments :: IO Arguments
 getArguments = execParser opts
   where
     opts = info (arguments <**> helper)
       ( fullDesc
-     <> progDesc "Used as a test bed to check your IDE will work"
      <> header "ghcide - the core of a Haskell IDE")
 
 arguments :: Parser Arguments
 arguments = Arguments
       <$> switch (long "lsp" <> help "Start talking to an LSP server")
       <*> optional (strOption $ long "cwd" <> metavar "DIR" <> help "Change to this directory")
-      <*> many (argument str (metavar "FILES/DIRS..."))
       <*> switch (long "version" <> help "Show ghcide and GHC versions")
       <*> optional (strOption $ long "shake-profiling" <> metavar "DIR" <> help "Dump profiling reports to this directory")
       <*> switch (long "ot-memory-profiling" <> help "Record OpenTelemetry info to the eventlog. Needs the -l RTS flag to have an effect")
       <*> switch (long "test" <> help "Enable additional lsp messages used by the testsuite")
       <*> switch (long "test-no-kick" <> help "Disable kick. Useful for testing cancellation")
       <*> option auto (short 'j' <> help "Number of threads (0: automatic)" <> metavar "NUM" <> value 0 <> showDefault)
       <*> switch (long "verbose" <> help "Include internal events in logging output")
+      <*> ( hsubparser (command "typecheck" (info (Typecheck <$> fileCmd) fileInfo)
+                   <> command "hiedb" (info (DbCmd <$> cmdParser <**> helper) hieInfo)
+                   <> command "lsp" (info (pure LSP <**> helper) lspInfo)  )
+         <|> Typecheck <$> fileCmd )
+  where
+    fileCmd = many (argument str (metavar "FILES/DIRS..."))
+    lspInfo = fullDesc <> progDesc "Start talking to an LSP server"
+    fileInfo = fullDesc <> progDesc "Used as a test bed to check your IDE will work"
+    hieInfo = fullDesc <> progDesc "Query .hie files"

ghcide/exe/Main.hs

@@ -29,7 +29,7 @@ import Development.IDE.Types.Options
 import Development.IDE.Types.Logger
 import Development.IDE.Plugin
 import Development.IDE.Plugin.Test as Test
-import Development.IDE.Session (loadSession)
+import Development.IDE.Session (loadSession, setInitialDynFlags, getHieDbLoc, runWithDb)
 import Development.Shake (ShakeOptions (shakeThreads))
 import qualified Language.Haskell.LSP.Core as LSP
 import Language.Haskell.LSP.Messages
@@ -58,6 +58,9 @@ import Development.IDE.Plugin.HLS.GhcIde as GhcIde
 import Ide.Plugin.Config
 import Ide.PluginUtils (allLspCmdIds', getProcessID, pluginDescToIdePlugins)
 
+import HieDb.Types (LibDir(..))
+import HieDb.Run (Options(..), runCommand)
+
 ghcideVersion :: IO String
 ghcideVersion = do
   path <- getExecutablePath
@@ -78,6 +81,31 @@ main = do
     if argsVersion then ghcideVersion >>= putStrLn >> exitSuccess
     else hPutStrLn stderr {- see WARNING above -} =<< ghcideVersion
 
+    whenJust argsCwd IO.setCurrentDirectory
+
+    -- We want to set the global DynFlags right now, so that we can use
+    -- `unsafeGlobalDynFlags` even before the project is configured
+    libdir <- setInitialDynFlags
+
+    dir <- IO.getCurrentDirectory
+    dbLoc <- getHieDbLoc dir
+
+    case argFilesOrCmd of
+      DbCmd cmd -> do
+        let opts :: Options
+            opts = Options
+              { database = dbLoc
+              , trace = False
+              , quiet = False
+              , virtualFile = False
+              }
+        runCommand (LibDir $ fromJust libdir) opts cmd
+      Typecheck (Just -> argFilesOrCmd) | not argLSP -> runWithDb dbLoc $ runIde Arguments{..}
+      _ -> let argFilesOrCmd = Nothing in runWithDb dbLoc $ runIde Arguments{..}
+
+
+runIde :: Arguments' (Maybe [FilePath]) -> HieDb -> IndexQueue -> IO ()
+runIde Arguments{..} hiedb hiechan = do
     -- lock to avoid overlapping output on stdout
     lock <- newLock
     let logger p = Logger $ \pri msg -> when (pri >= p) $ withLock lock $
@@ -105,8 +133,8 @@ main = do
         options = def { LSP.executeCommandCommands = Just hlsCommands
                       , LSP.completionTriggerCharacters = Just "."
                       }
-
-    if argLSP then do
+    case argFilesOrCmd of
+      Nothing -> do
         t <- offsetTime
         hPutStrLn stderr "Starting LSP server..."
         hPutStrLn stderr "If you are seeing this in a terminal, you probably should have run ghcide WITHOUT the --lsp option!"
@@ -136,8 +164,8 @@ main = do
                   unless argsDisableKick $
                     action kick
             initialise caps rules
-                getLspId event wProg wIndefProg (logger logLevel) debouncer options vfs
-    else do
+                getLspId event wProg wIndefProg (logger logLevel) debouncer options vfs hiedb hiechan
+      Just argFiles -> do
         -- GHC produces messages with UTF8 in them, so make sure the terminal doesn't error
         hSetEncoding stdout utf8
         hSetEncoding stderr utf8
@@ -172,7 +200,7 @@ main = do
                     }
             defOptions = defaultIdeOptions sessionLoader
             logLevel = if argsVerbose then minBound else Info
-        ide <- initialise def mainRule (pure $ IdInt 0) (showEvent lock) dummyWithProg (const (const id)) (logger logLevel) debouncer options vfs
+        ide <- initialise def mainRule (pure $ IdInt 0) (showEvent lock) dummyWithProg (const (const id)) (logger logLevel) debouncer options vfs hiedb hiechan
 
         putStrLn "\nStep 4/4: Type checking the files"
         setFilesOfInterest ide $ HashMap.fromList $ map ((, OnDisk) . toNormalizedFilePath') files
@@ -201,7 +229,7 @@ main = do
 
         unless (null failed) (exitWith $ ExitFailure (length failed))
 
-{-# ANN main ("HLint: ignore Use nubOrd" :: String) #-}
+{-# ANN runIde ("HLint: ignore Use nubOrd" :: String) #-}
 
 expandFiles :: [FilePath] -> IO [FilePath]
 expandFiles = concatMapM $ \x -> do

ghcide/ghcide.cabal

@@ -60,6 +60,7 @@ library
         hie-compat,
         hls-plugin-api >= 0.6,
         lens,
+        hiedb == 0.2.0.*,
         mtl,
         network-uri,
         parallel,
@@ -73,6 +74,7 @@ library
         safe-exceptions,
         shake >= 0.18.4,
         sorted-list,
+        sqlite-simple,
         stm,
         syb,
         text,
@@ -82,6 +84,9 @@ library
         utf8-string,
         vector,
         hslogger,
+        Diff,
+        vector,
+        bytestring-encoding,
         opentelemetry >=0.6.1,
         heapsize ==0.3.*
     if flag(ghc-lib)
@@ -249,6 +254,8 @@ executable ghcide
     if flag(ghc-lib)
       buildable: False
     default-language:   Haskell2010
+    include-dirs:
+        include
     hs-source-dirs:     exe
     ghc-options:
                 -threaded
@@ -262,13 +269,15 @@ executable ghcide
                 "-with-rtsopts=-I0 -A128M"
     main-is: Main.hs
     build-depends:
+        hiedb,
         aeson,
         base == 4.*,
         data-default,
         directory,
         extra,
         filepath,
         gitrev,
+        ghc,
         hashable,
         haskell-lsp,
         haskell-lsp-types,

ghcide/session-loader/Development/IDE/Session.hs

@@ -1,4 +1,6 @@
 {-# LANGUAGE TypeFamilies #-}
+{-# LANGUAGE CPP #-}
+#include "ghc-api-version.h"
 
 {-|
 The logic for setting up a ghcide session by tapping into hie-bios.
@@ -8,6 +10,9 @@ module Development.IDE.Session
   ,defaultLoadingOptions
   ,loadSession
   ,loadSessionWithOptions
+  ,setInitialDynFlags
+  ,getHieDbLoc
+  ,runWithDb
   ) where
 
 -- Unfortunately, we cannot use loadSession with ghc-lib since hie-bios uses
@@ -71,6 +76,15 @@ import Packages
 import Control.Exception (evaluate)
 import Data.Void
 
+import HieDb.Create
+import HieDb.Types
+import HieDb.Utils
+import Database.SQLite.Simple
+import Control.Concurrent.STM.TQueue
+import Control.Concurrent.STM (atomically)
+import Maybes (MaybeT(runMaybeT))
+import HIE.Bios.Cradle (yamlConfig)
+
 
 data CacheDirs = CacheDirs
   { hiCacheDir, hieCacheDir, oCacheDir :: Maybe FilePath}
@@ -91,6 +105,47 @@ defaultLoadingOptions = SessionLoadingOptions
     ,getCacheDirs = getCacheDirsDefault
     }
 
+-- | Sets `unsafeGlobalDynFlags` on using the hie-bios cradle and returns the GHC libdir
+setInitialDynFlags :: IO (Maybe FilePath)
+setInitialDynFlags = do
+  dir <- IO.getCurrentDirectory
+  hieYaml <- runMaybeT $ yamlConfig dir
+  cradle <- maybe (HieBios.loadImplicitCradle $ addTrailingPathSeparator dir) HieBios.loadCradle hieYaml
+  libDirRes <- getRuntimeGhcLibDir cradle
+  libdir <- case libDirRes of
+      CradleSuccess libdir -> pure $ Just libdir
+      CradleFail err -> do
+        hPutStrLn stderr $ "Couldn't load cradle for libdir: " ++ show err
+        return Nothing
+      CradleNone -> return Nothing
+  dynFlags <- mapM (dynFlagsForPrinting . LibDir) libdir
+  mapM_ setUnsafeGlobalDynFlags dynFlags
+  pure libdir
+
+-- | Wraps `withHieDb` to provide a database connection for reading, and a `HieWriterChan` for
+-- writing. Actions are picked off one by one from the `HieWriterChan` and executed in serial
+-- by a worker thread using a dedicated database connection.
+-- This is done in order to serialize writes to the database, or else SQLite becomes unhappy
+runWithDb :: FilePath -> (HieDb -> IndexQueue -> IO ()) -> IO ()
+runWithDb fp k =
+  withHieDb fp $ \writedb -> do
+    initConn writedb
+    chan <- newTQueueIO
+    race_ (writerThread writedb chan) (withHieDb fp (flip k chan))
+  where
+    writerThread db chan = forever $ do
+      k <- atomically $ readTQueue chan
+      k db `catch` \e@SQLError{} -> do
+        hPutStrLn stderr $ "Error in worker, ignoring: " ++ show e
+
+getHieDbLoc :: FilePath -> IO FilePath
+getHieDbLoc dir = do
+  let db = dirHash++"-"++takeBaseName dir++"-"++VERSION_ghc <.> "hiedb"
+      dirHash = B.unpack $ B16.encode $ H.hash $ B.pack dir
+  cDir <- IO.getXdgDirectory IO.XdgCache cacheDir
+  createDirectoryIfMissing True cDir
+  pure (cDir </> db)
+
 -- | Given a root directory, return a Shake 'Action' which setups an
 -- 'IdeGhcSession' given a file.
 -- Some of the many things this does:
@@ -715,8 +770,8 @@ notifyUserImplicitCradle fp =
     NotificationMessage "2.0" WindowShowMessage $ ShowMessageParams MtInfo $
       "No [cradle](https://github.com/mpickering/hie-bios#hie-bios) found for "
       <> T.pack fp <>
-      ".\n Proceeding with [implicit cradle](https://hackage.haskell.org/package/implicit-hie).\n\
-      \You should ignore this message, unless you see a 'Multi Cradle: No prefixes matched' error."
+      ".\n Proceeding with [implicit cradle](https://hackage.haskell.org/package/implicit-hie).\n" <>
+      "You should ignore this message, unless you see a 'Multi Cradle: No prefixes matched' error."
 
 notifyCradleLoaded :: FilePath -> FromServerMessage
 notifyCradleLoaded fp =