Crash in libuv for large directories if filesystem_watchers and git integration are enabled

[sebcode]

Description

nvim crashes reproducible for large directory trees if the options filesystem_watchers and git are enabled.

nvim: neovim/.deps/build/src/libuv/src/unix/core.c:270: uv__finish_close: Assertion `handle->flags & UV_HANDLE_CLOSING' failed.

I tried to debug this for a while and noticed that the git integration spawns git around 490 times triggered by watcher events when I install node dependencies before it crashes. I couldn't find the exact reason for the crash.

Workaround: For my use case I excluded node_modules from filesystem_watchers to avoid a huge pile of watcher events:

filesystem_watchers = {
  [...]
  ignore_dirs = {
    ".*/node_modules/.*",
  }
}

gdb stackstrace from generated corefile:

[Current thread is 1 (Thread 0x7fe4f3de7b80 (LWP 421383))]
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)
    at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>)
    at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)
    at ./nptl/pthread_kill.c:89
#3  0x00007fe4f3a3bc46 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007fe4f3a227fc in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007fe4f3a2271b in __assert_fail_base (
    fmt=0x7fe4f3bbac30 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=0x564852238d88 "!(handle->flags & UV_HANDLE_CLOSED)",
    file=0x564852238cc0 "/home/seb/devtmp/neovim/.deps/build/src/libuv/src/unix/core.c", line=271, function=<optimized out>) at ./assert/assert.c:92
#6  0x00007fe4f3a33596 in __GI___assert_fail (
    assertion=0x564852238d88 "!(handle->flags & UV_HANDLE_CLOSED)",
    file=0x564852238cc0 "/home/seb/devtmp/neovim/.deps/build/src/libuv/src/unix/core.c", line=271, function=0x564852238f00 <__PRETTY_FUNCTION__.9> "uv__finish_close")
    at ./assert/assert.c:101
#7  0x00005648520f541d in uv.finish_close ()
#8  0x00005648520f5615 in uv.run_closing_handles ()
#9  0x00005648520f58ec in uv_run ()
#10 0x0000564851e5b8b0 in loop_uv_run (loop=0x5648522f3ac0 <main_loop>, ms=0,
    once=true) at /home/seb/devtmp/neovim/src/nvim/event/loop.c:62
#11 0x0000564851e5b930 in loop_poll_events (loop=0x5648522f3ac0 <main_loop>, ms=0)
    at /home/seb/devtmp/neovim/src/nvim/event/loop.c:84
#12 0x0000564851f94801 in do_os_system (argv=0x5648547c2f80, input=0x0, len=0,
    output=0x7ffeb0c29bc0, nread=0x7ffeb0c29bb8, silent=true, forward_output=false)
    at /home/seb/devtmp/neovim/src/nvim/os/shell.c:860
#13 0x0000564851f9460a in os_system (argv=0x5648547c2f80, input=0x0, len=0,
    output=0x7ffeb0c29bc0, nread=0x7ffeb0c29bb8)
    at /home/seb/devtmp/neovim/src/nvim/os/shell.c:826
#14 0x0000564851e0ef03 in get_system_output_as_rettv (argvars=0x7ffeb0c29f40,
    rettv=0x7ffeb0c29ef0, retlist=false)
    at /home/seb/devtmp/neovim/src/nvim/eval.c:5729
#15 0x0000564851e41217 in f_system (argvars=0x7ffeb0c29f40, rettv=0x7ffeb0c29ef0,
    fptr=...) at /home/seb/devtmp/neovim/src/nvim/eval/funcs.c:8931
#16 0x0000564851e299df in call_internal_func (fname=0x564854589150 "system",
    argcount=1, argvars=0x7ffeb0c29f40, rettv=0x7ffeb0c29ef0)
    at /home/seb/devtmp/neovim/src/nvim/eval/funcs.c:207
#17 0x0000564851e50e41 in call_func (funcname=0x7fe4f3f065c8 "system", len=6,
    rettv=0x7ffeb0c29ef0, argcount_in=1, argvars_in=0x7ffeb0c29f40,
    funcexe=0x7ffeb0c29f00) at /home/seb/devtmp/neovim/src/nvim/eval/userfunc.c:1571
#18 0x0000564851eedfae in nlua_call (lstate=0x7fe4f3f03380)
    at /home/seb/devtmp/neovim/src/nvim/lua/executor.c:1103
#19 0x00005648521717a6 in lj_BC_FUNCC ()
#20 0x000056485215d9f9 in lua_pcall (L=0x7fe4f3f03380, nargs=<optimized out>,
    nresults=0, errfunc=<optimized out>) at lj_api.c:1116
#21 0x0000564851eeb2bd in nlua_pcall (lstate=0x7fe4f3f03380, nargs=0, nresults=0)

Neovim version

NVIM v0.8.2
Build type: Debug
LuaJIT 2.1.0-beta3

(Crashes for official build with `Build type: RelWithDebInfo` as well)

Operating system and version

Linux rz 5.19.0-29-generic #30-Ubuntu

nvim-tree version

96506fe

Minimal config

vim.cmd [[set runtimepath=$VIMRUNTIME]]
vim.cmd [[set packpath=/tmp/nvt-min/site]]
local package_root = "/tmp/nvt-min/site/pack"
local install_path = package_root .. "/packer/start/packer.nvim"
local function load_plugins()
  require("packer").startup {
    {
      "wbthomason/packer.nvim",
      "nvim-tree/nvim-tree.lua",
      "nvim-tree/nvim-web-devicons",
      -- ADD PLUGINS THAT ARE _NECESSARY_ FOR REPRODUCING THE ISSUE
    },
    config = {
      package_root = package_root,
      compile_path = install_path .. "/plugin/packer_compiled.lua",
      display = { non_interactive = true },
    },
  }
end
if vim.fn.isdirectory(install_path) == 0 then
  print "Installing nvim-tree and dependencies."
  vim.fn.system { "git", "clone", "--depth=1", "https://github.com/wbthomason/packer.nvim", install_path }
end
load_plugins()
require("packer").sync()
vim.cmd [[autocmd User PackerComplete ++once echo "Ready!" | lua setup()]]
vim.opt.termguicolors = true
vim.opt.cursorline = true

-- MODIFY NVIM-TREE SETTINGS THAT ARE _NECESSARY_ FOR REPRODUCING THE ISSUE
_G.setup = function()
  require("nvim-tree").setup {
	git = {
		enable = true,
		ignore = false,
	},
	filesystem_watchers = {
		enable = true,
		debounce_delay = 50,
	},
	log = {
		enable = true,
		types = {
			all = false,
			config = false,
			copy_paste = false,
			dev = false,
			diagnostics = false,
			git = true,
			profile = false,
			watcher = true,
        },
	},
	diagnostics = {
		enable = true,
		show_on_dirs = false,
	},
	update_focused_file = {
		enable = true
	},
	renderer = {
		group_empty = true,
	},
	filters = {
		dotfiles = false,
	},
  }
end

Steps to reproduce

git clone git@github.com:facebook/create-react-app.git
cd create-react-app
nvim -nu /tmp/nvt-min.lua
:NvimTreeToggle

In another terminal:
cd create-react-app
rm -rf node_modules && yarn #(Repeat a few times)

Expected behavior

No crash

Actual behavior

nvim crashes with

nvim: neovim/.deps/build/src/libuv/src/unix/core.c:270: uv__finish_close: Assertion `handle->flags & UV_HANDLE_CLOSING' failed.

[alex-courtis]

I can reproduce, after opening the node_modules directory.

We are running git jobs for gitignored directories. The issue does not occur when git is disabled.

[alex-courtis]

Idea:

git_toplevel returns nil when the directory is ignored.

Experimentation with always returning nil resulted in good behaviour with no UI latency. The overhead of the extra git calls is problematic.

An expiring cache of git toplevel may be a solution. We have this already cwd_to_project_root

[alex-courtis]

On each FS directory event we must:

• know the git toplevel
• update the git status
• update the contents

[gegoune]

Shouldn't something like node_modules be default of nvim-tree.filesystem_watchers.ignore_dirs?

[alex-courtis]

Shouldn't something like node_modules be default of nvim-tree.filesystem_watchers.ignore_dirs?

ignoring works as it prevents FS and git operations.

node_modules is pretty specific...

How about something more generic like:

    filesystem_watchers = {
      enable = true,
      debounce_delay = 50,
      ignore_dirs = {},
      gitignored = false,
    },

[gegoune]

Ignoring gitignore entries was my other thought but wouldn't cover cases for outside of repositories. But definitely is a good way forward.

[alex-courtis]

Ignoring gitignore entries was my other thought but wouldn't cover cases for outside of repositories. But definitely is a good way forward.

I reckon we should be OK for outside-of-repo in this case: the problem is definitely git, not FS refreshes.

ignore_dirs is the (only?) solution for non-repo cases.

I am now giving up my dream of always having watchers enabled.

[alex-courtis]

get_toplevel is used to determine the root of the git repo for all directories. When this is valid a cwd_to_project_root mapping is made and cached.

When refreshing a path with a cwd_to_project_root mapping a git status will be performed. This should not be done for ignored directories.

Solution: git toplevel determine whether the directory is ignored. Insert into a new array cwd_ignored instead of cwd_to_project_root in that case. get_project_root returns nil for cwd_ignored.

Challenge: get_toplevel determine toplevel and ignored in one git call.

[alex-courtis]

git check-ignore can take paths from stdin. That might be used to quickly build an initial list of ignored directories when building cwd_to_project_root, adding it to the project.

[DriedYellowPeach]

I may have the same issue, I also have a large directory with hundreds of files, and sometimes when I create new file, my neovim abort, but my abort message is: core.c:271: uv__finish_close: Assertion `handle->flags & UV_HANDLE_CLOSED failed. Didn't know whether this is the same issue.

[alex-courtis]

I may have the same issue, I also have a large directory with hundreds of files, and sometimes when I create new file, my neovim abort, but my abort message is: core.c:271: uv__finish_close: Assertion `handle->flags & UV_HANDLE_CLOSED failed. Didn't know whether this is the same issue.

It sounds similar. I'm still brainstorming ideas.

In the meantime, you can use :help nvim-tree.filesystem_watchers.ignore_dirs to exclude those large directories.