Command::output panics on Linux if more than 1024 file descriptors are open

[matklad]

Issue originally reported by @knsd.

Command::open uses select on Linux:

rust/src/libstd/sys/unix/pipe.rs

Line 90 in 4465f22

libc::select(max + 1, &mut read, ptr::null_mut(), ptr::null_mut(),

to read from both stderr and stdout. This won't always work, because select is limited to file descriptors less than FD_SETSIZE (1024).

Thus, if you raise ulimit -n to more than 1024, the following code panics

use std::fs::File;
use std::process::Command;

const FD_SETSIZE: usize = 1024;
const STDIO: usize = 3;
const EXE: &'static str = "/usr/bin/env";

fn main() {
    let mut files = Vec::new();

    for _ in 0..FD_SETSIZE - STDIO * 2 {
        let file = File::open(&EXE).unwrap();
        files.push(file)
    }

    let output = Command::new(EXE).output();
    println!("{:?}", output)
}

thread 'main' panicked at 'index out of bounds: the len is 16 but the index is 16', /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libcore/slice.rs:664

[alexcrichton]

Wow.

Times like this are when I'm super glad we use rust, otherwise this'd be a super weird segfault!

We should probably just do epoll/kqueue and/or fall back to spawning threads when this happens.

[alexcrichton]

@matklad out of curiosity how'd you discover this?

[Stebalien]

You could also just use poll for maximum compatibility.

[matklad]

@matklad out of curiosity how'd you discover this?

It's better to ask @knsd :)

You could also just use poll for maximum compatibility.

Looks like poll does not always work on mac: https://daniel.haxx.se/blog/2016/10/11/poll-on-mac-10-12-is-broken/. Perhaps it does not matter in this simple case of two file descriptors without timeouts.

[internetionals]

Under Linux this could have also been solved by allocating a dynamic bitmap as fdset. That way you can easily watch hundreds of thousands of filedescriptors using plain 'select'.

Note that watching fdsets with filedescriptors higher than 8192 or so would see increasing runtime overhead, as the kernel has to scan the bitmap for possible fds to watch on each call to select.