[SR-2941] [libdispatch] kqtest fails randomly with ``bind-1: Address already in use``

[swift-ci]

Previous ID	SR-2941
Radar	None
Original Reporter	erg (JIRA User)
Type	Bug
Status	Resolved
Resolution	Done

Additional Detail from JIRA

Votes	0
Component/s	libdispatch
Labels	Bug, Linux
Assignee	dgrove-oss (JIRA)
Priority	Medium

md5: 10e6c90a348bb7f243bf43226a8393c3

Issue Description:

The test needs to be more robust because it's causing CI failures. Also, the fix should go into the swift-3.0-branch once it's merged into master.

https://ci.swift.org/job/oss-swift-incremental-RA-linux-ubuntu-15_10/8200/console

============================================================================
Testsuite summary for libpthread_workqueue 0.9.4

1. TOTAL: 3

2. PASS: 3

3. SKIP: 0

4. XFAIL: 0

5. FAIL: 0

6. XPASS: 0

7. ERROR: 0

   make[3]: Leaving directory '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/libdispatch-linux-x86_64/libpwq'
   make[2]: Leaving directory '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/libdispatch-linux-x86_64/libpwq'
   make[1]: Leaving directory '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/libdispatch-linux-x86_64/libpwq'
   Making check in libkqueue
   make[1]: Entering directory '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/libdispatch-linux-x86_64/libkqueue'
   make kqtest
   make[2]: Entering directory '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/libdispatch-linux-x86_64/libkqueue'
   /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/llvm-linux-x86_64/bin/clang -DHAVE_CONFIG_H -I. -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue -g -O0 -Wall -Werror -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/include -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/test -I. -g -O2 -MT test/kqtest-main.o -MD -MP -MF test/.deps/kqtest-main.Tpo -c -o test/kqtest-main.o test -f 'test/main.c' || echo '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/'test/main.c
   mv -f test/.deps/kqtest-main.Tpo test/.deps/kqtest-main.Po
   /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/llvm-linux-x86_64/bin/clang -DHAVE_CONFIG_H -I. -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue -g -O0 -Wall -Werror -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/include -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/test -I. -g -O2 -MT test/kqtest-kevent.o -MD -MP -MF test/.deps/kqtest-kevent.Tpo -c -o test/kqtest-kevent.o test -f 'test/kevent.c' || echo '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/'test/kevent.c
   mv -f test/.deps/kqtest-kevent.Tpo test/.deps/kqtest-kevent.Po
   /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/llvm-linux-x86_64/bin/clang -DHAVE_CONFIG_H -I. -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue -g -O0 -Wall -Werror -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/include -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/test -I. -g -O2 -MT test/kqtest-test.o -MD -MP -MF test/.deps/kqtest-test.Tpo -c -o test/kqtest-test.o test -f 'test/test.c' || echo '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/'test/test.c
   mv -f test/.deps/kqtest-test.Tpo test/.deps/kqtest-test.Po
   /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/llvm-linux-x86_64/bin/clang -DHAVE_CONFIG_H -I. -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue -g -O0 -Wall -Werror -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/include -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/test -I. -g -O2 -MT test/kqtest-proc.o -MD -MP -MF test/.deps/kqtest-proc.Tpo -c -o test/kqtest-proc.o test -f 'test/proc.c' || echo '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/'test/proc.c
   mv -f test/.deps/kqtest-proc.Tpo test/.deps/kqtest-proc.Po
   /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/llvm-linux-x86_64/bin/clang -DHAVE_CONFIG_H -I. -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue -g -O0 -Wall -Werror -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/include -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/test -I. -g -O2 -MT test/kqtest-read.o -MD -MP -MF test/.deps/kqtest-read.Tpo -c -o test/kqtest-read.o test -f 'test/read.c' || echo '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/'test/read.c
   mv -f test/.deps/kqtest-read.Tpo test/.deps/kqtest-read.Po
   /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/llvm-linux-x86_64/bin/clang -DHAVE_CONFIG_H -I. -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue -g -O0 -Wall -Werror -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/include -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/test -I. -g -O2 -MT test/kqtest-signal.o -MD -MP -MF test/.deps/kqtest-signal.Tpo -c -o test/kqtest-signal.o test -f 'test/signal.c' || echo '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/'test/signal.c
   mv -f test/.deps/kqtest-signal.Tpo test/.deps/kqtest-signal.Po
   /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/llvm-linux-x86_64/bin/clang -DHAVE_CONFIG_H -I. -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue -g -O0 -Wall -Werror -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/include -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/test -I. -g -O2 -MT test/kqtest-timer.o -MD -MP -MF test/.deps/kqtest-timer.Tpo -c -o test/kqtest-timer.o test -f 'test/timer.c' || echo '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/'test/timer.c
   mv -f test/.deps/kqtest-timer.Tpo test/.deps/kqtest-timer.Po
   /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/llvm-linux-x86_64/bin/clang -DHAVE_CONFIG_H -I. -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue -g -O0 -Wall -Werror -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/include -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/test -I. -g -O2 -MT test/kqtest-vnode.o -MD -MP -MF test/.deps/kqtest-vnode.Tpo -c -o test/kqtest-vnode.o test -f 'test/vnode.c' || echo '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/'test/vnode.c
   mv -f test/.deps/kqtest-vnode.Tpo test/.deps/kqtest-vnode.Po
   /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/llvm-linux-x86_64/bin/clang -DHAVE_CONFIG_H -I. -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue -g -O0 -Wall -Werror -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/include -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/test -I. -g -O2 -MT test/kqtest-user.o -MD -MP -MF test/.deps/kqtest-user.Tpo -c -o test/kqtest-user.o test -f 'test/user.c' || echo '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/'test/user.c
   mv -f test/.deps/kqtest-user.Tpo test/.deps/kqtest-user.Po
   /bin/bash ./libtool --tag=CC --mode=link /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/llvm-linux-x86_64/bin/clang -g -O0 -Wall -Werror -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/include -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/test -I. -g -O2 -o kqtest test/kqtest-main.o test/kqtest-kevent.o test/kqtest-test.o test/kqtest-proc.o test/kqtest-read.o test/kqtest-signal.o test/kqtest-timer.o test/kqtest-vnode.o test/kqtest-user.o -lpthread -lrt libkqueue.la
   libtool: link: /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/llvm-linux-x86_64/bin/clang -g -O0 -Wall -Werror -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/include -I/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift-corelibs-libdispatch/libkqueue/test -I. -g -O2 -o kqtest test/kqtest-main.o test/kqtest-kevent.o test/kqtest-test.o test/kqtest-proc.o test/kqtest-read.o test/kqtest-signal.o test/kqtest-timer.o test/kqtest-vnode.o test/kqtest-user.o ./.libs/libkqueue.a -lpthread -lrt
   make[2]: Leaving directory '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/libdispatch-linux-x86_64/libkqueue'
   make check-TESTS
   make[2]: Entering directory '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/libdispatch-linux-x86_64/libkqueue'
   make[3]: Entering directory '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/libdispatch-linux-x86_64/libkqueue'
   FAIL: kqtest

   libkqueue 2.1.0: ./test-suite.log

8. TOTAL: 1

9. PASS: 0

10. SKIP: 0

11. XFAIL: 0

12. FAIL: 1

13. XPASS: 0

14. ERROR: 0

.. contents:: :depth: 2

FAIL: kqtest

bind-1: Address already in use
Running 1 iterations
1: test_peer_close_detection()
2: test_kqueue()
3: test_kevent()
4: test_ev_receipt()
unable to bind to port 23456
*** TEST FAILED ***
FAIL kqtest (exit status: 1)

============================================================================
Testsuite summary for libkqueue 2.1.0

1. TOTAL: 1
2. PASS: 0
3. SKIP: 0
4. XFAIL: 0
5. FAIL: 1
6. XPASS: 0

7. ERROR: 0

   See ./test-suite.log

   Makefile:1235: recipe for target 'test-suite.log' failed
   make[3]: *** [test-suite.log] Error 1
   make[3]: Leaving directory '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/libdispatch-linux-x86_64/libkqueue'
   Makefile:1341: recipe for target 'check-TESTS' failed
   make[2]: *** [check-TESTS] Error 2
   make[2]: Leaving directory '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/libdispatch-linux-x86_64/libkqueue'
   Makefile:1547: recipe for target 'check-am' failed
   make[1]: *** [check-am] Error 2
   make[1]: Leaving directory '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/libdispatch-linux-x86_64/libkqueue'
   Makefile:456: recipe for target 'check-recursive' failed
   make: *** [check-recursive] Error 1
   /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift/utils/build-script: fatal error: command terminated with a non-zero exit status 2, aborting
   /home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
   Build step 'Execute shell' marked build as failure
   [xUnit] [INFO] - Starting to record.
   [xUnit] [INFO] - Processing JUnit
   [xUnit] [INFO] - [JUnit] - 1 test report file(s) were found with the pattern 'buildbot_incremental/swift-linux-x86_64/swift-test-results/x86_64-unknown-linux-gnu/lit-tests.xml' relative to '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10' for the testing framework 'JUnit'.
   [xUnit] [INFO] - Converting '/home/buildnode/jenkins/workspace/oss-swift-incremental-RA-linux-ubuntu-15_10/buildbot_incremental/swift-linux-x86_64/swift-test-results/x86_64-unknown-linux-gnu/lit-tests.xml' .
   [xUnit] [INFO] - Check 'Failed Tests' threshold.
   [xUnit] [INFO] - Check 'Skipped Tests' threshold.
   [xUnit] [INFO] - Setting the build status to FAILURE
   [xUnit] [INFO] - Stopping recording.
   [BFA] Scanning build for known causes...
   [BFA] No failure causes found
   [BFA] Done. 0s
   Email was triggered for: Failure - Any
   Sending email for trigger: Failure - Any
   Last: 8199 SUCCESS
   Current: 8200 FAILURE
   Finished: FAILURE

[swift-ci]

Comment by Doug Coleman (JIRA)

Still seeing this one.

FAIL: kqtest

bind-1: Address already in use
Running 1 iterations
1: test_peer_close_detection()
2: test_kqueue()
3: test_kevent()
4: test_ev_receipt()
unable to bind to port 23456

• • • TEST FAILED ***
      FAIL kqtest (exit status: 1)

[swift-ci]

Comment by Doug Coleman (JIRA)

The tests should not hardcode a port. Instead, ask for a random port from the OS and pass that port to the next test to use it.

[swift-ci]

Comment by Doug Coleman (JIRA)

David, would you mind prioritizing this one so the CI stops breaking PRs? Thanks!

[swift-ci]

Comment by David Grove (JIRA)

Submitted a PR for changing the test case to the upstream project: mheily/libkqueue#31

We have to wait for that to be accepted, then update the gitsubmodule version of libkqueue used by libdispatch. There have been other upstream changes since the last time we synched, so probably don't want to pick them up until people return from vacations in January.

[swift-ci]

Comment by David Grove (JIRA)

upstream PR merged quickly. But I still am inclined to wait until January to pull the kqueue changes into Swift; not something to attempt right before the CI system gets locked down for the holiday break.

[swift-ci]

Comment by Doug Coleman (JIRA)

Thank you for fixing this!

+1 for merging it into swift because it's a self-contained test and a regression to always-failing rather than sometimes-failing would be obvious. Also, we won't have to think about it over the break!

[swift-ci]

Comment by David Grove (JIRA)

#203 updated master to the latest versions of libpwq and libkqueue. That includes the upstream fix to the kqtest for this issue.