[SR-3002] using DispatchData.enumerateBytes on Linux leaks the block

[weissi]

Previous ID	SR-3002
Radar	None
Original Reporter	@weissi
Type	Bug
Status	Resolved
Resolution	Done

Attachment: Download

Additional Detail from JIRA

Votes	0
Component/s	libdispatch
Labels	Bug, Leak, Linux
Assignee	dgrove-oss (JIRA)
Priority	Medium

md5: 323c438d5e940e5a5a3e6dc87ed9b40f

is blocked by:

• SR-2313 Add withoutActuallyEscaping

Issue Description:

Description

When using DispatchData.enumerateBytes on Linux, it leaks the block which leaks to pretty bad memory leaks...

The example program is

import Dispatch

while true {
    let d = DispatchData.empty
    d.enumerateBytes { _ in }
}

just leave that running on Linux and it'll leak a lot of memory, on Darwin (with the Apple Swift from Xcode it's fine).

If you're lazy, just run the attached test.sh with bash test.sh under Linux or Darwin and it'll create, compile, run (for 10s), and print statistics for the test program.

Numbers

See below for some numbers:

Linux

this is created with Swift from https://swift.org/builds/development/ubuntu1604/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04.tar.gz but the same problem happens with all Swift versions I have tested.

The kernel is Linux thing 4.4.0-36-generic #​55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ bash test.sh
swift source in '/tmp/swift-dispatch-mem-leak_MgQ7Lq/test.swift'
swift binary in '/tmp/swift-dispatch-mem-leak_MgQ7Lq/test'
=====
import Dispatch

while true {
    let d = DispatchData.empty
    d.enumerateBytes { _ in }
}
=====
Swift version 3.0-dev (LLVM 2f56c9717e, Clang b0df436efa, Swift 25d58f9228)
Target: x86_64-unknown-linux-gnu
=====
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
jweiss   18062  100 62.5 10309652 10253740 pts/3 R+ 18:51   0:10 ./test
test.sh: line 28: 18062 Killed                  ./test
OK

that's 10253740 kb resident in 10s, that's a 10GB leak.

Darwin

$ bash test.sh
swift source in '/tmp/swift-dispatch-mem-leak_iorCwW/test.swift'
swift binary in '/tmp/swift-dispatch-mem-leak_iorCwW/test'
=====
import Dispatch

while true {
    let d = DispatchData.empty
    d.enumerateBytes { _ in }
}
=====
Apple Swift version 3.0 (swiftlang-800.0.46.2 clang-800.0.38)
Target: x86_64-apple-macosx10.9
=====
USER       PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
johannes 34757 100.0  0.0  2445456   6832 s002  R+    6:50pm   0:09.99 ./test
test.sh: line 28: 34757 Killed: 9               ./test
OK

that's 6MB resident in 10s, much better.

Valgrind output

$ valgrind --leak-check=full /tmp/swift-dispatch-mem-leak_yqOZxx/test
==18530== Memcheck, a memory error detector
==18530== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==18530== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==18530== Command: /tmp/swift-dispatch-mem-leak_yqOZxx/test
==18530== 
^C==18530== 
==18530== Process terminating with default action of signal 2 (SIGINT)
==18530==    at 0x400C73: main (in /tmp/swift-dispatch-mem-leak_yqOZxx/test)
==18530== 
==18530== HEAP SUMMARY:
==18530==     in use at exit: 59,364,626 bytes in 1,482,300 blocks
==18530==   total heap usage: 1,482,310 allocs, 10 frees, 59,444,330 bytes allocated
==18530== 
==18530== 57 (16 direct, 41 indirect) bytes in 1 blocks are definitely lost in loss record 4 of 9
==18530==    at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18530==    by 0x515EBFA: _swift_stdlib_getUnsafeArgvArgc (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libswiftCore.so)
==18530==    by 0x507139F: globalinit_33_FD9A49A256BEB6AF7C48013347ADC3BA_func3 (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libswiftCore.so)
==18530==    by 0x5E98AD8: __pthread_once_slow (pthread_once.c:116)
==18530==    by 0x515C4A9: swift_once (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libswiftCore.so)
==18530==    by 0x400C5D: main (in /tmp/swift-dispatch-mem-leak_yqOZxx/test)
==18530== 
==18530== 192 bytes in 6 blocks are possibly lost in loss record 5 of 9
==18530==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18530==    by 0x514F9E5: swift_slowAlloc (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libswiftCore.so)
==18530==    by 0x514FA1E: _swift_allocObject_ (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libswiftCore.so)
==18530==    by 0x4102463: _TFV8Dispatch12DispatchData14enumerateBytesfT5blockFTGSRVs5UInt8_SiRSb_T__T_ (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libdispatch.so)
==18530==    by 0x400C9B: main (in /tmp/swift-dispatch-mem-leak_yqOZxx/test)
==18530== 
==18530== 240 bytes in 5 blocks are possibly lost in loss record 6 of 9
==18530==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18530==    by 0x41006D8: _Block_copy_internal (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libdispatch.so)
==18530==    by 0x41024C0: _TFV8Dispatch12DispatchData14enumerateBytesfT5blockFTGSRVs5UInt8_SiRSb_T__T_ (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libdispatch.so)
==18530==    by 0x400C9B: main (in /tmp/swift-dispatch-mem-leak_yqOZxx/test)
==18530== 
==18530== 59,291,376 (35,574,864 direct, 23,716,512 indirect) bytes in 741,143 blocks are definitely lost in loss record 9 of 9
==18530==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18530==    by 0x41006D8: _Block_copy_internal (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libdispatch.so)
==18530==    by 0x41024C0: _TFV8Dispatch12DispatchData14enumerateBytesfT5blockFTGSRVs5UInt8_SiRSb_T__T_ (in /home/jweiss/extsrc/swift-DEVELOPMENT-SNAPSHOT-2016-10-18-a-ubuntu16.04/usr/lib/swift/linux/libdispatch.so)
==18530==    by 0x400C9B: main (in /tmp/swift-dispatch-mem-leak_yqOZxx/test)
==18530== 
==18530== LEAK SUMMARY:
==18530==    definitely lost: 35,574,880 bytes in 741,144 blocks
==18530==    indirectly lost: 23,716,553 bytes in 741,142 blocks
==18530==      possibly lost: 432 bytes in 11 blocks
==18530==    still reachable: 72,761 bytes in 3 blocks
==18530==         suppressed: 0 bytes in 0 blocks
==18530== Reachable blocks (those to which a pointer was found) are not shown.
==18530== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==18530== 
==18530== For counts of detected and suppressed errors, rerun with: -v
==18530== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)

[swift-ci]

Comment by David Grove (JIRA)

I can reproduce.

The fix is that we really need to be calling CDispatch.dispatch_data_apply (not swift_dispatch_data_apply). Trying to do that however results in a Swift compiler error because it thinks the block argument being captured in the closure argument to dispatch_data_apply is going to escape. We need the functionality of https://bugs.swift.org/browse/SR-2313 to cleanly fix the leak. Will have to figure out the least ugly way to kludge around it not being available...

[swift-ci]

Comment by David Grove (JIRA)

Submitted #186 using unsafeBitCast in place of withoutActuallyEscaping as suggested by Jordan. Once the PR is merged, I will resolve this issue and create a new one to track that we need to update this code once withoutActuallyEscaping is implemented.

[swift-ci]

Comment by David Grove (JIRA)

Fix merged into Swift3 branch: #189 Leaving issue open until merged to master as well.

[swift-ci]

Comment by David Grove (JIRA)

fixed now merged to master as well.