[SR-9888] TestFoundation test suite failing while building Swift Toolchain

[swift-ci]

Previous ID	SR-9888
Radar	None
Original Reporter	stamba (JIRA User)
Type	Bug

Attachment: Download

Environment

powerpc64le

Additional Detail from JIRA

Votes	0
Component/s	Foundation
Labels	Bug
Assignee	None
Priority	Medium

md5: fd480f09b61b34fa44e58b785ed8fd5c

Issue Description:

Hi,

I have been trying to build Swift 5.0 master on PowerPC(powerpc64le) using the build-toolchain script to run the test suite as well. I had been facing issues with tests in lldb suite(#SR-9808). After skipping the failing lldb tests, I am able to complete the libdispatch test suite. However, facing the following issues in TestFoundation test suite:-

[257/257][100%][91.650s] Running utility command for TestFoundation

• • • Running tests for foundation

• • • • • • test ---

+ /usr/bin/cmake --build /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le – -j8 test

[0/1][ 0%][0.000s] Running tests...

Test project /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le

Start 1: TestFoundation

1/1 Test #1: TestFoundation ...................***Failed 0.00 sec

/root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/TestFoundation/TestFoundation: error while loading shared libraries: libicui18nswift.so.61: cannot open shared object file: No such file or directory

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 0.00 sec

The following tests FAILED:

1 - TestFoundation (Failed)

Errors while running CTest

FAILED: CMakeFiles/test.util

cd /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le && /usr/bin/ctest --force-new-ctest-process

ninja: build stopped: subcommand failed.

Building the standard library for: swift-stdlib-linux-powerpc64le

Running Swift tests for: check-swift-all-linux-powerpc64le check-swift-all-optimize-linux-powerpc64le

./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

[belkadan]

I'm pretty sure we've seen this missing ICU before. @compnerd?

[spevans]

1. Check that the ICU libraries are actually being built and copied into the swift build directory:

Should look something like:

$ find build/buildbot_linux/swift-linux-x86_64/ -name 'libicu*.so'
build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libicudataswift.so
build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libicui18nswift.so
build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libicutestswift.so
build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libicuucswift.so
build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libicutuswift.so

2. It looks like you are building as root? I would avoid this and instead use a regular user id as they may cause some unexpected issue.

[swift-ci]

Comment by Sarvesh Tamba (JIRA)

1.) Following are the ICU libraries built across the swift build directory:-

root@37e196fab451:~/swift-source# find build/buildbot_linux/ -name 'libicu*.so'
build/buildbot_linux/libicu-linux-powerpc64le/stubdata/libicudataswift.so
build/buildbot_linux/libicu-linux-powerpc64le/tools/ctestfw/libicutestswift.so
build/buildbot_linux/libicu-linux-powerpc64le/lib/libicuucswift.so
build/buildbot_linux/libicu-linux-powerpc64le/lib/libicudataswift.so
build/buildbot_linux/libicu-linux-powerpc64le/lib/libicutuswift.so
build/buildbot_linux/libicu-linux-powerpc64le/lib/libicui18nswift.so
build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le/libicuucswift.so
build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le/libicudataswift.so
build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le/libicutestswift.so
build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le/libicutuswift.so
build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le/libicui18nswift.so

2.) Trying to build swift as non-root user, will update here once done.

[swift-ci]

Comment by Sarvesh Tamba (JIRA)

@spevans -

I tried building swift as a non-root user, facing the same issue as in root user run:-

[257/257][100%][102.676s] Running utility command for TestFoundation

• • • Running tests for foundation ---

    • test ---

      • /usr/bin/cmake --build /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le – -j7 test
        [0/1][ 0%][0.000s] Running tests...
        Test project /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le
        Start 1: TestFoundation
        1/1 Test Fixing whitespace formatting #1: TestFoundation ...................***Failed 0.01 sec
        /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/TestFoundation/TestFoundation: error while loading shared libraries: libicui18nswift.so.61: cannot open shared object file: No such file or directory

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 0.01 sec

The following tests FAILED:
1 - TestFoundation (Failed)
Errors while running CTest
FAILED: CMakeFiles/test.util
cd /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le && /usr/bin/ctest --force-new-ctest-process
ninja: build stopped: subcommand failed.
Building the standard library for: swift-stdlib-linux-powerpc64le
Running Swift tests for: check-swift-all-linux-powerpc64le check-swift-all-optimize-linux-powerpc64le
./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
sar@e2641fd15168:~$

Please suggest if I am missing anything.

[swift-ci]

Comment by Sarvesh Tamba (JIRA)

1.) Tried setting ICU_ROOT & DICU_ROOT in "swift/utils/build-script-impl". However faced the same errors as before - "error while loading shared libraries: libicui18nswift.so.61: cannot open shared object file: No such file or directory". Correct me if this is not correct approach to set the two variables.
2.) Used the 'llvm-readelf' utility to find details about the libraries linked with TestFoundation binary:-

sar@e2641fd15168:~$ /home/sar/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/llvm-readelf -dynamic-table /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/TestFoundation/TestFoundation
DynamicSection [ (40 entries)
Tag Type Name/Value
0x0000000000000003 PLTGOT 0x7d9eb8
0x0000000000000002 PLTRELSZ 41448 (bytes)
0x0000000000000017 JMPREL 0x86b80
0x0000000000000014 PLTREL RELA
0x0000000000000007 RELA 0x32118
0x0000000000000008 RELASZ 346728 (bytes)
0x0000000000000009 RELAENT 24 (bytes)
0x000000006ffffff9 RELACOUNT 5957
0x0000000000000015 DEBUG 0x0
0x0000000070000000 PPC64_GLINK 0x6457d0
0x0000000000000006 SYMTAB 0x290
0x000000000000000b SYMENT 24 (bytes)
0x0000000000000005 STRTAB 0x103b8
0x000000000000000a STRSZ 132946 (bytes)
0x000000006ffffef5 GNU_HASH 0x30b10
0x0000000000000001 NEEDED Shared library: [libswiftCore.so]
0x0000000000000001 NEEDED Shared library: [libpthread.so.0]
0x0000000000000001 NEEDED Shared library: [libutil.so.1]
0x0000000000000001 NEEDED Shared library: [libdl.so.2]
0x0000000000000001 NEEDED Shared library: [libm.so.6]
0x0000000000000001 NEEDED Shared library: [libdispatch.so]
0x0000000000000001 NEEDED Shared library: [libswiftDispatch.so]
0x0000000000000001 NEEDED Shared library: [libBlocksRuntime.so]
0x0000000000000001 NEEDED Shared library: [libXCTest.so]
0x0000000000000001 NEEDED Shared library: [libswiftGlibc.so]
0x0000000000000001 NEEDED Shared library: [libFoundation.so]
0x0000000000000001 NEEDED Shared library: [libstdc++.so.6]
0x0000000000000001 NEEDED Shared library: [libgcc_s.so.1]
0x0000000000000001 NEEDED Shared library: [libc.so.6]
0x000000000000000c INIT 0x90d80
0x000000000000000d FINI 0x6472f0
0x000000000000001a FINI_ARRAY 0x792220
0x000000000000001c FINI_ARRAYSZ 8 (bytes)
0x0000000000000019 INIT_ARRAY 0x792228
0x000000000000001b INIT_ARRAYSZ 16 (bytes)
0x000000000000001d RUNPATH /home/sar/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux:/home/sar/swift-source/build/buildbot_linux/libdispatch-linux-powerpc64le/src
0x000000006ffffff0 VERSYM 0x30b48
0x000000006ffffffe VERNEED 0x320b8
0x000000006fffffff VERNEEDNUM 3
0x0000000000000000 NULL 0x0
]
Figured out that the directories mentioned in RUNPATH did not have the required library - libicui18nswift.so.61

sar@e2641fd15168:~$ ll /home/sar/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux
total 16004
drwxr-xr-x 3 sar sar 4096 Feb 13 07:11 ./
drwxr-xr-x 5 sar sar 4096 Feb 13 07:00 ../
-rwxr-xr-x 1 sar sar 8805520 Feb 13 07:11 libswiftCore.so*
-rwxr-xr-x 1 sar sar 84192 Feb 13 07:11 libswiftGlibc.so*
rw-rr- 1 sar sar 4674 Feb 13 06:57 libswiftImageInspectionShared.a
rw-rr- 1 sar sar 1100452 Feb 13 06:57 libswiftReflection.a
-rwxr-xr-x 1 sar sar 688424 Feb 13 06:57 libswiftRemoteMirror.so*
-rwxr-xr-x 1 sar sar 72576 Feb 13 07:11 libswiftRuntimeUnittest.so*
-rwxr-xr-x 1 sar sar 71280 Feb 13 07:11 libswiftSIMDOperators.so*
-rwxr-xr-x 1 sar sar 3534296 Feb 13 07:11 libswiftStdlibCollectionUnittest.so*
-rwxr-xr-x 1 sar sar 237328 Feb 13 07:11 libswiftStdlibUnicodeUnittest.so*
-rwxr-xr-x 1 sar sar 961320 Feb 13 07:11 libswiftStdlibUnittest.so*
-rwxr-xr-x 1 sar sar 529968 Feb 13 07:11 libswiftSwiftOnoneSupport.so*
-rwxr-xr-x 1 sar sar 97056 Feb 13 07:11 libswiftSwiftPrivate.so*
-rwxr-xr-x 1 sar sar 81360 Feb 13 07:11 libswiftSwiftPrivateLibcExtras.so*
-rwxr-xr-x 1 sar sar 82736 Feb 13 07:11 libswiftSwiftPrivateThreadExtras.so*
drwxr-xr-x 4 sar sar 4096 Feb 13 09:46 powerpc64le/
However, the /home/sar/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le has all the required libraries. I tried setting RUNPATH through various files which referenced RUNPATH, but it did not help much.

Finally I manually copied the libraries in the directory mentioned in the RUNPATH and it worked. The TestFoundation test suite was executed, though it failed for 4 test cases since I was running as a root user. Running as normal non root user worked and all the test cases for TestFoundation passed successfully:-

[256/257][ 99%][98.957s] Generating TestFoundation.dir/TestFoundation
[257/257][100%][99.479s] Running utility command for TestFoundation

• • • Running tests for foundation ---

    • test ---

/usr/bin/cmake --build /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le – -j7 test
[0/1][ 0%][0.000s] Running tests...
Test project /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le
Start 1: TestFoundation
1/1 Test #1: TestFoundation ................... Passed 53.48 sec
100% tests passed, 0 tests failed out of 1

Total Test time (real) = 53.49 sec

• • test finished --

    • Finished tests for foundation ---

[swift-ci]

Comment by Sarvesh Tamba (JIRA)

Used the 'llvm-readelf' utility to find details about the libraries linked with lldb binary and realised that the directories mentioned in RUNPATH might not have the required libraries.

Similar to reasons mentioned in my earlier issue "https://forums.swift.org/t/testfoundation-test-suite-failing-while-building-swift-toolchain/20279/6", I manually copied the libraries and it worked. The lldb test suite was executed, though it fails now for only 7 test cases:-
Failing Tests (7):
LLDB :: SwiftREPL/ErrorReturn.test
LLDB :: tools/lldb-mi/symbol/symbol-list-lines.test
lldb-Suite :: lang/swift/dwarfimporter/C/TestSwiftDWARFImporterC.py
lldb-Suite :: lang/swift/reference_storage_types/TestSwiftReferenceStorageTypes.py
lldb-Suite :: lang/swift/repl_in_c/TestSwiftReplInC.py
lldb-Suite :: lang/swift/unknown_self/TestSwiftUnknownSelf.py
lldb-Suite :: lang/swift/variables/swift/TestSwiftTypes.py

[swift-ci]

Comment by Sarvesh Tamba (JIRA)

Attached to this issue are the details of the final 7 failing lldb tests.