[Tools] mbed Online Build System support and improvements #2180

screamerbg · 2016-07-15T23:04:18Z

added/improved chroot support globally
added RESPONSE_FILES flag to support optional response files (on linux the cmd param length is 2 megabytes). Default True
added unified handling for archive and link response file (similar to includes)
added COMPILE_C_AS_CPP flag to support compiling of c files as cpp. Default False
added mbedToolchain.init() for post init hooks
added support to identify compiler warning/error column (supports ARMCC, GCC and IAR). Errors/warnings now report file@line,col
added global TOOLCHAIN_PATHS which allows overriding/changing of the toolchain paths. Also simplified ARM-related paths
added target.json to mbed library release (by @0xc0170)
added caching to mbedToolchain.need_update() to reduce IO hits
improved run_cmd() performance by removing unnecessary check about the command being executed (should be checked once in the relevant toolchain instead)
migrated compile_worker() to utils.py for lightweight thread initialization
removed remnants of Goanna support (should be reimplemented as hooks to compile/link/archive instead)
fixes for Python 2.7 compatibility (by @0xc0170)
fixes for Exporters (by @0xc0170)

theotherjimmy · 2016-07-15T23:21:09Z

tools/toolchains/__init__.py

-
+
+            if self.stat_cache.has_key(d):
+                if self.stat_cache[d] >= target_mod_time:


should be:

if self.stat_cache.has_key(d) and self.stat_cache[d] >= target_mod_time:

unless you are planning on adding an else case to the second if.

theotherjimmy · 2016-07-15T23:28:34Z

tools/export/exporters.py

        config = Config(self.target, prj_paths)
        for src in ['lib', 'src']:
-            resources = reduce(add, [self.__scan_and_copy(join(path, src), trg_path) for path in prj_paths])


Remove the

from operators import add

from the top of this file. It's no longer needed/wanted.

screamerbg · 2016-07-15T23:44:12Z

@mbed-bot: TEST

HOST_OSES=Windows
BUILD_TOOLCHAINS=GCC_ARM,ARM,IAR
TARGETS=K64F,LPC1768,NUCLEO_F411RE

theotherjimmy · 2016-07-15T23:46:59Z

I want to test it before I give it 👍 , but these changes look fine to me.

screamerbg · 2016-07-16T09:02:32Z

@mbed-bot: TEST

HOST_OSES=windows
BUILD_TOOLCHAINS=GCC_ARM,ARM,IAR
TARGETS=K64F,LPC1768

mbed-bot · 2016-07-16T20:40:08Z

[Build ${MBED_BUILD_ID}]
FAILURE: Something went wrong when building and testing.

mbed-bot · 2016-07-16T20:40:40Z

[Build ${MBED_BUILD_ID}]
FAILURE: Something went wrong when building and testing.

bridadan · 2016-07-16T20:40:51Z

@mbed-bot: TEST

HOST_OSES=windows
BUILD_TOOLCHAINS=GCC_ARM,ARM,IAR
TARGETS=K64F,LPC1768

mbed-bot · 2016-07-16T21:15:46Z

[Build 638]
FAILURE: Something went wrong when building and testing.

bridadan · 2016-07-16T21:35:31Z

Looks like there's build failures for K64F with the RTOS for toolchains ARM and IAR:

Building library RTX (K64F, ARM)
Scan: rtx
Scan: mbed
Scan: rtos
Compile: HAL_CM.c
[Warning] rt_HAL_CM.h@209,0:  #3731-D: intrinsic is deprecated
[Warning] rt_HAL_CM.h@212,0:  #3731-D: intrinsic is deprecated
[Warning] rt_HAL_CM.h@214,0:  #3731-D: intrinsic is deprecated
[Warning] rt_HAL_CM.h@216,0:  #3731-D: intrinsic is deprecated
Compile: RTX_Conf_CM.c
Compile: HAL_CM4.c
[Warning] rt_HAL_CM.h@209,0:  #3731-D: intrinsic is deprecated
[Warning] rt_HAL_CM.h@212,0:  #3731-D: intrinsic is deprecated
[Warning] rt_HAL_CM.h@214,0:  #3731-D: intrinsic is deprecated
[Warning] rt_HAL_CM.h@216,0:  #3731-D: intrinsic is deprecated
[Error] HAL_CM4.c@167,0: A1854E: Unknown opcode 'VSTMDBEQ', maybe wrong target CPU?
[Error] HAL_CM4.c@187,0: A1854E: Unknown opcode 'VLDMIANE', maybe wrong target CPU?
[Error] HAL_CM4.c@250,0: A1854E: Unknown opcode 'VSTMDBEQ', maybe wrong target CPU?
[Error] HAL_CM4.c@269,0: A1854E: Unknown opcode 'VLDMIANE', maybe wrong target CPU?

screamerbg · 2016-07-16T21:42:01Z

@bridadan Hold on 2min

screamerbg · 2016-07-16T23:36:19Z

@mbed-bot: TEST

HOST_OSES=windows
BUILD_TOOLCHAINS=GCC_ARM,ARM,IAR
TARGETS=K64F,LPC1768,NUCLEO_F411RE

mbed-bot · 2016-07-17T00:12:14Z

[Build 642]
SUCCESS: Building succeeded and tests were run! Be sure to check the test results

0xc0170 · 2016-07-18T07:50:06Z

Some of this changes are part of the #2175 (the release PR).

This patch (=1 commit) should be split (based on the list in the first message, there would be set of commits).

screamerbg · 2016-07-18T12:22:00Z

@0xc0170 Agreed that the patch is big, but unfortunately the intermediate commits contained integration code which was later removed. Additionally some of the commits contained information about the online build system set up, which should rather stay in mbed's kitchen.

Due to time constraints I don't think I'll be able to split it into separate patches as you suggested. I'd rather use the time to document which methods are used by the online build system, which might break it if changed.

bogdanm · 2016-07-18T13:15:41Z

tools/export/uvision4.py

@@ -98,7 +95,7 @@ def generate(self, progen_build=False):
                project_data['common']['macros'].pop(i)
            i += 1
        project_data['common']['macros'].append('__ASSERT_MSG')
-        project_data['common']['build_dir'] = join(project_data['common']['build_dir'], 'uvision4')
+        project_data['common']['build_dir'] = project_data['common']['build_dir'] + '\\' + 'uvision4'


Why \\ instead of join ?

This change is part of the release branch (as I noted above). this will get rebased and those changes will be removed. The reason is that for keil it needs to be windows paths. It was join, but then keil reported an error if I tested this in the online compiler)

I think @0xc0170 stumbled upon bug in uVision that '/' is not accepted as valid path separator.

bogdanm · 2016-07-18T13:19:20Z

This kind of patch (that contains unrelated changes in a single commit) is hard to follow and understand. There are some changes (for example https://github.com/mbedmicro/mbed/pull/2180/files#diff-486f048c4c5775670840b518bf8b0ad5R630) that are not commented and that I don't understand. I understand we're in a hurry, but I personally don't feel ready to accept this patch.

bogdanm · 2016-07-19T08:34:44Z

#2175 has been merged, can you please rebase?

screamerbg · 2016-07-19T10:08:52Z

@bogdanm now rebased. Reduced from 18 to 12 files :)

@0xc0170

* added/improved global chroot support * added RESPONSE_FILES flag to support optional response files (on linux the cmd param length is 2 megabytes). Default True * added unified handling for archive and link response file (similar to includes) * added COMPILE_C_AS_CPP flag to support compiling of c files as cpp. Default False * added mbedToolchain.init() for post __init__ hooks * added caching to mbedToolchain.need_update() to reduce IO hits * added support to identify compiler warning/error column (supports ARMCC, GCC and IAR). Errors/warnings now report file@line,col * added global TOOLCHAIN_PATHS which allows overriding/changing of the toolchain paths. Also simplified ARM-related paths * added target.json to mbed library release (by @0xc0170)* migrated compile_worker() to utils.py for lightweight thread initialization * improved run_cmd() performance by removing unnecessary check about the command being executed (should be checked once in the relevant toolchain instead) * removed remnants of Goanna support (should be reimplemented as hooks to compile/link/archive instead) * fixes for Python 2.7 compatibility (by @0xc0170) * fixes for Exporters (by @0xc0170)

screamerbg · 2016-07-19T10:16:53Z

@mbed-bot: TEST

HOST_OSES=windows
BUILD_TOOLCHAINS=GCC_ARM,ARM,IAR
TARGETS=K64F,LPC1768,NUCLEO_F411RE

mbed-bot · 2016-07-19T11:12:51Z

[Build 652]
SUCCESS: Building succeeded and tests were run! Be sure to check the test results

mbed-bot · 2016-07-19T11:47:29Z

[Build 653]
SUCCESS: Building succeeded and tests were run! Be sure to check the test results

bogdanm · 2016-07-19T12:30:30Z

tools/targets.py

@@ -61,6 +61,9 @@ class Target:
    # need to be computed differently than regular attributes
    __cumulative_attributes = ['extra_labels', 'macros', 'device_has', 'features']

+    # List of targets that were added dynamically using "add_py_targets" (see below)
+    __py_targets = set()


__py_targets is no longer needed. Please remove it.

bogdanm · 2016-07-19T13:30:25Z

There are still loads of changes that I don't understand here, but the tests pass, so I'm going to merge this in. Keeping a close eye on it in case it breaks things.

screamerbg force-pushed the online-build-system branch 2 times, most recently from cb18774 to 3906136 Compare July 15, 2016 23:17

theotherjimmy reviewed Jul 15, 2016
View reviewed changes

screamerbg force-pushed the online-build-system branch from 3906136 to f31ea23 Compare July 15, 2016 23:22

theotherjimmy reviewed Jul 15, 2016
View reviewed changes

screamerbg force-pushed the online-build-system branch from f31ea23 to 7307f64 Compare July 15, 2016 23:35

screamerbg changed the title ~~Patches related mbed online build system support~~ mbed Online Build System support Jul 15, 2016

screamerbg changed the title ~~mbed Online Build System support~~ mbed Online Build System support and improvements Jul 15, 2016

screamerbg changed the title ~~mbed Online Build System support and improvements~~ [Tools] mbed Online Build System support and improvements Jul 15, 2016

screamerbg force-pushed the online-build-system branch from 7307f64 to 5d3c973 Compare July 16, 2016 17:18

screamerbg force-pushed the online-build-system branch 3 times, most recently from 54f500a to a36c8d1 Compare July 16, 2016 23:36

bogdanm reviewed Jul 18, 2016
View reviewed changes

screamerbg force-pushed the online-build-system branch from ec60314 to 782c933 Compare July 19, 2016 10:06

screamerbg added 2 commits July 19, 2016 11:14

Document code and mark which methods are used by the online build system

ad87f9d

screamerbg force-pushed the online-build-system branch from 782c933 to ad87f9d Compare July 19, 2016 10:16

bogdanm reviewed Jul 19, 2016
View reviewed changes

bogdanm merged commit fd757d3 into master Jul 19, 2016

bogdanm deleted the online-build-system branch July 19, 2016 13:30



		if self.stat_cache.has_key(d):
		if self.stat_cache[d] >= target_mod_time:

[Tools] mbed Online Build System support and improvements #2180

[Tools] mbed Online Build System support and improvements #2180

Uh oh!

Conversation

screamerbg commented Jul 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theotherjimmy Jul 15, 2016

Choose a reason for hiding this comment

Uh oh!

theotherjimmy Jul 15, 2016

Choose a reason for hiding this comment

Uh oh!

screamerbg commented Jul 15, 2016

Uh oh!

theotherjimmy commented Jul 15, 2016

Uh oh!

screamerbg commented Jul 16, 2016

Uh oh!

mbed-bot commented Jul 16, 2016

Uh oh!

mbed-bot commented Jul 16, 2016

Uh oh!

bridadan commented Jul 16, 2016

Uh oh!

mbed-bot commented Jul 16, 2016

Uh oh!

bridadan commented Jul 16, 2016

Uh oh!

screamerbg commented Jul 16, 2016

Uh oh!

screamerbg commented Jul 16, 2016

Uh oh!

mbed-bot commented Jul 17, 2016

Uh oh!

0xc0170 commented Jul 18, 2016

Uh oh!

screamerbg commented Jul 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bogdanm Jul 18, 2016

Choose a reason for hiding this comment

Uh oh!

0xc0170 Jul 18, 2016

Choose a reason for hiding this comment

Uh oh!

screamerbg Jul 18, 2016

Choose a reason for hiding this comment

Uh oh!

bogdanm commented Jul 18, 2016

Uh oh!

bogdanm commented Jul 19, 2016

Uh oh!

screamerbg commented Jul 19, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

screamerbg commented Jul 19, 2016

Uh oh!

mbed-bot commented Jul 19, 2016

Uh oh!

mbed-bot commented Jul 19, 2016

Uh oh!

bogdanm Jul 19, 2016

Choose a reason for hiding this comment

Uh oh!

bogdanm commented Jul 19, 2016

Uh oh!

Uh oh!

screamerbg commented Jul 15, 2016 •

edited

Loading

screamerbg commented Jul 18, 2016 •

edited

Loading

screamerbg commented Jul 19, 2016 •

edited

Loading