Refactor export subsystem #2245

theotherjimmy · 2016-07-25T21:19:41Z

Makes several broad changes:

removes dead code that dealt with the online build system
replaces export function with a much simpler one that:
- does not copy any sources
- the zip file hits the disk (only offline)
- the mbed_config.h hits the disk
- the project files hit the disk
- nothing else hits the disk
exporters use Resource object scanned with a toolchain
progen exporters don't optionally build a project instead they have a
build function that may be called afterwards
much of the code passes pylint (have a score of 9 or above):
- project.py
- project_api.py
- export/__init__.py
- export/exporters.py
- test/export/build_test.py

sg- · 2016-07-26T03:48:54Z

tools/export/__init__.py

-limitations under the License.
-"""
+# mbed SDK
+# Copyright (c) 2011-2013 ARM Limited


copyright needs an update

sg- · 2016-07-26T05:26:28Z

linking to #2202

0xc0170 · 2016-07-26T08:36:44Z

tools/export/exporters.py

+        export_dir    - the directory of the exported project files
+        project_name  - the name of the project
+        toolchain     - an instance of class toolchain
+        extra_symbols - a list of extra macros for the toolchain


extra_symbols is keyword argument

Yes it is. I'll update the comment. Thanks for the catch.

0xc0170 · 2016-07-26T09:46:15Z

Looks good in general, I wonder why this was squashed to one commit, if this is just a preview or it is intended to be integrated ? I see many logical changes here that shall have some more documentation and provided as separate commits. What do you think?

What is the speed optimization ? how much does this improve? I am interested in does not copy any sources how much that took.

theotherjimmy · 2016-07-26T14:19:31Z

What is the speed optimization ? how much does this improve? I am interested in does not copy any sources how much that took.

On my machine exporting from the previous exporters took ~5 seconds. exporting with this branch takes ~0.1 seconds.

theotherjimmy · 2016-07-26T14:21:38Z

Looks good in general, I wonder why this was squashed to one commit, if this is just a preview or it is intended to be integrated ? I see many logical changes here that shall have some more documentation and provided as separate commits. What do you think?

We did development as a branch without regard for history, so squashing removed ~ 10 merge commits. We could extract the logical changes into a clean history, but that would take some time.

sg- · 2016-07-27T01:25:50Z

@screamerbg Please review

screamerbg · 2016-07-27T12:35:31Z

LGТМ. Note that this will require some integration effort with the website.

0xc0170 · 2016-07-29T06:46:25Z

What's the status of this PR ? does it depend on any progen feature?

theotherjimmy · 2016-07-29T15:25:21Z

does it depend on any progen feature?

Yes. It depends on exporting get_tool_template.

What's the status of this PR?

I did a bit of integration with the website yesterday, and found a few bugs. The website integration is robust now, So I think this could be merged soon.

theotherjimmy · 2016-07-29T15:27:02Z

But first I have to rebase.

bridadan · 2016-07-29T18:05:29Z

/morph export-build

mbed-bot · 2016-07-29T18:32:23Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph export-build

bridadan · 2016-07-29T18:38:17Z

Looks like plenty of targets exported and built correctly, but some did not:

13:24:16 Successful: 
13:24:16   * ARCH_PRO::iar  MBED_BLINKY
13:24:16   * DISCO_F429ZI::iar  MBED_BLINKY
13:24:16   * DISCO_F746NG::iar  MBED_BLINKY
13:24:16   * DISCO_L476VG::iar  MBED_BLINKY
13:24:16   * K22F::iar  MBED_BLINKY
13:24:16   * K64F::iar  MBED_BLINKY
13:24:16   * KL25Z::iar MBED_BLINKY
13:24:16   * KL46Z::iar MBED_BLINKY
13:24:16   * LPC1768::iar   MBED_BLINKY
13:24:16   * LPC4088::iar   MBED_BLINKY
13:24:16   * LPC4088_DM::iar    MBED_BLINKY
13:24:16   * MAX32600MBED::iar  MBED_BLINKY
13:24:16   * MAXWSNENV::iar MBED_BLINKY
13:24:16   * MOTE_L152RC::iar   MBED_BLINKY
13:24:16   * MTS_DRAGONFLY_F411RE::iar  MBED_BLINKY
13:24:16   * MTS_MDOT_F411RE::iar   MBED_BLINKY
13:24:16   * NRF51_DK::iar  MBED_BLINKY
13:24:16   * NUCLEO_F070RB::iar MBED_BLINKY
13:24:16   * NUCLEO_F072RB::iar MBED_BLINKY
13:24:16   * NUCLEO_F091RC::iar MBED_BLINKY
13:24:16   * NUCLEO_F103RB::iar MBED_BLINKY
13:24:16   * NUCLEO_F303RE::iar MBED_BLINKY
13:24:16   * NUCLEO_F401RE::iar MBED_BLINKY
13:24:16   * NUCLEO_F411RE::iar MBED_BLINKY
13:24:16   * NUCLEO_F429ZI::iar MBED_BLINKY
13:24:16   * NUCLEO_F446RE::iar MBED_BLINKY
13:24:16   * NUCLEO_F746ZG::iar MBED_BLINKY
13:24:16   * NUCLEO_L152RE::iar MBED_BLINKY
13:24:16   * NUCLEO_L476RG::iar MBED_BLINKY
13:24:16   * RZ_A1H::iar    MBED_BLINKY
13:24:16   * UBLOX_C027::iar    MBED_BLINKY
13:24:16 Failed: 
13:24:16   * DISCO_F469NI::iar  MBED_BLINKY
13:24:16   * EFM32PG_STK3401::iar   MBED_BLINKY
13:24:16   * NUCLEO_F410RB::iar MBED_BLINKY
13:24:16   * NUCLEO_F767ZI::iar MBED_BLINKY
13:24:16   * NUCLEO_L073RZ::iar MBED_BLINKY
13:24:16   * NUCLEO_L432KC::iar MBED_BLINKY

13:32:12 Successful: 
13:32:12   * ARCH_PRO::uvision5 MBED_BLINKY
13:32:12   * DISCO_F429ZI::uvision5 MBED_BLINKY
13:32:12   * DISCO_F469NI::uvision5 MBED_BLINKY
13:32:12   * DISCO_F746NG::uvision5 MBED_BLINKY
13:32:12   * DISCO_L476VG::uvision5 MBED_BLINKY
13:32:12   * EFM32PG_STK3401::uvision5  MBED_BLINKY
13:32:12   * K64F::uvision5 MBED_BLINKY
13:32:12   * KL25Z::uvision5    MBED_BLINKY
13:32:12   * LPC1768::uvision5  MBED_BLINKY
13:32:12   * MTS_DRAGONFLY_F411RE::uvision5 MBED_BLINKY
13:32:12   * MTS_MDOT_F411RE::uvision5  MBED_BLINKY
13:32:12   * NUCLEO_F070RB::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_F072RB::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_F091RC::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_F103RB::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_F303RE::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_F401RE::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_F410RB::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_F411RE::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_F429ZI::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_F446RE::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_F746ZG::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_L073RZ::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_L152RE::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_L432KC::uvision5    MBED_BLINKY
13:32:12   * NUCLEO_L476RG::uvision5    MBED_BLINKY
13:32:12   * UBLOX_C027::uvision5   MBED_BLINKY
13:32:12 Failed: 
13:32:12   * NRF51_DK::uvision5 MBED_BLINKY
13:32:12   * NUCLEO_F767ZI::uvision5    MBED_BLINKY

I doubt that these failures were caused by the refactor, most likely that these issues already exist?

FYI @theotherjimmy @sarahmarshy @0xc0170

sg- · 2016-07-29T19:00:31Z

@bridadan Can we get the version of each IDE that is being used for tracking in that log (in general)

bridadan · 2016-08-01T19:30:20Z

Yeah that'd be good to add at the top.

Currently we're using "IAR Embedded Workbench for ARM 7.40.3.8938" and "uVision V5.20.0.0".

bridadan · 2016-08-01T22:57:42Z

I just tried this on master (commit 4b50628) and I was able to reproduce these same issues (except the NRF51_DK passed on uvision5).

@theotherjimmy You said you were able to build all of these targets with IAR on master right? What version of IAR are you using?

it will no longer barf when: - a linker scirpt is None - an attribute that is a set it will also export the correct library include paths

Affects these ides: - Atmel Studio - Code Red (I don't think we support this) - Coide - DS-5 - E2Studio - EMblocks - KDS - Simplicity v3 - SW 4 STM32 also corrects flags usage in EMBlocks

Compatible with new c/asm/cpp flag separation.

The dict allows the user of the exporter api to specify the result directory of particular groups of scanned dirs. This will be used by the online exporters to spoof everything being in the same directory when they are not. It may also be used by tests, if they would like to export something that looks exactly like a normal project.

mbed-bot · 2016-09-06T19:25:34Z

Result: ABORTED

Your command has finished executing! Here's what you wrote!

/morph export-build

theotherjimmy · 2016-09-06T19:25:51Z

/morph export-build

mbed-bot · 2016-09-06T19:47:27Z

Result: ABORTED

Your command has finished executing! Here's what you wrote!

/morph export-build

Required to prevent that pesky -1million something error code from ruining our CI

theotherjimmy · 2016-09-06T19:51:42Z

/morph export-build

mbed-bot · 2016-09-06T21:15:34Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph export-build

theotherjimmy · 2016-09-06T21:16:41Z

Just had a look at the results, I'm pretty sure they are not a regression.

theotherjimmy · 2016-09-06T21:48:24Z

LGTM.

Release mbed-os-5.1.4 Changes: New Targets: 2504: [Disco_F769NI] adding new target [#2504] 2654: DELTA_DFBM_NQ620 platform porting [#2654] 2615: [MTM_MTCONNECT04S] Added support for MTM_MTCONNECT04S [#2615] 2548: Nucleof303ze [#2548] Fixes: 2678: Fixing NCS36510 compile on Linux #2678 2657: [MAX326xx] Removed echoing of characters and carriage return. #2657 2651: Use lp_timer to count time in the deepsleep tests #2651 2645: NUCLEO_F446ZE - Enable mbed5 release version #2645 2643: Fix thread self termination #2643 2634: Updated USBHost for library changes #2634 2633: Updated USBDevice to use Callback #2633 2630: Test names not dependent on disk location of root #2630 2624: CFSTORE Bugfix for realloc() moving KV area and cfstore_file_t data structures not being updated correctly #2624 2623: DISCO_L476VG - Add Serial Flow Control pins + add SERIAL_FC macro #2623 2617: STM32F2xx - Enable Serial Flow Control #2617 2613: Correctly providing directories to build_apis #2613 2607: Fix uvisor memory tracing #2607 2604: Tools - Fix fill section size variation #2604 2601: Adding ON Semiconductor copyright notice to source and header files. #2601 2597: [HAL] Fixed "intrinsic is deprecated" warnings #2597 2596: [HAL] Improve memory tracer #2596 2594: Fix TCPServer constructor #2594 2593: Add app config command line switch for test and make #2593 2589: [NUC472] Fix heap configuration error with armcc #2589 2588: Timing tests drift refactor #2588 2587: add PTEx pins as option for SPI on Hexiwear - for SD Card Interface #2587 2584: Set size of callback irq array to IrqCnt #2584 2583: github issue and PR templates #2583 2582: [GCC_CR] fix runtime hang for baremetal build #2582 2580: lwip - Add check for previously-bound socket #2580 2579: lwip - Fix handling of max sockets in socket_accept #2579 2578: Fix double free in NanostackInterface #2578 2576: Add smoke test that builds example programs with mbed-cli #2576 2575: tools-config! - Allow an empty or mal-formed config to be passed to the config system #2575 2562: Fix GCC lazy init race condition and add test #2562 2559: [utest]: Allow the linker to remove any part of utest if not used #2559 2545: Added define guards for SEQUENTIAL_FLASH_JOURNAL_MAX_LOGGED_BLOBS so #2545 2538: STM32F4xx - Add support of ADC internal channels (Temp, VRef, VBat) #2538 2521: [NUCLEO_F207ZG] Add MBED5 capability #2521 2514: Updated FlexCan and SAI SDK drivers #2514 2487: Runtime dynamic memory tracing #2487 2442: Malloc heap info #2442 2419: [STM32F1] Add asynchronous serial #2419 2393: [tools] Prevent trace-backs from incomplete args #2393 2245: Refactor export subsystem #2245 2130: stm32 : reduce number of device.h files #2130

sg- reviewed Jul 26, 2016
View reviewed changes

sg- mentioned this pull request Jul 26, 2016

Export - define ProGenDef outside of the loop #2202

Merged

0xc0170 reviewed Jul 26, 2016
View reviewed changes

theotherjimmy force-pushed the exporter-refactor branch from 6408dac to 78c46b2 Compare July 27, 2016 20:43

sarahmarshy force-pushed the exporter-refactor branch from a040b1e to d66d67f Compare July 28, 2016 15:31

theotherjimmy force-pushed the exporter-refactor branch 2 times, most recently from f6b7b74 to b44c5cc Compare July 29, 2016 00:18

theotherjimmy force-pushed the exporter-refactor branch from b44c5cc to 2e116f1 Compare July 29, 2016 15:38

theotherjimmy force-pushed the exporter-refactor branch from 2e116f1 to 023ee70 Compare August 1, 2016 21:36

sarahmarshy and others added 13 commits September 6, 2016 14:24

progen exporter template

0016ddf

Requirements update

f4a686f

Allow exporting to in-memory zip file

67fae3a

Made exporting safer

3e2526a

it will no longer barf when: - a linker scirpt is None - an attribute that is a set it will also export the correct library include paths

Reinstated the zip exporter

36c750b

Fix renaming issues in non-CI ides

e5de39e

Affects these ides: - Atmel Studio - Code Red (I don't think we support this) - Coide - DS-5 - E2Studio - EMblocks - KDS - Simplicity v3 - SW 4 STM32 also corrects flags usage in EMBlocks

Create projectfiles directory when exporting

2196d50

Compatible with new c/asm/cpp flag separation.

Set vpath correctly when exporting to projectfiles directory

6686411

Fix some tracebacks, add zip exporter to the CLI

c550f9d

Fix include paths for fragmented projects; remove deadcode

b5c1899

Add repo_dirs, repo_files, lib_builds, and lib_refs to zips

a6be472

Removed projectfiles directory. debug-info default option

7ee6213

theotherjimmy force-pushed the exporter-refactor branch from d77fc78 to 7ee6213 Compare September 6, 2016 19:25

Bump Progen version

6564bda

Required to prevent that pesky -1million something error code from ruining our CI

sg- added ready for merge and removed needs: CI labels Sep 7, 2016

sg- merged commit da3b07d into ARMmbed:master Sep 7, 2016

sg- added release-version: 5.1.4 and removed ready for merge labels Sep 7, 2016

janjongboom mentioned this pull request Sep 9, 2016

All exporters broken with mbed OS 5 #2591

Closed

theotherjimmy mentioned this pull request Sep 9, 2016

Export to GCC @ Online Compiler #2507

Closed

Refactor export subsystem #2245

Refactor export subsystem #2245

Uh oh!

Conversation

theotherjimmy commented Jul 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sg- Jul 26, 2016

Choose a reason for hiding this comment

Uh oh!

sg- commented Jul 26, 2016

Uh oh!

0xc0170 Jul 26, 2016

Choose a reason for hiding this comment

Uh oh!

theotherjimmy Jul 26, 2016

Choose a reason for hiding this comment

Uh oh!

0xc0170 commented Jul 26, 2016

Uh oh!

theotherjimmy commented Jul 26, 2016

Uh oh!

theotherjimmy commented Jul 26, 2016

Uh oh!

sg- commented Jul 27, 2016

Uh oh!

screamerbg commented Jul 27, 2016

Uh oh!

0xc0170 commented Jul 29, 2016

Uh oh!

theotherjimmy commented Jul 29, 2016

Uh oh!

theotherjimmy commented Jul 29, 2016

Uh oh!

bridadan commented Jul 29, 2016

Uh oh!

mbed-bot commented Jul 29, 2016

Result: FAILURE

Uh oh!

bridadan commented Jul 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sg- commented Jul 29, 2016

Uh oh!

bridadan commented Aug 1, 2016

Uh oh!

bridadan commented Aug 1, 2016

Uh oh!

mbed-bot commented Sep 6, 2016

Result: ABORTED

Uh oh!

theotherjimmy commented Sep 6, 2016

Uh oh!

mbed-bot commented Sep 6, 2016

Result: ABORTED

Uh oh!

theotherjimmy commented Sep 6, 2016

Uh oh!

mbed-bot commented Sep 6, 2016

Result: FAILURE

Uh oh!

theotherjimmy commented Sep 6, 2016

Uh oh!

theotherjimmy commented Sep 6, 2016

Uh oh!

Uh oh!

theotherjimmy commented Jul 25, 2016 •

edited

Loading

bridadan commented Jul 29, 2016 •

edited

Loading