ng serve Incremental Build Times Increased Massively When Upgrading From Angular v12 -> v15

[fearghalom]

Command

serve

Is this a regression?

• Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was

12.2.17

Description

I recently tried to upgrade our large enterprise application (~1300 components) from Angular v12 to v15. After the upgrade, I noticed that 'ng serve' incremental build times had increased from around (15s - 25s) on Angular v12 to (115s - 130s) on Angular v15. This kind of degradation in build performance was not expected. Here are my settings in the angular.json file for our 'development' config.

"development": {
"buildOptimizer": false,
"optimization": false,
"vendorChunk": true,
"extractLicenses": false,
"sourceMap": true,
"namedChunks": true,
"aot": false,
"progress": true
}

I tried a number of thing to see if I could bring down the time. One that gave a limited improvement was to move the entries from the "scripts" section in the angular.json file and directly reference them in the index.html file via script tags instead. This reduced the incremental build times to (95s - 100s)

Minimal Reproduction

I wasn't able to reproduce issue in a smaller application and I can't share our code base.

Exception or Error

No response

Your Environment

_                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/


Angular CLI: 15.0.4
Node: 14.21.2
Package Manager: npm 6.14.17
OS: win32 x64

Angular: 15.0.4
... animations, cli, common, compiler, compiler-cli, core, forms
... language-service, localize, platform-browser
... platform-browser-dynamic, router

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1500.4
@angular-devkit/build-angular   15.0.4
@angular-devkit/core            15.0.4
@angular-devkit/schematics      15.0.4
@angular/cdk                    15.0.3
@angular/flex-layout            14.0.0-beta.41
@angular/material               15.0.3
@schematics/angular             15.0.4
rxjs                            6.6.0
typescript                      4.8.4

Anything else relevant?

No response

[alan-agius4]

Can you please share the entire build and serve sections of your angular.json and the exact command that you are executing?

[fearghalom]

Can you please share the entire build and serve sections of your angular.json and the exact command that you are executing?

    "build": {
      "builder": "@angular-builders/custom-webpack:browser",
      "options": {
        "allowedCommonJsDependencies": [
          "lodash",
          "highcharts",
          "moment-timezone",
          "daterangepicker",
          "jquery",
          "wolfy87-eventemitter",
          "css-element-queries",
          "pdfmake/build/pdfmake",
          "pdfmake/build/vfs_fonts",
          "html2canvas",
          "ipaddr.js",
          "deepmerge",
          "moment/moment",
          "vis",
          "messageformat"
        ],
        "outputPath": "dist/my-app",
        "index": "src/index.html",
        "main": "src/main.ts",
        "polyfills": [
          "zone.js"
        ],
        "tsConfig": "tsconfig.app.json",
        "inlineStyleLanguage": "scss",
        "preserveSymlinks": true,
        "assets": [
          "src/assets"
        ],
        "styles": [
          "./node_modules/please-wait/build/please-wait.css",
          "./node_modules/bootstrap/dist/css/bootstrap.min.css",        
          "./node_modules/vis/dist/vis.css",
          "./node_modules/fullpage.js/dist/jquery.fullpage.min.css",
          "./node_modules/ag-grid-community/dist/styles/ag-grid.css",
          "./node_modules/ag-grid-community/dist/styles/ag-theme-balham.css",
          "./node_modules/@angular/material/_theming.scss",
          "./node_modules/angular-archwizard/archwizard.scss",
          "./node_modules/daterangepicker/daterangepicker.css",
          "style.scss"
        ],
        "stylePreprocessorOptions": {
          "includePaths": [
            "src/assets/sass"
          ]
        },
        "scripts": [
          "./node_modules/.../script1.js",
          "./node_modules/.../script2.js",
      "./node_modules/.../script2.js",
        ]
      },
      "configurations": {
        "production": {
          "outputHashing": "none",
          "customWebpackConfig": {
            "path": "./custom-webpack.prod.js"
          },
          "buildOptimizer": true,
          "optimization": false,
          "sourceMap": false,
          "aot": true
        },
        "development": {
          "buildOptimizer": false,
          "optimization": false,
          "vendorChunk": true,
          "extractLicenses": false,
          "sourceMap": true,
          "namedChunks": true,
          "aot": false,
          "progress": true
        }
      }
    },
    "serve": {
      "builder": "@angular-devkit/build-angular:dev-server",
      "options": {
        "browserTarget": "my-app:build:development",
        "port": 3000
      },
      "configurations": {
        "production": {
          "browserTarget": "my-app:build:production"
        },
        "development": {
          "browserTarget": "my-app:build:development"
        },
        "defaultConfiguration": "development"
      }
    },

The command: node --max-old-space-size=16384 ./node_modules/@angular/cli/bin/ng serve

[alan-agius4]

@fearghalom, you are using a 3rd party builder that is not supported by the Angular team. Can you try to see if the problem persists if you use the supported builder?

[fearghalom]

I was using the custom 3rd party builder for the production build but still using "@angular-devkit/build-angular:dev-server" for the dev build (ng serve).
I updated the angular.json anyway to remove the usage of the custom builder but it made no impact to the build times.

    "build": {
      "builder": "@angular-devkit/build-angular:browser",
      "options": {
        "allowedCommonJsDependencies": [
          "lodash",
          "highcharts",
          "moment-timezone",
          "daterangepicker",
          "jquery",
          "wolfy87-eventemitter",
          "css-element-queries",
          "pdfmake/build/pdfmake",
          "pdfmake/build/vfs_fonts",
          "html2canvas",
          "ipaddr.js",
          "deepmerge",
          "moment/moment",
          "vis",
          "messageformat"
        ],
        "outputPath": "dist/my-app",
        "index": "src/index.html",
        "main": "src/main.ts",
        "polyfills": [
          "zone.js"
        ],
        "tsConfig": "tsconfig.app.json",
        "inlineStyleLanguage": "scss",
        "preserveSymlinks": true,
        "assets": [
          "src/assets"
        ],
        "styles": [
          "./node_modules/please-wait/build/please-wait.css",
          "./node_modules/bootstrap/dist/css/bootstrap.min.css",        
          "./node_modules/vis/dist/vis.css",
          "./node_modules/fullpage.js/dist/jquery.fullpage.min.css",
          "./node_modules/ag-grid-community/dist/styles/ag-grid.css",
          "./node_modules/ag-grid-community/dist/styles/ag-theme-balham.css",
          "./node_modules/@angular/material/_theming.scss",
          "./node_modules/angular-archwizard/archwizard.scss",
          "./node_modules/daterangepicker/daterangepicker.css",
          "style.scss"
        ],
        "stylePreprocessorOptions": {
          "includePaths": [
            "src/assets/sass"
          ]
        },
        "scripts": [
          "src/assests/script1.js",
          "src/assests/script2.js",
		  "src/assests/script2.js",
        ]
      },
      "configurations": {
        "production": {
          "outputHashing": "none",
          "customWebpackConfig": {
            "path": "./custom-webpack.prod.js"
          },
          "buildOptimizer": true,
          "optimization": false,
          "sourceMap": false,
          "aot": true
        },
        "development": {
          "buildOptimizer": false,
          "optimization": false,
          "vendorChunk": true,
          "extractLicenses": false,
          "sourceMap": true,
          "namedChunks": true,
          "aot": false,
          "progress": true
        }
      }
    },
    "serve": {
      "builder": "@angular-devkit/build-angular:dev-server",
      "options": {
        "browserTarget": "my-app:build:development",
        "port": 3000
      },
      "configurations": {
        "production": {
          "browserTarget": "my-app:build:production"
        },
        "development": {
          "browserTarget": "my-app:build:development"
        },
        "defaultConfiguration": "development"
      }
    },

[alan-agius4]

Hi @fearghalom, in that case we'd require an CPU profile to investigate this any further.

[fearghalom]

Hi Alan, do you have instructions on how to do that?

[alan-agius4]

See: https://github.com/angular/angular-cli/blob/main/docs/DEVELOPER.md#cpu-profiling

[fearghalom]

cpuprofile.zip

[alan-agius4]

@fearghalom, are these profile for a build? I am not seeing much happening in these. Mostly it's just the process being idle.

[fearghalom]

These cpu profile logs were recorded for 'ng serve':
node --cpu-prof --max-old-space-size=16384 ./node_modules/@angular/cli/bin/ng serve

I noticed that there were 5 .cpuprofile files produced during the initial build but nothing was logged after that.

[alan-agius4]

Since you are profiling a rebuild in this case it would probably better to profile manually using the steps mentioned in #8259 (comment)

[fearghalom]

Here's the profile for the rebuild:
rebuild_profile_CPU-20230203T225405.zip

[alan-agius4]

Thanks for the profile.

I did take a look at it and I can see is that there are a lot of IO operations specifically readlinkSync with aggregated total time of 57.79s.

See screenshots below;

Also, I recommend is to update to Node.js 16, remove --max-old-space-size=16384 which should not be needed with Node.js 16 and see if the problem is still reproducible.

That said, I highly doubt that this is something that we can action from our end.

[fearghalom]

Thanks Alan for taking a look. I had actually upgraded to Node v18.10.0 to perform the profiling as it was crashing when I tried to use v14.21.2. I gave Node v16 a go also but it didn't make any difference to the rebuild times.

Can anything further be done on this? We probably can't move to v15 with these rebuild times. As I mentioned previously, I tried with angular 14 and the rebuild times were also slow but Angular 13 was comparable to Angular 12.

One thing I did notice that brought down the rebuild time, was to remove the entries from the "scripts" section in the angular.json file and reference them directly in the index.html file. Doing this, knocked about 30 seconds off the rebuild time.
I profiled it and it seems that having even one item in the scripts section of the angular.json file, results in the Webpack DescriptionFileUtils.loadDescriptionFile() method being invoked for what looks like every dependency in the project. When I leave the "scripts" section blank in angular.json, the number of times DescriptionFileUtils.loadDescriptionFile() is invoked decreases significantly on rebuilds.