parallelize ci with separate auto-{nightly, beta, stable} branches

[matthiaskrgr]

It seems currently the average ci-run takes around 2.5 hours to complete which means there are only 9 chances to get a PR merged in a day (besides rollups).

Given that currently a lot of builds either hit the 3 hours timeout (especially on appveyor) or fail, this is not a lot.
There seems to be a single test-brach (auto) which is based on respective master(nightly), beta or stable branch and the gets the PR applied on top and then runs throuth CI.

Is it possible to split this up into 3 branches, one for each of stable, beta and nightly release, so auto-nightly, auto-beta and auto-stable which run and test in parallel (so that beta changes and nightly changes can be tested at the same time).
This might help a bit.

I also found this neat site which can visualize travis build time: http://scribu.net/travis-stats/#rust-lang/rust/auto
It seems a significant bump in total build time was introduced in 0c6091f :

[kennytm]

We have a cap on the Travis and AppVeyor build jobs, so even if it is separated into 3 branches, they may still be running in sequence.

[matthiaskrgr]

Ah, what's the limit?
Anyway usually there are some jobs that are already done after ~1.5 hours which then could be allocated to another branch I think.

[kennytm]

BTW thanks for that site! It looks pretty useful. The "sum of job run times" is not a good metric since all jobs are running in parallel, the "Build wall times" is better.

[kennytm]

@matthiaskrgr It is close to the current number of jobs, about 50 for Travis and I think we reached the max for AppVeyor already IIRC. Allocating more jobs is possible by paying more, but the beta/stable-backport traffic is not that high it is probably not worth the investment to apply this (split into 3 auto branches), compared to parellizing a single PR test more (e.g. #48487).

[matthiaskrgr]

Hmm, ok...

[alexcrichton]

Yes as @kennytm mentioned we're limited by physical capacity here, so unfortunately we can't do this.