Better sharding mechanism

[malfet]

Right now sharding is done in round-robin manner per folder, see https://github.com/pytorch/tutorials/blob/master/.jenkins/build.sh
Which creates a very unbalanced experience as some folders have 20+ tutorials, but many have as few as 3-4 tutorials, which is guaranteed to schedule more tutorials build in first shards and leave later shards with just 1 or 2 tutorials to build

This can be addressed by enumerating all tutorials and dividing them between shards evenly

cc @malfet @janeyx99

[clee2000]

its already round robin sharded, we just get really unlucky b/c some tutorials take 20 min to build and they end up on the same shard

should do meta data thing like multigpu to keep track of tutorials that take a long time to run and shard based on that

[clee2000]

as stated in #1910

there are three files (can see them in the metadata file) that are long running (20-30 minutes) and are responsible for the unbalance in the shards. The long run time comes from running a training loop, so we should see what can be done about not running this training loop everytime (perhaps caching the result or reducing the number of epochs)

[svekars]

Any update on this by any chance?