Skip to content

refactor: remove --time flags in favor of cargo time command #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 13, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 23 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,10 +89,6 @@ cargo solve <day>

The `solve` command runs your solution against real puzzle inputs. To run an optimized build of your code, append the `--release` flag as with any other rust program.

By default, `solve` executes your code once and shows the execution time. If you append the `--time` flag to the command, the runner will run your code between `10` and `10.000` times (depending on execution time of first execution) and print the average execution time.

For example, running a benchmarked, optimized execution of day 1 would look like `cargo solve 1 --release --time`. Displayed _timings_ show the raw execution time of your solution without overhead like file reads.

#### Submitting solutions

> [!IMPORTANT]
Expand All @@ -116,15 +112,33 @@ cargo all
# Total: 0.20ms
```

This runs all solutions sequentially and prints output to the command-line. Same as for the `solve` command, the `--release` flag runs an optimized build and the `--time` flag outputs benchmarks.
This runs all solutions sequentially and prints output to the command-line. Same as for the `solve` command, the `--release` flag runs an optimized build.

### ➡️ Benchmark your solutions

```sh
# output:
# Day 08
# ------
# Part 1: 1 (39.0ns @ 10000 samples)
# Part 2: 2 (39.0ns @ 10000 samples)
#
# Total (Run): 0.00ms
#
# Stored updated benchmarks.
```

The `cargo time` command allows you to benchmark your code and store timings in the readme. When benching, the runner will run your code between `10` and `10.000` times, depending on execution time of first execution, and print the average execution time.

### ➡️ Update readme benchmarks
`cargo time` has three modes of execution:

The template can write benchmark times to the readme via the `cargo time` command.
1. `cargo time` without arguments incrementally benches solutions that do not have been stored in the readme yet and skips the rest.
2. `cargo time <day>` benches a single solution.
3. `cargo time --all` benches all solutions.

By default, this command checks for missing benchmarks, runs those solutions, and then updates the table. If you want to (re-)time all solutions, run `cargo time --all`. If you want to (re-)time one specific solution, run `cargo time <day>`.
By default, `cargo time` does not write to the readme. In order to do so, append the `--store` flag: `cargo time --store`.

Please note that these are not _scientific_ benchmarks, understand them as a fun approximation. 😉 Timings, especially in the microseconds range, might change a bit between invocations.
> Please note that these are not _scientific_ benchmarks, understand them as a fun approximation. 😉 Timings, especially in the microseconds range, might change a bit between invocations.

### ➡️ Run all tests

Expand Down
1 change: 0 additions & 1 deletion src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
pub mod template;

// Use this file to add helper functions and additional modules.

14 changes: 6 additions & 8 deletions src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,17 +24,16 @@ mod args {
Solve {
day: Day,
release: bool,
time: bool,
dhat: bool,
submit: Option<u8>,
},
All {
release: bool,
time: bool,
},
Time {
all: bool,
day: Option<Day>,
store: bool,
},
#[cfg(feature = "today")]
Today,
Expand All @@ -46,14 +45,15 @@ mod args {
let app_args = match args.subcommand()?.as_deref() {
Some("all") => AppArguments::All {
release: args.contains("--release"),
time: args.contains("--time"),
},
Some("time") => {
let all = args.contains("--all");
let store = args.contains("--store");

AppArguments::Time {
all,
day: args.opt_free_from_str()?,
store,
}
}
Some("download") => AppArguments::Download {
Expand All @@ -70,7 +70,6 @@ mod args {
day: args.free_from_str()?,
release: args.contains("--release"),
submit: args.opt_value_from_str("--submit")?,
time: args.contains("--time"),
dhat: args.contains("--dhat"),
},
#[cfg(feature = "today")]
Expand Down Expand Up @@ -101,8 +100,8 @@ fn main() {
std::process::exit(1);
}
Ok(args) => match args {
AppArguments::All { release, time } => all::handle(release, time),
AppArguments::Time { day, all } => time::handle(day, all),
AppArguments::All { release } => all::handle(release),
AppArguments::Time { day, all, store } => time::handle(day, all, store),
AppArguments::Download { day } => download::handle(day),
AppArguments::Read { day } => read::handle(day),
AppArguments::Scaffold { day, download } => {
Expand All @@ -114,10 +113,9 @@ fn main() {
AppArguments::Solve {
day,
release,
time,
dhat,
submit,
} => solve::handle(day, release, time, dhat, submit),
} => solve::handle(day, release, dhat, submit),
#[cfg(feature = "today")]
AppArguments::Today => {
match Day::today() {
Expand Down
4 changes: 2 additions & 2 deletions src/template/commands/all.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use crate::template::{all_days, run_multi::run_multi};

pub fn handle(is_release: bool, is_timed: bool) {
run_multi(&all_days().collect(), is_release, is_timed);
pub fn handle(is_release: bool) {
run_multi(&all_days().collect(), is_release, false);
}
6 changes: 1 addition & 5 deletions src/template/commands/solve.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ use std::process::{Command, Stdio};

use crate::template::Day;

pub fn handle(day: Day, release: bool, time: bool, dhat: bool, submit_part: Option<u8>) {
pub fn handle(day: Day, release: bool, dhat: bool, submit_part: Option<u8>) {
let mut cmd_args = vec!["run".to_string(), "--bin".to_string(), day.to_string()];

if dhat {
Expand All @@ -23,10 +23,6 @@ pub fn handle(day: Day, release: bool, time: bool, dhat: bool, submit_part: Opti
cmd_args.push(submit_part.to_string());
}

if time {
cmd_args.push("--time".to_string());
}

let mut cmd = Command::new("cargo")
.args(&cmd_args)
.stdout(Stdio::inherit())
Expand Down
24 changes: 13 additions & 11 deletions src/template/commands/time.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@ use crate::template::run_multi::run_multi;
use crate::template::timings::Timings;
use crate::template::{all_days, readme_benchmarks, Day};

pub fn handle(day: Option<Day>, recreate_all: bool) {
pub fn handle(day: Option<Day>, run_all: bool, store: bool) {
let stored_timings = Timings::read_from_file();

let days_to_run = day.map_or_else(
|| {
if recreate_all {
if run_all {
all_days().collect()
} else {
// when the `--all` flag is not set, filter out days that are fully benched.
Expand All @@ -23,16 +23,18 @@ pub fn handle(day: Option<Day>, recreate_all: bool) {

let timings = run_multi(&days_to_run, true, true).unwrap();

let merged_timings = stored_timings.merge(&timings);
merged_timings.store_file().unwrap();
if store {
let merged_timings = stored_timings.merge(&timings);
merged_timings.store_file().unwrap();

println!();
match readme_benchmarks::update(merged_timings) {
Ok(()) => {
println!("Stored updated benchmarks.");
}
Err(_) => {
eprintln!("Failed to store updated benchmarks.");
println!();
match readme_benchmarks::update(merged_timings) {
Ok(()) => {
println!("Stored updated benchmarks.");
}
Err(_) => {
eprintln!("Failed to store updated benchmarks.");
}
}
}
}