Skip to content

Commit 3e1b6db

Browse files
pepeiborrapepeiborra
committed
HLS benchmark suite
Port the ghcide benchmark suite to HLS and benchmark plugin "configurations" independently. This includes the following changes to the ghcide benchmark suite and HLS: - Support for "configurations" which are defined as sets of plugin ids. The benchmark will be run with only these plugins enabled and all others disabled - Support for configurable concurrency. This relies on RTS -ol and -po flags to place the RTS traces in the target location rather than in the cwd This change requires two commits, the next one places ghcide/bench/hist/Main.hs into its final location to help 'git' recognize the change as a file move
1 parent b7716a4 commit 3e1b6db

File tree

15 files changed

+442
-239
lines changed

15 files changed

+442
-239
lines changed

‎.github/workflows/bench.yml

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -67,10 +67,10 @@ jobs:
6767
- run: cabal configure --enable-benchmarks --max-backjumps 12000
6868

6969
- name: Build
70-
run: cabal build ghcide:benchHist
70+
run: cabal build haskell-language-server:benchmark
7171

7272
- name: Bench init
73-
run: cabal bench ghcide:benchHist -j --benchmark-options="all-binaries"
73+
run: cabal bench -j --benchmark-options="all-binaries"
7474

7575
# tar is required to preserve file permissions
7676
# compression speeds up upload/download nicely
@@ -134,14 +134,14 @@ jobs:
134134
tar xzf cabal.tar.gz --directory ~/.cabal
135135
136136
- name: Bench
137-
run: cabal bench ghcide:benchHist -j --benchmark-options="${{ matrix.example }}"
137+
run: cabal bench -j --benchmark-options="${{ matrix.example }}"
138138

139139
- name: Display results
140140
run: |
141-
column -s, -t < ghcide/bench-results/unprofiled/${{ matrix.example }}/results.csv | tee ghcide/bench-results/unprofiled/${{ matrix.example }}/results.txt
141+
column -s, -t < bench-results/unprofiled/${{ matrix.example }}/results.csv | tee bench-results/unprofiled/${{ matrix.example }}/results.txt
142142
143143
- name: tar benchmarking artifacts
144-
run: find ghcide/bench-results -name "*.csv" -or -name "*.svg" -or -name "*.html" | xargs tar -czf benchmark-artifacts.tar.gz
144+
run: find bench-results -name "*.csv" -or -name "*.svg" -or -name "*.html" | xargs tar -czf benchmark-artifacts.tar.gz
145145

146146
- name: Archive benchmarking artifacts
147147
uses: actions/upload-artifact@v3
@@ -150,7 +150,7 @@ jobs:
150150
path: benchmark-artifacts.tar.gz
151151

152152
- name: tar benchmarking logs
153-
run: find ghcide/bench-results -name "*.log" -or -name "*.eventlog" -or -name "*.hp" | xargs tar -czf benchmark-logs.tar.gz
153+
run: find bench-results -name "*.log" -or -name "*.eventlog" -or -name "*.hp" | xargs tar -czf benchmark-logs.tar.gz
154154

155155
- name: Archive benchmark logs
156156
uses: actions/upload-artifact@v3

‎.gitignore

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,10 @@ test/testdata/**/hie.yaml
3434
/.direnv/
3535
/.envrc
3636

37-
# ghcide-bench
37+
# bench
3838
*.identifierPosition
3939
/bench/example
40+
/bench-results
4041

4142
# nix
4243
result

‎ghcide/bench/README.md renamed to ‎bench/README.md

Lines changed: 10 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,17 @@
11

22
# Benchmarks
33

4-
This folder contains two Haskell programs that work together to simplify the
5-
performance analysis of ghcide:
6-
7-
- `exe/Main.hs` - a standalone benchmark runner. Run with `stack run ghcide-bench`
8-
- `hist/Main.hs` - a Shake script for running the benchmark suite over a set of commits.
9-
- Run with `stack bench ghcide` or `cabal bench ghcide`,
10-
- Requires a `ghcide-bench` binary in the PATH (usually provided by stack/cabal),
11-
- Calls `cabal` (or `stack`, configurable) internally to build the project,
12-
- Driven by the `bench/config.yaml` configuration file.
13-
By default it compares HEAD with "master"
4+
This folder contains a Shake script to simplify the performance analysis of HLS.
5+
It drives the `ghcide-bench` benchmark suite over a set of commits and experiments.
6+
To run it, use `cabal bench`.
7+
To configure it, edit `bench/config.yaml`.
8+
By default it compares HEAD with "origin/master"
149

1510
# Examples and experiments
1611

1712
The benchmark suites runs a set of experiments (hover, completion, edit, etc.)
1813
over all the defined examples (currently Cabal and lsp-types). Examples are defined
19-
in `ghcide/bench/config.yaml` whereas experiments are coded in `ghcide/bench/lib/Experiments.hs`.
14+
in `bench/config.yaml` whereas experiments are coded in `ghcide/bench/lib/Experiments.hs`.
2015

2116
# Phony targets
2217

@@ -34,11 +29,14 @@ The Shake script supports a number of phony targets that allow running a subset
3429
* profiled-Cabal-3.0.0.0
3530
: runs the Cabal example, with heap profiling
3631

32+
* all-binaries
33+
: build all the HLS binaries for each of the versions under analysis
34+
3735
* etc
3836

3937
`--help` lists all the phony targets. Invoke it with:
4038

41-
cabal bench ghcide --benchmark-options="--help"
39+
cabal bench --benchmark-options="--help"
4240

4341
```
4442
Targets:

‎ghcide/bench/config.yaml renamed to ‎bench/config.yaml

Lines changed: 90 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,13 @@ buildTool: cabal
77
# Output folder for the experiments
88
outputFolder: bench-results
99

10+
# Heap profile interval in seconds (+RTS -i)
11+
# Comment out to disable heap profiling
12+
profileInterval: 1
13+
14+
# Number of concurrent benchmark and warmup runs
15+
parallelism: 1
16+
1017
# Example project used to run the experiments
1118
# Can either be a Hackage package (name,version)
1219
# or a local project (path) with a valid `hie.yaml` file
@@ -18,36 +25,14 @@ examples:
1825
modules:
1926
- src/Distribution/Simple.hs
2027
- src/Distribution/Types/Module.hs
21-
extra-args: [] # extra ghcide command line args
22-
- name: cabal-1module
23-
package: Cabal
24-
version: 3.6.3.0
25-
modules:
26-
- src/Distribution/Simple.hs
27-
- name: cabal-conservative
28-
package: Cabal
29-
version: 3.6.3.0
30-
modules:
31-
- src/Distribution/Simple.hs
32-
- src/Distribution/Types/Module.hs
33-
extra-args: # extra ghcide command line args
34-
- --conservative-change-tracking
28+
extra-args: [] # extra HLS command line args
3529
# Small-sized project with TH
3630
- name: lsp-types
3731
package: lsp-types
3832
version: 1.5.0.0
3933
modules:
4034
- src/Language/LSP/Types/WatchedFiles.hs
4135
- src/Language/LSP/Types/CallHierarchy.hs
42-
- name: lsp-types-conservative
43-
package: lsp-types
44-
version: 1.5.0.0
45-
modules:
46-
- src/Language/LSP/Types/WatchedFiles.hs
47-
- src/Language/LSP/Types/CallHierarchy.hs
48-
extra-args:
49-
- --conservative-change-tracking
50-
# Small-sized project with TH
5136
# Small but heavily multi-component example
5237
# Disabled as it is far to slow. hie-bios >0.7.2 should help
5338
# - name: HLS
@@ -90,27 +75,87 @@ versions:
9075
# parent: <tag/branch/commithash> # version to compare with in .diff graphs
9176

9277

93-
# - ghcide-v0.0.5
94-
# - ghcide-v0.0.6
95-
# - ghcide-v0.1.0
96-
# - ghcide-v0.2.0
97-
# - ghcide-v0.3.0
98-
# - ghcide-v0.4.0
99-
# - ghcide-v0.5.0
100-
# - ghcide-v0.6.0
101-
# - ghcide-v0.7.0
102-
# - ghcide-v0.7.1
103-
# - ghcide-v0.7.2
104-
# - ghcide-v0.7.3
105-
# - ghcide-v0.7.4
106-
# - ghcide-v0.7.5
107-
# - 1.0.0
108-
# - ghcide-v1.1.0
109-
# - ghcide-v1.2.0
110-
# - ghcide-v1.3.0
111-
- upstream: origin/master
78+
# - 1.8.0.0
79+
# - upstream: origin/master
80+
# - HEAD~1
11281
- HEAD
11382

114-
# Heap profile interval in seconds (+RTS -i)
115-
# Comment out to disable heap profiling
116-
profileInterval: 1
83+
configurations:
84+
- None: []
85+
- Core:
86+
- callHierarchy
87+
- codeRange
88+
- eval
89+
- ghcide-code-actions-bindings
90+
- ghcide-code-actions-fill-holes
91+
- ghcide-code-actions-imports-exports
92+
- ghcide-code-actions-type-signatures
93+
- ghcide-completions
94+
- ghcide-type-lenses
95+
- pragmas
96+
- Ghcide:
97+
- ghcide-code-actions-bindings
98+
- ghcide-code-actions-fill-holes
99+
- ghcide-code-actions-imports-exports
100+
- ghcide-code-actions-type-signatures
101+
- ghcide-completions
102+
- ghcide-type-lenses
103+
- All:
104+
- alternateNumberFormat
105+
- callHierarchy
106+
- changeTypeSignature
107+
- class
108+
- codeRange
109+
- eval
110+
- explicitFixity
111+
- floskell
112+
- fourmolu
113+
- gadt
114+
- ghcide-code-actions-bindings
115+
- ghcide-code-actions-fill-holes
116+
- ghcide-code-actions-imports-exports
117+
- ghcide-code-actions-type-signatures
118+
- ghcide-completions
119+
- ghcide-type-lenses
120+
- hlint
121+
- importLens
122+
- moduleName
123+
- ormolu
124+
- pragmas
125+
- qualifyImportedNames
126+
- refineImports
127+
- rename
128+
- stylish-haskell
129+
- alternateNumberFormat
130+
- brittany
131+
- callHierarchy
132+
- changeTypeSignature
133+
- class
134+
- codeRange
135+
- eval
136+
- explicitFixity
137+
- floskell
138+
- fourmolu
139+
- gadt
140+
- ghcide-code-actions-bindings
141+
- ghcide-code-actions-fill-holes
142+
- ghcide-code-actions-imports-exports
143+
- ghcide-code-actions-type-signatures
144+
- ghcide-completions
145+
# - ghcide-core # implicitly included in all configurations
146+
# - ghcide-hover-and-symbols # implicitly included in all configurations
147+
- ghcide-type-lenses
148+
- haddockComments
149+
- hlint
150+
- importLens
151+
- moduleName
152+
- ormolu
153+
- pragmas
154+
- qualifyImportedNames
155+
- refineImports
156+
- rename
157+
- retrie
158+
- splice
159+
- stan
160+
- stylish-haskell
161+
- tactics

‎docs/contributing/contributing.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -208,11 +208,11 @@ If you are touching performance sensitive code, take the time to run a different
208208
benchmark between HEAD and master using the benchHist script. This assumes that
209209
"master" points to the upstream master.
210210

211-
Run the benchmarks with `cabal bench ghcide`.
211+
Run the benchmarks with `cabal bench`.
212212

213-
It should take around 25 minutes and the results will be stored in the `ghcide/bench-results` folder. To interpret the results, see the comments in the `ghcide/bench/hist/Main.hs` module.
213+
It should take around 25 minutes and the results will be stored in the `bench-results` folder. To interpret the results, see the comments in the `bench/Main.hs` module.
214214

215-
More details in [bench/README](../../ghcide/bench/README.md)
215+
More details in [bench/README](../../bench/README.md)
216216

217217
### Tracing
218218

‎exe/Main.hs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
{-# LANGUAGE CPP #-}
44
{-# LANGUAGE NamedFieldPuns #-}
55
{-# LANGUAGE OverloadedStrings #-}
6+
{-# LANGUAGE PackageImports #-}
67
module Main(main) where
78

89
import Control.Arrow ((&&&))

‎ghcide/bench-results/.artifactignore

Lines changed: 0 additions & 4 deletions
This file was deleted.

0 commit comments

Comments
 (0)