Skip to content

Commit 9ea52cb

Browse files
obscurerichardobscurerichard
authored
Merge pull request #15 from ModusCreateOrg/feature/scale-test
Add JMeter-driven scale testing, driven by the Jenkinsfile
2 parents fbc1a68 + 9dc6a78 commit 9ea52cb

File tree

12 files changed

+326
-57
lines changed

12 files changed

+326
-57
lines changed

‎.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
*.ini
12
*.pyc
23
*.retry
34
*.tmp
@@ -10,4 +11,5 @@
1011
/terraform/tf.plan
1112
__pycache__
1213
build/
14+
jmeter.log
1315
venv/

‎Jenkinsfile

Lines changed: 54 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -60,13 +60,49 @@ properties([
6060
defaultValue: false,
6161
description: 'Destroy Terraform resources?'
6262
),
63+
string(
64+
name: 'Terraform_Targets',
65+
defaultValue: '',
66+
description: '''Specific Terraform resource or resource names to target
67+
(Use this to modify or delete less than the full set of resources'''
68+
),
69+
text(
70+
name: 'Extra_Variables',
71+
defaultValue: '',
72+
description: '''Terraform Variables to define for this run.
73+
Allows you to override declared variables.
74+
Put one variable per line, in JSON or HCL like this:
75+
associate_public_ip_address = "true"'''
76+
),
6377
booleanParam(
6478
name: 'Rotate_Servers',
6579
defaultValue: false,
6680
description: """Rotate server instances in Auto Scaling Group?
6781
You should do this if you changed ASG size or baked a new AMI.
6882
"""
69-
83+
),
84+
booleanParam(
85+
name: 'Run_JMeter',
86+
defaultValue: false,
87+
description: "Execute a JMeter load test against the stack"
88+
),
89+
string(
90+
name: 'JMETER_threads',
91+
defaultValue: '2',
92+
description: """number of jmeter threads. Resulting ASG stable sizes for t2.large instances are:
93+
- 2 threads, 3 instances;
94+
- 4 threads, 7 instances;
95+
"""
96+
),
97+
string(
98+
name: 'JMETER_ramp_duration',
99+
defaultValue: '900',
100+
description: 'period in seconds of ramp-up time.'
101+
),
102+
string(
103+
name: 'JMETER_duration',
104+
defaultValue: '1800',
105+
description: 'time in seconds to the whole Jmeter test'
70106
),
71107
string(
72108
name: 'CAPTCHA_Guess',
@@ -78,27 +114,13 @@ properties([
78114
defaultValue: captcha_hash,
79115
description: 'Hash for CAPTCHA answer (DO NOT modify)'
80116
),
81-
string(
82-
name: 'Terraform_Targets',
83-
defaultValue: '',
84-
description: '''Specific Terraform resource or resource names to target
85-
(Use this to modify or delete less than the full set of resources'''
86-
),
87-
text(
88-
name: 'Extra_Variables',
89-
defaultValue: '',
90-
description: '''Terraform Variables to define for this run.
91-
Allows you to override declared variables.
92-
Put one variable per line, in JSON or HCL like this:
93-
associate_public_ip_address = "true"'''
94-
),
95117
])
96118
])
97119

98120
stage('Preflight') {
99121

100122
// Check CAPTCHA
101-
def should_validate_captcha = params.Run_Packer || params.Apply_Terraform || params.Destroy_Terraform
123+
def should_validate_captcha = params.Run_Packer || params.Apply_Terraform || params.Destroy_Terraform || params.Run_JMeter
102124

103125
if (should_validate_captcha) {
104126
if (params.CAPTCHA_Guess == null || params.CAPTCHA_Guess == "") {
@@ -216,3 +238,19 @@ if (params.Rotate_Servers) {
216238
}
217239
}
218240
}
241+
242+
if (params.Run_JMeter) {
243+
stage('Run JMeter') {
244+
node {
245+
unstash 'src'
246+
wrap.call({
247+
sh ("""
248+
HOST=\$(./bin/terraform.sh output route53-dns)
249+
./bin/jmeter.sh -Jthreads=${params.JMETER_threads} -Jramp_duration=${params.JMETER_ramp_duration} -Jduration=${params.JMETER_duration} -Jhost=\$HOST
250+
ls -l build
251+
""")
252+
archiveArtifacts artifacts: 'build/*.jtl, build/*.xml, build/*.csv, build/*.html', fingerprint: true
253+
})
254+
}
255+
}
256+
}

‎README.md

Lines changed: 26 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -63,17 +63,9 @@ In order to make developing the Ansible playbooks faster, a Vagrantfile is provi
6363

6464
Install [Vagrant](https://www.vagrantup.com/). Change directory into the root of the repository at the command line and issue the command `vagrant up`. You can add or edit Ansible playbooks and support scripts then re-run the provisioning with `vagrant provision` to refine the remediations. This is more efficient that re-running packer and baking new AMIs for every change.
6565

66-
### Jenkins
67-
68-
A `Jenkinsfile` is provided that will allow Jenkins to execute Packer and Terraform. In order for Jenkins to do this, it needs to have AWS credentials set up, preferably through an IAM role, granting full control of EC2 and VPC resources in that account. Packer needs this in order to create AMIs, key pairs, etc, and Terraform needs this to create a VPC and EC2 resources. This could be pared down further through some careful logging and role work.
69-
70-
The scripts here assume that Jenkins is running on EC2 and uses instance data from the Jenkins executor to infer what VPC and subnet to launch the new EC2 instance into. The AWS profile IAM user associated with your Jenkins instance or the Jenkins user's AWS credentials should have full control of EC2 in the account you are using.
71-
72-
This script relies on Jenkins having a secret file containing the Google application credentials in JSON with the id "terraform-demo.json". You will need to add that to your Jenkins server's credentials.
73-
7466
### Terraform
7567

76-
This Terraform setup stores its state in Amazon S3 and uses DynamoDB for locking. There is a bit of setup required to bootstrap that configuration. Yu can use [this repository](https://github.com/monterail/terraform-bootstrap-example) to use Terraform to do that bootstrap process. The `backend.tfvars` file in that repo should be modified as follows to work with this project:
68+
This Terraform setup stores its state in Amazon S3 and uses DynamoDB for locking. There is a bit of setup required to bootstrap that configuration. You can use [this repository](https://github.com/monterail/terraform-bootstrap-example) to use Terraform to do that bootstrap process. The `backend.tfvars` file in that repo should be modified as follows to work with this project:
7769

7870
(Replace us-east-1 and XXXXXXXXXXXX with the AWS region and your account ID)
7971
```
@@ -97,10 +89,34 @@ These commands will then set up cloud resources using terraform:
9789
# check to see if everything worked - use the same variables here as above
9890
terraform destroy -var 'domain=example.net'
9991

92+
Alternatively, use the wrapper script in `bin/terraform.sh` which will work interactively or from CI:
93+
94+
bin/terraform.sh plan
95+
bin/terraform.sh apply
96+
bin/terraform.sh plan-destroy
97+
bin/terraform.sh destroy
98+
10099
This assumes that you already have a Route 53 domain in your AWS account created.
101100
You need to either edit variables.tf to match your domain and AWS zone or specify these values as command line `var` parameters.
102101

103-
The application loads an image from Google storage. To get it loading correctly, look in the X file and replace `example-media-website-storage.storage.googleapis.com` with a DNS reference for your Google storage location.
102+
The application loads an image from Google storage. To get it loading correctly, edit the `application/assets/css/main.css` file and replace `example-media-website-storage.storage.googleapis.com` with a DNS reference for your Google storage location.
103+
104+
### CodeDeploy
105+
106+
The application enclosed in this demo is packaged and deployed using [AWS CodeDeploy](https://aws.amazon.com/codedeploy/). The script `codedeploy/bin/build.sh` will package the application so that it can be deployed on the AMI built with Ansible and Packer.
107+
108+
The application contains both a simple HTML web site, and a Python app that has an API endpoint of `/api/spin` that spins the CPU of the server, in order to more easily test CPU-sensing auto scaling scale-out operations.
109+
110+
### JMeter
111+
112+
A JMeter test harness that will allow testing of a the application
113+
### Jenkins
114+
115+
A `Jenkinsfile` is provided that will allow Jenkins to execute Packer and Terraform, package a CodeDeploy application, and even run JMeter performance tests. In order for Jenkins to do this, it needs to have AWS credentials set up, preferably through an IAM role, granting full control of EC2 and VPC resources in that account, and write access to the S3 bucket used for storing CodeDeploy applications. Packer needs this in order to create AMIs, key pairs, etc, Terraform needs this to create a VPC and EC2 resources, and CodeDeploy needs this to store the artifact it creates. This could be pared down further through some careful logging and role work.
116+
117+
The scripts here assume that Jenkins is running on EC2 and uses instance data from the Jenkins executor to infer what VPC and subnet to launch the new EC2 instance into. The AWS profile IAM user associated with your Jenkins instance or the Jenkins user's AWS credentials should have full control of EC2 in the account you are using.
118+
119+
This script relies on Jenkins having a secret file containing the Google application credentials in JSON with the id "terraform-demo.json". You will need to add that to your Jenkins server's credentials.
104120

105121
# Modus Create
106122

‎bin/jmeter.sh

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/usr/bin/env bash
2+
3+
# Set bash unofficial strict mode http://redsymbol.net/articles/unofficial-bash-strict-mode/
4+
set -euo pipefail
5+
6+
# Set DEBUG to true for enhanced debugging: run prefixed with "DEBUG=true"
7+
${DEBUG:-false} && set -vx
8+
# Credit to https://stackoverflow.com/a/17805088
9+
# and http://wiki.bash-hackers.org/scripting/debuggingtips
10+
export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
11+
12+
# Credit to http://stackoverflow.com/a/246128/424301
13+
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
14+
BASE_DIR="$DIR/.."
15+
BUILD_DIR="$BASE_DIR/build"
16+
export BUILD_DIR
17+
18+
mkdir -p "$BUILD_DIR"
19+
docker run -t -v "$BASE_DIR:/repo" justb4/jmeter -n -t /repo/jmeter/api-spin.jmx -l /repo/build/jmeter.jtl "$@"

‎bin/spin-3.sh

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/usr/bin/env bash
2+
3+
ab -c 2 -t 900 http://devops-infra-demo.modus.app/api/spin

‎bin/spin-7.sh

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/usr/bin/env bash
2+
3+
ab -c 4 -t 900 http://devops-infra-demo.modus.app/api/spin

‎bin/spin.sh

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/usr/bin/env bash
2+
3+
ab -c 8 -t 900 http://devops-infra-demo.modus.app/api/spin

‎bin/terraform.sh

Lines changed: 26 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -31,12 +31,12 @@ BUILD_DIR="$BASE_DIR/build"
3131
DOCKER_TERRAFORM=$(get_docker_terraform)
3232
DOCKER_LANDSCAPE=$(get_docker_landscape)
3333

34-
verb=${1:?You must specify a verb: plan, plan-destroy, apply}
34+
verb=${1:?You must specify a verb: plan, plan-destroy, apply, show, output}
3535

3636
# Inject Google application credentials into env file for docker
3737
GOOGLE_APPLICATION_CREDENTIALS_OVERRIDE=${GOOGLE_APPLICATION_CREDENTIALS_OVERRIDE:-}
3838
if [[ -n "$GOOGLE_APPLICATION_CREDENTIALS_OVERRIDE" ]]; then
39-
echo "Overriding Google Application Credentials"
39+
echo "Overriding Google Application Credentials" 1>&2
4040
GOOGLE_APPLICATION_CREDENTIALS="$GOOGLE_APPLICATION_CREDENTIALS_OVERRIDE"
4141
fi
4242

@@ -60,6 +60,18 @@ EOF
6060
# http://redsymbol.net/articles/bash-exit-traps/
6161
trap clean_root_owned_docker_files EXIT
6262

63+
function show() {
64+
local -i retcode
65+
#shellcheck disable=SC2086
66+
$DOCKER_TERRAFORM show
67+
}
68+
69+
function output () {
70+
local -i retcode
71+
local extra
72+
#shellcheck disable=SC2086
73+
$DOCKER_TERRAFORM output "$@"
74+
}
6375

6476
function plan() {
6577
local extra
@@ -94,7 +106,7 @@ function plan() {
94106
}
95107

96108
function plan-destroy() {
97-
cat <<EOF
109+
cat 1>&2 <<EOF
98110
99111
*******************************************************
100112
************ **************
@@ -123,13 +135,19 @@ plan-destroy)
123135
apply)
124136
Message="Executing terraform apply."
125137
;;
138+
show)
139+
Message="Executing terraform show."
140+
;;
141+
output)
142+
Message="Executing terraform output."
143+
;;
126144
*)
127-
echo 'Unrecognized verb "'"$verb"'" specified. Use plan, plan-destroy, or apply'
145+
echo 'Unrecognized verb "'"$verb"'" specified. Use plan, plan-destroy, apply, or show' 1>&2
128146
exit 1
129147
;;
130148
esac
131-
132-
echo "$Message"
133-
init_terraform
134-
"$verb"
149+
shift
150+
echo "$Message" 1>&2
151+
init_terraform 1>&2
152+
"$verb" "$@"
135153

‎codedeploy/bin/build.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ done
5858

5959
(
6060
cd "$BUILD_DIR"
61-
zip -r "$ARCHIVE" \
61+
zip -q -r "$ARCHIVE" \
6262
appspec.yml \
6363
bin \
6464
ansible \
@@ -73,4 +73,4 @@ docker run --rm -v "$DOCKER_DIR:/src" "$CONTAINERNAME" /bin/bash -c \
7373
"rm -rf /src/venv"
7474

7575
cd "$BUILD_DIR"
76-
aws s3 cp "$ARCHIVE" "s3://$BUCKET/$ARCHIVE"
76+
aws s3 cp "$ARCHIVE" "s3://$BUCKET/$ARCHIVE" --quiet

0 commit comments

Comments
 (0)