Skip to content

Commit f7f3e1c

Browse files
dmatch01dmatch01
authored
Quota management porting of initial codebase (#269)
* Move POC rest impl. to quota plubins. * Initial checkin of new quota management. * Partial update to e2e script to enable instatiate 2 differently configured mcad controllers. * Added kuttl installation to e2e test. * Synced version file. * Modified kuttl installation to include sudo for commands. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Added startup of MCAD with Quota Management enabled Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Added kuttl test command to e2e testing. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Provided an alternative to the travis_wait command. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Added full e2e testing back. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Added explicit kuttl config file. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Set e2e test to above if kuttl test fails. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Uptick of release number. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Fix syntax error. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Added creation of test namespace to kuttl testing steps. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Test kuttl failure in Travis run. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Test fix to make e2e test failure in background mode. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Test fix to make e2e test failure in background mode. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Test fix to make e2e test kuttle success run in background mode. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Fix job names to match file names and increased timeout for preemption to happen. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Increase kuttle timeout. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Test fix to make e2e test failure in background mode. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Test fix to make e2e test failure in background mode. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Test fix to make e2e test kuttle success run in background mode. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Use different color code for e2e PID message. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Debugging kuttl failure. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Rearrange setup of kuttl test for more stability of test. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Introduced forced error test to validate new Travis workaround for long running Travis job. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Force final failure test for e2e kuttl test. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Removed forced failure for Travis testing. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Added delay step 3 to give time for MCAD to process subtrees. Signed-off-by: dmatch01 <darroyo@us.ibm.com> * Updated QM test to use Helm3 Signed-off-by: dmatch01 <darroyo@us.ibm.com> --------- Signed-off-by: dmatch01 <darroyo@us.ibm.com>
1 parent ce300d9 commit f7f3e1c

File tree

125 files changed

+10283
-414
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

125 files changed

+10283
-414
lines changed

‎.travis.yml

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ before_script:
2020
- export TEST_LOG_LEVEL=4
2121

2222
script:
23-
# - make
23+
- BLUE='\033[34m'
2424
- make mcad-controller
2525
- make run-test
2626
# Process 'make images' when NOT a pull request
@@ -29,4 +29,11 @@ script:
2929
- 'if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then unset quay_repository && make images; fi'
3030
# Process 'make push-images' when NOT a pull request
3131
- 'if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then make push-images; fi'
32-
- travis_wait 80 make run-e2e
32+
# Run the e2e and handle Travis condition where Travis fails on no output for 10 minutes. No output for 10 minutes
33+
# is fine for our test cases that waits for 10 minutes before requeuing a job. The other workaround was to
34+
# use 'travis_wait n' which unfortunately does not stream output until the very end so monitoring the Travis log
35+
# during runtime is not possible.
36+
- make run-e2e &
37+
- PID=$! && echo -e "${BLUE}make run e2e pid=${PID}"
38+
- while [ -e /proc/${PID} ]; do echo -n "." && sleep 30; done
39+
- wait ${PID}

‎Makefile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ generate-code:
5353
go build -o ${BIN_DIR}/deepcopy-gen ./cmd/deepcopy-gen/
5454
$(info Generating deepcopy...)
5555
${BIN_DIR}/deepcopy-gen -i ./pkg/apis/controller/v1beta1/ -O zz_generated.deepcopy
56+
${BIN_DIR}/deepcopy-gen -i ./pkg/apis/quotaplugins/quotasubtree/v1 -O zz_generated.deepcopy
5657

5758
images: verify-tag-name
5859
$(info List executable directory)

‎config/crd/bases/ibm.com_quotasubtree-v1.yaml

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
apiVersion: apiextensions.k8s.io/v1
2+
kind: CustomResourceDefinition
3+
metadata:
4+
name: quotasubtrees.ibm.com
5+
finalizers: []
6+
spec:
7+
group: ibm.com
8+
scope: Namespaced
9+
names:
10+
kind: QuotaSubtree
11+
singular: quotasubtree
12+
plural: quotasubtrees
13+
shortNames:
14+
- qst
15+
versions:
16+
- name: v1
17+
served: true
18+
storage: true
19+
subresources:
20+
status: {}
21+
schema:
22+
openAPIV3Schema:
23+
type: object
24+
properties:
25+
status:
26+
type: object
27+
x-kubernetes-preserve-unknown-fields: true
28+
spec:
29+
type: object
30+
properties:
31+
parent:
32+
type: string
33+
parentNamespace:
34+
type: string
35+
children:
36+
type: array
37+
items:
38+
type: object
39+
properties:
40+
name:
41+
type: string
42+
namespace:
43+
type: string
44+
quotas:
45+
type: object
46+
properties:
47+
disabled:
48+
type: boolean
49+
hardLimit:
50+
type: boolean
51+
requests:
52+
type: object
53+
properties:
54+
cpu:
55+
x-kubernetes-int-or-string: true
56+
pattern: '^[0-9]*(m)*$'
57+
memory:
58+
x-kubernetes-int-or-string: true
59+
pattern: '^[0-9]*(Ei|Pi|Ti|Gi|Mi|Ki|E|P|T|G|M|K)*$'
60+
nvidia.com/gpu:
61+
x-kubernetes-int-or-string: true
62+
pattern: '^[0-9]*$'

‎deployment/mcad-controller/templates/deployment.yaml

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,69 @@ subjects:
102102
---
103103
apiVersion: apiextensions.k8s.io/v1
104104
kind: CustomResourceDefinition
105+
metadata:
106+
name: quotasubtrees.ibm.com
107+
finalizers: []
108+
spec:
109+
group: ibm.com
110+
scope: Namespaced
111+
names:
112+
kind: QuotaSubtree
113+
singular: quotasubtree
114+
plural: quotasubtrees
115+
shortNames:
116+
- qst
117+
versions:
118+
- name: v1
119+
served: true
120+
storage: true
121+
subresources:
122+
status: {}
123+
schema:
124+
openAPIV3Schema:
125+
type: object
126+
properties:
127+
status:
128+
type: object
129+
x-kubernetes-preserve-unknown-fields: true
130+
spec:
131+
type: object
132+
properties:
133+
parent:
134+
type: string
135+
parentNamespace:
136+
type: string
137+
children:
138+
type: array
139+
items:
140+
type: object
141+
properties:
142+
name:
143+
type: string
144+
namespace:
145+
type: string
146+
quotas:
147+
type: object
148+
properties:
149+
disabled:
150+
type: boolean
151+
hardLimit:
152+
type: boolean
153+
requests:
154+
type: object
155+
properties:
156+
cpu:
157+
x-kubernetes-int-or-string: true
158+
pattern: '^[0-9]*(m)*$'
159+
memory:
160+
x-kubernetes-int-or-string: true
161+
pattern: '^[0-9]*(Ei|Pi|Ti|Gi|Mi|Ki|E|P|T|G|M|K)*$'
162+
nvidia.com/gpu:
163+
x-kubernetes-int-or-string: true
164+
pattern: '^[0-9]*$'
165+
---
166+
apiVersion: apiextensions.k8s.io/v1
167+
kind: CustomResourceDefinition
105168
metadata:
106169
annotations:
107170
controller-gen.kubebuilder.io/version: v0.4.1

‎hack/boilerplate/boilerplate.go.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
1414
limitations under the License.
1515
*/
1616
/*
17-
Copyright 2019, 2021 The Multi-Cluster App Dispatcher Authors.
17+
Copyright 2019, 2021, 2022, YEAR The Multi-Cluster App Dispatcher Authors.
1818

1919
Licensed under the Apache License, Version 2.0 (the "License");
2020
you may not use this file except in compliance with the License.

‎hack/run-e2e-kind.sh

Lines changed: 79 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -41,18 +41,25 @@ export IMAGE_REPOSITORY_MCAD="${1}"
4141
export IMAGE_TAG_MCAD="${2}"
4242
export MCAD_IMAGE_PULL_POLICY="${3-Always}"
4343
export IMAGE_MCAD="${IMAGE_REPOSITORY_MCAD}:${IMAGE_TAG_MCAD}"
44+
export KUTTL_VERSION=0.15.0
45+
export KUTTL_TEST_OPT="--config ${ROOT_DIR}/kuttl-test.yaml"
46+
# FOR DEBUGGING
47+
#export KUTTL_TEST_OPT="--config ${ROOT_DIR}/kuttl-test.yaml --skip-delete"
4448

4549
sudo apt-get update && sudo apt-get install -y apt-transport-https
4650
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
4751
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
4852
sudo apt-get update
4953
# Using older version due to older version of kubernetes cluster"
50-
sudo apt-get install -y --allow-unauthenticated kubectl=1.17.0-00
54+
sudo apt-get install -y --allow-unauthenticated kubectl=1.17.0-00
5155

5256
# Download kind binary (0.6.1)
5357
sudo curl -o /usr/local/bin/kind -L https://github.com/kubernetes-sigs/kind/releases/download/v0.11.0/kind-linux-amd64
5458
sudo chmod +x /usr/local/bin/kind
5559

60+
# Download kuttl plugin
61+
sudo curl -sSLf --output /tmp/kubectl-kuttl https://github.com/kudobuilder/kuttl/releases/download/v${KUTTL_VERSION}/kubectl-kuttl_${KUTTL_VERSION}_linux_x86_64 && sudo mv /tmp/kubectl-kuttl /usr/local/bin && sudo chmod a+x /usr/local/bin/kubectl-kuttl
62+
5663
# check if kind installed
5764
function check-prerequisites {
5865
echo "checking prerequisites"
@@ -71,7 +78,15 @@ function check-prerequisites {
7178
else
7279
echo -n "found kubectl, " && kubectl version --short --client
7380
fi
74-
81+
82+
kubectl kuttl version >/dev/null 2>&1
83+
if [[ $? -ne 0 ]]; then
84+
echo "kuttl plugin for kubectl not installed, exiting."
85+
exit 1
86+
else
87+
echo -n "found kuttl plugin for kubectl, " && kubectl kuttl version
88+
fi
89+
7590
if [[ $IMAGE_REPOSITORY_MCAD == "" ]]
7691
then
7792
echo "No MCAD image was provided."
@@ -289,16 +304,49 @@ function kube-test-env-up {
289304

290305
echo "Installing Podgroup CRD"
291306

307+
helm version
308+
309+
# Turn off master taints
310+
kubectl taint nodes --all node-role.kubernetes.io/master-
311+
312+
}
313+
314+
function mcad-quota-management-up {
315+
cd $ROOT_DIR/deployment
316+
317+
# start mcad controller
318+
echo "Starting MCAD Controller for Quota Management Testing..."
319+
echo "helm upgrade --install mcad-controller . namespace kube-system wait set loglevel=10 set resources.requests.cpu=1000m set resources.requests.memory=1024Mi set resources.limits.cpu=4000m set resources.limits.memory=4096Mi set image.repository=$IMAGE_REPOSITORY_MCAD set image.tag=$IMAGE_TAG_MCAD set image.pullPolicy=$MCAD_IMAGE_PULL_POLICY set configMap.quotaEnabled='true' set quotaManagement.rbac.apiGroup=ibm.com set quotaManagement.rbac.resource=quotasubtrees set configMap.name=mcad-controller-configmap set configMap.preemptionEnabled='true'"
320+
helm upgrade --install mcad-controller . --namespace kube-system --wait --set loglevel=10 --set resources.requests.cpu=1000m --set resources.requests.memory=1024Mi --set resources.limits.cpu=4000m --set resources.limits.memory=4096Mi --set image.repository=$IMAGE_REPOSITORY_MCAD --set image.tag=$IMAGE_TAG_MCAD --set image.pullPolicy=$MCAD_IMAGE_PULL_POLICY --set configMap.quotaEnabled='"true"' --set quotaManagement.rbac.apiGroup=ibm.com --set quotaManagement.rbac.resource=quotasubtrees --set configMap.name=mcad-controller-configmap --set configMap.preemptionEnabled='"true"'
321+
sleep 10
322+
}
323+
324+
function mcad-quota-management-down {
325+
326+
# Helm chart install name
327+
helm_chart_name=$(helm list --short)
328+
329+
# start mcad controller
330+
echo "Stopping MCAD Controller for Quota Management Testing..."
331+
echo "helm delete ${helm_chart_name}"
332+
helm delete ${helm_chart_name}
333+
sleep 20
334+
}
335+
336+
function mcad-up {
292337
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/scheduler-plugins/277b6bdec18f8a9e9ccd1bfeaf4b66495bfc6f92/config/crd/bases/scheduling.sigs.k8s.io_podgroups.yaml
293338

294-
cd deployment/mcad-controller
339+
cd $ROOT_DIR/deployment/mcad-controller
295340

296341
# start mcad controller
297342
echo "Starting MCAD Controller..."
298343
echo "helm install mcad-controller namespace kube-system wait set loglevel=2 set resources.requests.cpu=1000m set resources.requests.memory=1024Mi set resources.limits.cpu=4000m set resources.limits.memory=4096Mi set image.repository=$IMAGE_REPOSITORY_MCAD set image.tag=$IMAGE_TAG_MCAD set image.pullPolicy=$MCAD_IMAGE_PULL_POLICY"
299344
helm upgrade --install mcad-controller . --namespace kube-system --wait --set loglevel=2 --set resources.requests.cpu=1000m --set resources.requests.memory=1024Mi --set resources.limits.cpu=4000m --set resources.limits.memory=4096Mi --set configMap.name=mcad-controller-configmap --set configMap.podCreationTimeout='"120000"' --set configMap.quotaEnabled='"false"' --set coscheduler.rbac.apiGroup=scheduling.sigs.k8s.io --set coscheduler.rbac.resource=podgroups --set image.repository=$IMAGE_REPOSITORY_MCAD --set image.tag=$IMAGE_TAG_MCAD --set image.pullPolicy=$MCAD_IMAGE_PULL_POLICY
300345

301346
sleep 10
347+
}
348+
349+
function mcad-env-status {
302350
echo "Listing MCAD Controller Helm Chart and Pod YAML..."
303351
helm list
304352
mcad_pod=$(kubectl get pods -n kube-system | grep mcad-controller | awk '{print $1}')
@@ -307,7 +355,6 @@ function kube-test-env-up {
307355
kubectl get pod ${mcad_pod} -n kube-system -o yaml
308356
fi
309357

310-
311358
sleep 10
312359
echo "Listing MCAD Controller Helm Chart and Pod YAML..."
313360
helm list
@@ -317,10 +364,6 @@ function kube-test-env-up {
317364
kubectl get pod ${mcad_pod} -n kube-system -o yaml
318365
fi
319366

320-
# Turn off master taints
321-
kubectl taint nodes --all node-role.kubernetes.io/master-
322-
323-
324367
# This is meant to orchestrate initial cluster configuration such that accounting tests can be consistent
325368
echo "---"
326369
echo "Orchestrate cluster..."
@@ -341,13 +384,41 @@ function kube-test-env-up {
341384
kubectl describe nodes
342385
}
343386

387+
function kuttl-tests {
388+
mcad-quota-management-up
389+
mcad-env-status
390+
cd ${ROOT_DIR}
391+
echo "==============>>>>> Running Quota Management Kuttl E2E tests... <<<<<=============="
392+
echo "kubectl kuttl test ${KUTTL_TEST_OPT}"
393+
kubectl kuttl test ${KUTTL_TEST_OPT}
394+
if [[ $? -ne 0 ]]; then
395+
echo "quota management kuttl e2e tests failure, exiting."
396+
exit 1
397+
else
398+
# Takes a bit of time for namespace created in kuttl testing to completely delete.
399+
sleep 40
400+
fi
401+
mcad-quota-management-down
402+
}
344403

345404
trap cleanup EXIT
346405

347406
kind-up-cluster
348407

349408
kube-test-env-up
350409

410+
###
411+
# Quota management testing
412+
###
413+
kuttl-tests
414+
415+
###
416+
# Non-quota management testing
417+
###
418+
mcad-up
419+
420+
mcad-env-status
421+
351422
cd ${ROOT_DIR}
352423

353424
echo "==========================>>>>> Running E2E tests... <<<<<=========================="

‎kuttl-test.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
apiVersion: kuttl.dev/v1beta1
2+
kind: TestSuite
3+
testDirs:
4+
- ./test/e2e-kuttl/
5+
timeout: 180

0 commit comments

Comments
 (0)