[quotamanagement] No preemption for borrowed resource when parent has hardLimit=true

[KPostOffice]

When the parent node has hardLimit: true the child nodes won't preempt one another for resources that have been borrowed

Files used

testing-forest.yaml

apiVersion: ibm.com/v1
kind: QuotaSubtree
metadata:
  name: context-root
  namespace: kube-system
  labels:
    tree: quota_context
spec:
  children:
    - name: context-root
      quotas:
        hardLimit: true
        requests:
          cpu: 2000
          memory: 8000Mi
---
apiVersion: ibm.com/v1
kind: QuotaSubtree
metadata:
  name: context-root-children
  namespace: kube-system
  labels:
    tree: quota_context
spec:
  parent: context-root
  children:
    - name: alpha
      quotas:
        hardLimit: false
        requests:
          cpu: 1000
          memory: 4000Mi
    - name: beta
      quotas:
        requests:
          cpu: 1000
          memory: 4000Mi

---

borrower.yaml

apiVersion: mcad.ibm.com/v1beta1
kind: AppWrapper
metadata:
  name: borrower
  namespace: test1
  labels:
    quota_context: "alpha"
spec:
  priority: 100
  service:
    spec: {}
  resources:
    metadata: {}
    GenericItems:
      - metadata: {}
        completionstatus: Complete
        replicas: 1
        custompodresources:
        - replicas:  1
          requests:
            cpu: 2000m
            memory: 8000Mi
          limits:
            cpu: 2000m
            memory: 8000Mi
        generictemplate:
          apiVersion: batch/v1
          kind: Job
          metadata:
            name: borrower
            namespace: test1
            labels:
              app: borrower
          spec:
            parallelism: 1
            completions:
            template:
              metadata:
                labels:
                  app: borrower
                  size: "1"
              spec:
                restartPolicy: Never
                containers:
                  - name: borrower
                    image: registry.access.redhat.com/ubi8/ubi:latest
                    command:
                      - /bin/sh
                      - -c
                      - while true; do sleep 10; done
                    resources:
                      requests:
                        cpu: "2000m"
                        memory: "8000Mi"
                      limits:
                        cpu: "2000m"
                        memory: "8000Mi"

---

preempter.yaml

apiVersion: mcad.ibm.com/v1beta1
kind: AppWrapper
metadata:
  name: preempter
  namespace: test1
  labels:
    quota_context: "beta"
spec:
  priority: 100
  service:
    spec: {}
  resources:
    metadata: {}
    GenericItems:
      - metadata: {}
        completionstatus: Complete
        replicas: 1
        custompodresources:
        - replicas:  1
          requests:
            cpu: 750m
            memory: 3750Mi
          limits:
            cpu: 750m
            memory: 3750Mi
        generictemplate:
          apiVersion: batch/v1
          kind: Job
          metadata:
            name: preempter
            namespace: test1
            labels:
              app: preempter
          spec:
            parallelism: 1
            completions: 1
            template:
              metadata:
                labels:
                  app: preempter
                  size: "1"
              spec:
                restartPolicy: Never
                containers:
                  - name: preempter
                    image: registry.access.redhat.com/ubi8/ubi:latest
                    command:
                      - /bin/sh
                      - -c
                      - while true; do sleep 10; done
                    resources:
                      requests:
                        cpu: "750m"
                        memory: "3750Mi"
                      limits:
                        cpu: "750m"
                        memory: "3750Mi"

Result

After applying testing-forest.yaml and then borrower.yaml I waited for the borrower pod to come up. Then I applied the preempter.yaml. The preempter.yaml failed with the following status:

Status:
  Conditions:
    Last Transition Micro Time:  2023-05-17T19:49:10.330321Z
    Last Update Micro Time:      2023-05-17T19:49:10.330321Z
    Status:                      True
    Type:                        Init
    Last Transition Micro Time:  2023-05-17T19:49:10.330450Z
    Last Update Micro Time:      2023-05-17T19:49:10.330450Z
    Reason:                      AwaitingHeadOfLine
    Status:                      True
    Type:                        Queueing
    Last Transition Micro Time:  2023-05-17T19:49:10.346083Z
    Last Update Micro Time:      2023-05-17T19:49:10.346083Z
    Reason:                      FrontOfQueue.
    Status:                      True
    Type:                        HeadOfLine
    Last Transition Micro Time:  2023-05-17T19:49:10.359413Z
    Last Update Micro Time:      2023-05-17T19:49:10.359413Z
    Message:                     Insufficient quota to dispatch AppWrapper.
    Reason:                      AppWrapperNotRunnable.  Failed to allocate quota on quota designation 'quota_context'
    Status:                      True
    Type:                        Backoff

Expected

The expected behavior is that the preempter appwrapper causes the preemption of the borrower and keeps the borrower from running.