Skip to content

Quota release when AppWrapper is completed #372

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 29, 2023

Conversation

metalcycling
Copy link
Collaborator

No description provided.

@metalcycling metalcycling requested a review from asm582 May 15, 2023 21:31
@asm582
Copy link
Member

asm582 commented May 15, 2023

Thanks, I think we should release the quota when AW does not have Running or RunningHoldCompletion

@metalcycling
Copy link
Collaborator Author

I've been thinking about this, and I believe the only time the quota should be released is when the AppWrapper is Completed, Preempted, or Failed. Other times can cause problems. For example if the AppWrapper is Dispatched and there is no quota in the tree reserved for that state (because it wasn't Running or RunningHoldCompletion), then the next AppWrapper would be dispatched since the system thinks the quota is free. We should only release quota when we are certain the AppWrapper is done, either by completing, by failing, or by being preempted.

@asm582
Copy link
Member

asm582 commented May 16, 2023

I've been thinking about this, and I believe the only time the quota should be released is when the AppWrapper is Completed, Preempted, or Failed. Other times can cause problems. For example if the AppWrapper is Dispatched and there is no quota in the tree reserved for that state (because it wasn't Running or RunningHoldCompletion), then the next AppWrapper would be dispatched since the system thinks the quota is free. We should only release quota when we are certain the AppWrapper is done, either by completing, by failing, or by being preempted.

Are we not talking about the same thing?

@metalcycling metalcycling marked this pull request as ready for review May 22, 2023 00:45
@metalcycling
Copy link
Collaborator Author

metalcycling commented May 22, 2023

After testing this change for all the AppWrapper conditions (meaning testing quota after triggering any of these conditions):

  • Pending
  • Running
  • Deleted
  • Failed
  • Completed
  • RunningHoldCompletion

I can see quota is being release correctly for all these cases once this code change is added. This ready for review.

@asm582
Copy link
Member

asm582 commented May 22, 2023

After testing this change for all the AppWrapper conditions (meaning testing quota after triggering any of these conditions):

  • Pending
  • Running
  • Deleted
  • Failed
  • Completed
  • RunningHoldCompletion

I can see quota is being release correctly for all these cases once this code change is added. This ready for review.

Thanks, why should the quota be released for RunningHoldCompletion status?

@metalcycling
Copy link
Collaborator Author

I didn't mean it was released for that status. Just that I tested all those cases. I can see how that is confusing. I'll update the text to clarify which is which.

Copy link
Contributor

@z103cb z103cb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good!

@tardieu tardieu mentioned this pull request May 25, 2023
Copy link
Member

@asm582 asm582 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed in the call, as per my understanding the quota is not released when AW is in status Running and RunningHoldCompletion

@z103cb z103cb merged commit ab6d742 into project-codeflare:main May 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants