A successful Job will most likely read as Failed. #127

Activity

1. kerthcet

   added

   bugCategorizes issue or PR as related to a bug.Categorizes issue or PR as related to a bug.

   on Oct 1, 2026
2. kerthcet commented on Oct 1, 2026

   

   kerthcet

   on Oct 1, 2026

   MemberAuthor

   More actions

   /milestone v0.2.0
3. InftyAI-Agent

   added

   needs-triageIndicates an issue or PR lacks a label and requires one.Indicates an issue or PR lacks a label and requires one.

   needs-priorityIndicates a PR lacks a label and requires one.Indicates a PR lacks a label and requires one.

   on Oct 1, 2026
4. InftyAI-Agent

   added this to the v0.2.0 milestone on Oct 1, 2026
5. kerthcet commented on Oct 1, 2026

   

   kerthcet

   on Oct 1, 2026

   MemberAuthor

   More actions

   /help
6. InftyAI-Agent

   added

   help wantedExtra attention is neededExtra attention is needed

   on Oct 1, 2026
7. Sarthak-Shreshtha01 commented on Oct 1, 2026

   

   Sarthak-Shreshtha01

   on Oct 1, 2026

   Contributor

   More actions

   I skip reaping Pods in the Succeeded phase in the pod placement helpers, so a finished Job no longer reads as Failed. I added a test for it in the pod placement controller tests. (PR: #128)
8. kerthcet commented on Oct 1, 2026

   

   kerthcet

   on Oct 1, 2026

   Last edited by kerthcet

   MemberAuthor

   More actions

   Hi @Sarthak-Shreshtha01 thanks for your interest, I think this is more complex than just skipping reaping Pods.

   • I agree we should not delete the succeeded Pods since the corresponding instances are already terminated when we get the status.
   • Exit codes are not recorded, every terminated instances is marked as Failed, that's the main problem for this issue. We should tell them apart, if exitcode is 0, succeeded, else, failed. Something like this.
   • We should add validation to the Job, only allow restartPolicy == Never, because we don't support restart the container today, we can only recreate the whole Pod which will recreate the instance.
   • When recreating the failed Pods, we should respect the backoffLimit of the Job, once we hit the limit, we should not delete the failed pods as well.

   More advanced:

   • to avoid potential instance leak, once we have succeeded and failed Pods, we may still need to call the termination API but don't delete the Pods, let the Job controller to manage them.

   We may have more gaps to support Job here but we can optimize them step by step.