Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

onExit steps executed after deadline exceeded are stopped if an onExit pod spends >=10 seconds in Pending state #13060

Open
3 of 4 tasks
encigem opened this issue May 16, 2024 · 4 comments · May be fixed by #13016
Open
3 of 4 tasks

onExit steps executed after deadline exceeded are stopped if an onExit pod spends >=10 seconds in Pending state #13060

encigem opened this issue May 16, 2024 · 4 comments · May be fixed by #13016
Assignees
@jswxstw
Labels
area/controller Controller issues, panics area/exit-handler type/bug

Comments

@encigem
Copy link

encigem commented May 16, 2024

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

Expected Behavior: On timeout of a workflow, the onExit steps are expected to ignore the deadline and execute after the main steps are stopped.

Actual Behavior: On timeout of a workflow, if an onExit step takes 10 seconds or more to go from a Pending state to a Running state (default requeue time of 10 seconds is configured in the workflow-controller deployment), then the onExit step will fail immediately with message "Step exceeded its deadline" and the remaining onExit steps are not executed.

Link to original message on Slack: https://cloud-native.slack.com/archives/C01QW9QSSSK/p1715787843656389
This issue has been verified to also occur on latest v3.5.6 Argo-Workflows, by another user who has replied to the above Slack message.

The 10 second limit for onExit pods, after deadline is reached, seems to be tied to the controller environment variable DEFAULT_REQUEUE_TIME as i can get the issue to occur more frequently with lower requeue time values or less frequently with higher requeue time values applied to the workflow-controller deployment.

Version

v3.4.10

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: exit-test-
spec:
  entrypoint: main
  onExit: exit-handler
  activeDeadlineSeconds: 20
  arguments:
    parameters:
      - name: test-image
        value: "<some image>"
      - name: test-image2
        # define a large image here to get exit-handler pod to spend
        # at least 10 seconds in Pending state while it tries to pull
        # the image
        value: "<some larger image>"

  templates:
    - name: main
      steps:
      - - name: main-step
          template: test

    - name: exit-handler
      steps:
      - - name: exit-step1
          template: test2
      - - name: exit-step2
          template: test2
      - - name: exit-step3
          template: test2

    # sleeps to trigger the deadline and cause workflow timeout
    - name: test
      container:
        image: "{{workflow.parameters.test-image}}"
        command:
          - sh
          - -c
        args:
          - |
            sleep 100

    # fails if it spends >=10s in Pending state after deadline reached
    - name: test2
      container:
        image: "{{workflow.parameters.test-image2}}"
        command:
          - sh
          - -c
        args:
          - |
            echo "exit handler executed!"

Logs from the workflow controller

time="2024-05-16T17:33:23.063Z" level=info msg="Processing workflow" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.064Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:23.064Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:23.065Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:23.066Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:23.066Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:23.066Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.066Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.066Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.066Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.066Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.066Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.066Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:23.075Z" level=info msg="Updated phase  -> Running" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.075Z" level=debug msg="Evaluating node exit-test-xpqdl: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.075Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:23.075Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:23.075Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:23.075Z" level=debug msg="Initializing node exit-test-xpqdl: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.075Z" level=info msg="Steps node exit-test-xpqdl initialized Running" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.075Z" level=debug msg="Initializing node exit-test-xpqdl[0]: template: *v1alpha1.WorkflowStep invalid (https://argoproj.github.io/argo-workflows/templates/), boundaryID: exit-test-xpqdl" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.075Z" level=info msg="StepGroup node exit-test-xpqdl-4203995792 initialized Running" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.075Z" level=debug msg="Evaluating node exit-test-xpqdl[0].main-step: template: *v1alpha1.WorkflowStep (test), boundaryID: exit-test-xpqdl" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.075Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.075Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.075Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:23.075Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:23.075Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:23.075Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:23.075Z" level=debug msg="Initializing node exit-test-xpqdl[0].main-step: template: *v1alpha1.WorkflowStep (test), boundaryID: exit-test-xpqdl" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.075Z" level=info msg="Pod node exit-test-xpqdl-3770758008 initialized Pending" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.075Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:23.076Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:23.076Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:23.076Z" level=debug msg="Executing node exit-test-xpqdl[0].main-step with container template: test\n" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.076Z" level=debug namespace=project-il needLocation=true workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.076Z" level=debug msg="Creating Pod: exit-test-xpqdl[0].main-step (exit-test-xpqdl-test-3770758008)" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.077Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064036\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowRunning' Workflow Running"
time="2024-05-16T17:33:23.098Z" level=info msg="Created pod: exit-test-xpqdl[0].main-step (exit-test-xpqdl-test-3770758008)" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.098Z" level=info msg="Workflow step group node exit-test-xpqdl-4203995792 not yet completed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.098Z" level=info msg="TaskSet Reconciliation" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.098Z" level=info msg=reconcileAgentPod namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.099Z" level=debug msg="Log changes patch: {\"metadata\":{\"annotations\":{\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"labels\":{\"workflows.argoproj.io/phase\":\"Running\"}},\"spec\":{\"ttlStrategy\":{\"secondsAfterCompletion\":259200,\"secondsAfterFailure\":604800,\"secondsAfterSuccess\":864000}},\"status\":{\"artifactGCStatus\":{\"notSpecified\":true},\"artifactRepositoryRef\":{\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"accessKeySecret\":{\"key\":\"s3_access_key\",\"name\":\"user-s3-auth\"},\"bucket\":\"projectdev-cluster01\",\"endpoint\":\"seliis3obj.org:10444\",\"insecure\":false,\"keyFormat\":\"artifacts/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{workflow.name}}/{{pod.name}}\",\"secretKeySecret\":{\"key\":\"s3_secret_key\",\"name\":\"user-s3-auth\"}}},\"default\":true},\"nodes\":{\"exit-test-xpqdl\":{\"children\":[\"exit-test-xpqdl-4203995792\"],\"displayName\":\"exit-test-xpqdl\",\"finishedAt\":null,\"id\":\"exit-test-xpqdl\",\"name\":\"exit-test-xpqdl\",\"phase\":\"Running\",\"startedAt\":\"2024-05-16T17:33:23Z\",\"templateName\":\"main\",\"templateScope\":\"local/exit-test-xpqdl\",\"type\":\"Steps\"},\"exit-test-xpqdl-3770758008\":{\"boundaryID\":\"exit-test-xpqdl\",\"displayName\":\"main-step\",\"finishedAt\":null,\"id\":\"exit-test-xpqdl-3770758008\",\"name\":\"exit-test-xpqdl[0].main-step\",\"phase\":\"Pending\",\"startedAt\":\"2024-05-16T17:33:23Z\",\"templateName\":\"test\",\"templateScope\":\"local/exit-test-xpqdl\",\"type\":\"Pod\"},\"exit-test-xpqdl-4203995792\":{\"boundaryID\":\"exit-test-xpqdl\",\"children\":[\"exit-test-xpqdl-3770758008\"],\"displayName\":\"[0]\",\"finishedAt\":null,\"id\":\"exit-test-xpqdl-4203995792\",\"name\":\"exit-test-xpqdl[0]\",\"phase\":\"Running\",\"startedAt\":\"2024-05-16T17:33:23Z\",\"templateScope\":\"local/exit-test-xpqdl\",\"type\":\"StepGroup\"}},\"phase\":\"Running\",\"startedAt\":\"2024-05-16T17:33:23Z\"}}"
time="2024-05-16T17:33:23.124Z" level=info msg="Workflow update successful" namespace=project-il phase=Running resourceVersion=640064041 workflow=exit-test-xpqdl
time="2024-05-16T17:33:23.124Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064041\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node exit-test-xpqdl"
time="2024-05-16T17:33:23.124Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064041\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node exit-test-xpqdl[0]"
time="2024-05-16T17:33:26.101Z" level=info msg="Processing workflow" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:26.102Z" level=info msg="Task-result reconciliation" namespace=project-il numObjs=0 workflow=exit-test-xpqdl
time="2024-05-16T17:33:26.102Z" level=info msg="node changed" namespace=project-il new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=exit-test-xpqdl-3770758008 old.message= old.phase=Pending old.progress=0/1 workflow=exit-test-xpqdl
time="2024-05-16T17:33:26.103Z" level=debug msg="Evaluating node exit-test-xpqdl: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:26.103Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:26.103Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:26.103Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:26.103Z" level=debug msg="Executing node exit-test-xpqdl of Steps is Running" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:26.103Z" level=debug msg="Evaluating node exit-test-xpqdl[0].main-step: template: *v1alpha1.WorkflowStep (test), boundaryID: exit-test-xpqdl" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:26.103Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:26.103Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:26.103Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:26.103Z" level=debug msg="Executing node exit-test-xpqdl[0].main-step of Pod is Pending" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:26.103Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:26.103Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:26.103Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:26.103Z" level=debug msg="Executing node exit-test-xpqdl[0].main-step with container template: test\n" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:26.103Z" level=debug msg="Skipped pod exit-test-xpqdl[0].main-step (exit-test-xpqdl-3770758008) creation: already exists" namespace=project-il podPhase=Pending workflow=exit-test-xpqdl
time="2024-05-16T17:33:26.103Z" level=info msg="Workflow step group node exit-test-xpqdl-4203995792 not yet completed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:26.103Z" level=info msg="TaskSet Reconciliation" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:26.103Z" level=info msg=reconcileAgentPod namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:26.104Z" level=debug msg="Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"}],\"nodes\":{\"exit-test-xpqdl-3770758008\":{\"hostNodeName\":\"node-10-120-81-51\",\"message\":\"PodInitializing\"}}}}"
time="2024-05-16T17:33:26.125Z" level=info msg="Workflow update successful" namespace=project-il phase=Running resourceVersion=640064077 workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.126Z" level=info msg="Processing workflow" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.127Z" level=info msg="Task-result reconciliation" namespace=project-il numObjs=0 workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.127Z" level=info msg="node changed" namespace=project-il new.message= new.phase=Running new.progress=0/1 nodeID=exit-test-xpqdl-3770758008 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.127Z" level=debug msg="Evaluating node exit-test-xpqdl: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.127Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:29.127Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:29.127Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:29.127Z" level=debug msg="Executing node exit-test-xpqdl of Steps is Running" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.127Z" level=debug msg="Evaluating node exit-test-xpqdl[0].main-step: template: *v1alpha1.WorkflowStep (test), boundaryID: exit-test-xpqdl" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.127Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:29.127Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:29.127Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:29.127Z" level=debug msg="Executing node exit-test-xpqdl[0].main-step of Pod is Running" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.128Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:29.128Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:29.128Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:29.128Z" level=debug msg="Executing node exit-test-xpqdl[0].main-step with container template: test\n" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.128Z" level=debug msg="Skipped pod exit-test-xpqdl[0].main-step (exit-test-xpqdl-3770758008) creation: already exists" namespace=project-il podPhase=Running workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.128Z" level=info msg="Workflow step group node exit-test-xpqdl-4203995792 not yet completed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.128Z" level=info msg="TaskSet Reconciliation" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.128Z" level=info msg=reconcileAgentPod namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.128Z" level=debug msg="Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"True\",\"type\":\"PodRunning\"}],\"nodes\":{\"exit-test-xpqdl-3770758008\":{\"message\":null,\"phase\":\"Running\"}}}}"
time="2024-05-16T17:33:29.150Z" level=info msg="Workflow update successful" namespace=project-il phase=Running resourceVersion=640064096 workflow=exit-test-xpqdl
time="2024-05-16T17:33:29.150Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064096\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node exit-test-xpqdl[0].main-step"
time="2024-05-16T17:33:32.151Z" level=info msg="Processing workflow" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:32.152Z" level=info msg="Task-result reconciliation" namespace=project-il numObjs=0 workflow=exit-test-xpqdl
time="2024-05-16T17:33:32.152Z" level=info msg="node unchanged" namespace=project-il nodeID=exit-test-xpqdl-3770758008 workflow=exit-test-xpqdl
time="2024-05-16T17:33:32.152Z" level=debug msg="Evaluating node exit-test-xpqdl: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:32.152Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:32.152Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:32.152Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:32.152Z" level=debug msg="Executing node exit-test-xpqdl of Steps is Running" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:32.152Z" level=debug msg="Evaluating node exit-test-xpqdl[0].main-step: template: *v1alpha1.WorkflowStep (test), boundaryID: exit-test-xpqdl" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:32.152Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:32.152Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:32.152Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:32.152Z" level=debug msg="Executing node exit-test-xpqdl[0].main-step of Pod is Running" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:32.153Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:32.153Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:32.153Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2024-05-16T17:33:32.153Z" level=debug msg="Executing node exit-test-xpqdl[0].main-step with container template: test\n" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:32.153Z" level=debug msg="Skipped pod exit-test-xpqdl[0].main-step (exit-test-xpqdl-3770758008) creation: already exists" namespace=project-il podPhase=Running workflow=exit-test-xpqdl
time="2024-05-16T17:33:32.153Z" level=info msg="Workflow step group node exit-test-xpqdl-4203995792 not yet completed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:32.153Z" level=info msg="TaskSet Reconciliation" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:32.153Z" level=info msg=reconcileAgentPod namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.774Z" level=info msg="Processing workflow" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.774Z" level=info msg="Task-result reconciliation" namespace=project-il numObjs=0 workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=info msg="Pod failed: Pod was active on the node longer than the specified deadline" displayName=main-step namespace=project-il pod=exit-test-xpqdl-test-3770758008 templateName=test workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=info msg="node changed" namespace=project-il new.message="Pod was active on the node longer than the specified deadline" new.phase=Failed new.progress=0/1 nodeID=exit-test-xpqdl-3770758008 old.message= old.phase=Running old.progress=0/1 workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=debug msg="Evaluating node exit-test-xpqdl: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:55.775Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:55.775Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:55.775Z" level=debug msg="Executing node exit-test-xpqdl of Steps is Running" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=debug msg="Evaluating node exit-test-xpqdl[0].main-step: template: *v1alpha1.WorkflowStep (test), boundaryID: exit-test-xpqdl" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:55.775Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:55.775Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test)"
time="2024-05-16T17:33:55.775Z" level=debug msg="Node exit-test-xpqdl[0].main-step already completed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=info msg="Step group node exit-test-xpqdl-4203995792 deemed failed: child 'exit-test-xpqdl-3770758008' failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=info msg="node exit-test-xpqdl-4203995792 phase Running -> Failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=info msg="node exit-test-xpqdl-4203995792 message: child 'exit-test-xpqdl-3770758008' failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=info msg="node exit-test-xpqdl-4203995792 finished: 2024-05-16 17:33:55.775908404 +0000 UTC" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=info msg="step group exit-test-xpqdl-4203995792 was unsuccessful: child 'exit-test-xpqdl-3770758008' failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (test)"
time="2024-05-16T17:33:55.775Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (test)"
time="2024-05-16T17:33:55.775Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (test)"
time="2024-05-16T17:33:55.775Z" level=info msg="Outbound nodes of exit-test-xpqdl-3770758008 is [exit-test-xpqdl-3770758008]" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.775Z" level=info msg="Outbound nodes of exit-test-xpqdl is [exit-test-xpqdl-3770758008]" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=info msg="node exit-test-xpqdl phase Running -> Failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=info msg="node exit-test-xpqdl message: child 'exit-test-xpqdl-3770758008' failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=info msg="node exit-test-xpqdl finished: 2024-05-16 17:33:55.776025215 +0000 UTC" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=debug msg="Checking daemoned children of exit-test-xpqdl" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=info msg="TaskSet Reconciliation" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=info msg=reconcileAgentPod namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=info msg="Running OnExit handler: exit-handler" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=debug msg="Evaluating node exit-test-xpqdl.onExit: template: *v1alpha1.WorkflowStep (exit-handler), boundaryID: " namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:55.776Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:55.776Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:55.776Z" level=debug msg="Initializing node exit-test-xpqdl.onExit: template: *v1alpha1.WorkflowStep (exit-handler), boundaryID: " namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=info msg="Steps node exit-test-xpqdl-1312219273 initialized Running" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=debug msg="Initializing node exit-test-xpqdl.onExit[0]: template: *v1alpha1.WorkflowStep invalid (https://argoproj.github.io/argo-workflows/templates/), boundaryID: exit-test-xpqdl-1312219273" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=info msg="StepGroup node exit-test-xpqdl-1058664605 initialized Running" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=debug msg="Evaluating node exit-test-xpqdl.onExit[0].exit-step: template: *v1alpha1.WorkflowStep (test2), boundaryID: exit-test-xpqdl-1312219273" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:55.776Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:55.776Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:55.776Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (exit-handler)"
time="2024-05-16T17:33:55.776Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (exit-handler)"
time="2024-05-16T17:33:55.776Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (exit-handler)"
time="2024-05-16T17:33:55.776Z" level=debug msg="Initializing node exit-test-xpqdl.onExit[0].exit-step: template: *v1alpha1.WorkflowStep (test2), boundaryID: exit-test-xpqdl-1312219273" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=info msg="Pod node exit-test-xpqdl-1805001630 initialized Pending" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (exit-handler)"
time="2024-05-16T17:33:55.776Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (exit-handler)"
time="2024-05-16T17:33:55.776Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (exit-handler)"
time="2024-05-16T17:33:55.776Z" level=debug msg="Executing node exit-test-xpqdl.onExit[0].exit-step with container template: test2\n" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.776Z" level=debug namespace=project-il needLocation=true workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.777Z" level=debug msg="Creating Pod: exit-test-xpqdl.onExit[0].exit-step (exit-test-xpqdl-test2-1805001630)" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.801Z" level=info msg="Created pod: exit-test-xpqdl.onExit[0].exit-step (exit-test-xpqdl-test2-1805001630)" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.801Z" level=info msg="Workflow step group node exit-test-xpqdl-1058664605 not yet completed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.802Z" level=debug msg="Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"}],\"nodes\":{\"exit-test-xpqdl\":{\"finishedAt\":\"2024-05-16T17:33:55Z\",\"message\":\"child 'exit-test-xpqdl-3770758008' failed\",\"outboundNodes\":[\"exit-test-xpqdl-3770758008\"],\"phase\":\"Failed\"},\"exit-test-xpqdl-1058664605\":{\"boundaryID\":\"exit-test-xpqdl-1312219273\",\"children\":[\"exit-test-xpqdl-1805001630\"],\"displayName\":\"[0]\",\"finishedAt\":null,\"id\":\"exit-test-xpqdl-1058664605\",\"name\":\"exit-test-xpqdl.onExit[0]\",\"phase\":\"Running\",\"startedAt\":\"2024-05-16T17:33:55Z\",\"templateScope\":\"local/exit-test-xpqdl\",\"type\":\"StepGroup\"},\"exit-test-xpqdl-1312219273\":{\"children\":[\"exit-test-xpqdl-1058664605\"],\"displayName\":\"exit-test-xpqdl.onExit\",\"finishedAt\":null,\"id\":\"exit-test-xpqdl-1312219273\",\"name\":\"exit-test-xpqdl.onExit\",\"phase\":\"Running\",\"startedAt\":\"2024-05-16T17:33:55Z\",\"templateName\":\"exit-handler\",\"templateScope\":\"local/exit-test-xpqdl\",\"type\":\"Steps\"},\"exit-test-xpqdl-1805001630\":{\"boundaryID\":\"exit-test-xpqdl-1312219273\",\"displayName\":\"exit-step\",\"finishedAt\":null,\"id\":\"exit-test-xpqdl-1805001630\",\"name\":\"exit-test-xpqdl.onExit[0].exit-step\",\"phase\":\"Pending\",\"startedAt\":\"2024-05-16T17:33:55Z\",\"templateName\":\"test2\",\"templateScope\":\"local/exit-test-xpqdl\",\"type\":\"Pod\"},\"exit-test-xpqdl-3770758008\":{\"finishedAt\":\"2024-05-16T17:33:53Z\",\"message\":\"Pod was active on the node longer than the specified deadline\",\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/2024/05/16/exit-test-xpqdl/exit-test-xpqdl-test-3770758008/main.log\"}}],\"exitCode\":\"143\"},\"phase\":\"Failed\",\"resourcesDuration\":{\"cpu\":57,\"memory\":57}},\"exit-test-xpqdl-4203995792\":{\"finishedAt\":\"2024-05-16T17:33:55Z\",\"message\":\"child 'exit-test-xpqdl-3770758008' failed\",\"phase\":\"Failed\"}}}}"
time="2024-05-16T17:33:55.834Z" level=info msg="Workflow update successful" namespace=project-il phase=Running resourceVersion=640064257 workflow=exit-test-xpqdl
time="2024-05-16T17:33:55.835Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064257\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowNodeFailed' Failed node exit-test-xpqdl[0].main-step: Pod was active on the node longer than the specified deadline"
time="2024-05-16T17:33:55.835Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064257\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowNodeFailed' Failed node exit-test-xpqdl: child 'exit-test-xpqdl-3770758008' failed"
time="2024-05-16T17:33:55.835Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064257\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowNodeFailed' Failed node exit-test-xpqdl[0]: child 'exit-test-xpqdl-3770758008' failed"
time="2024-05-16T17:33:55.835Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064257\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node exit-test-xpqdl.onExit"
time="2024-05-16T17:33:55.835Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064257\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node exit-test-xpqdl.onExit[0]"
time="2024-05-16T17:33:55.841Z" level=info msg="cleaning up pod" action=labelPodCompleted key=project-il/exit-test-xpqdl-test-3770758008/labelPodCompleted
time="2024-05-16T17:33:58.804Z" level=info msg="Processing workflow" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.805Z" level=info msg="Task-result reconciliation" namespace=project-il numObjs=0 workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.805Z" level=info msg="node changed" namespace=project-il new.message="ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image \"selidocker.org/proj-project/internal-images/testware/pdf-test-ps:1.0.558\": failed to resolve reference \"selidocker.org/proj-project/internal-images/testware/pdf-test-ps:1.0.558\": selidocker.org/proj-project/internal-images/testware/pdf-test-ps:1.0.558: not found" new.phase=Pending new.progress=0/1 nodeID=exit-test-xpqdl-1805001630 old.message= old.phase=Pending old.progress=0/1 workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.805Z" level=info msg="node exit-test-xpqdl-1805001630 phase Pending -> Failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.805Z" level=info msg="node exit-test-xpqdl-1805001630 message: Step exceeded its deadline" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.805Z" level=info msg="node exit-test-xpqdl-1805001630 finished: 2024-05-16 17:33:58.805522001 +0000 UTC" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.805Z" level=debug msg="Evaluating node exit-test-xpqdl: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.805Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:58.805Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:58.805Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2024-05-16T17:33:58.805Z" level=debug msg="Node exit-test-xpqdl already completed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.805Z" level=info msg="TaskSet Reconciliation" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.805Z" level=info msg=reconcileAgentPod namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.805Z" level=info msg="Running OnExit handler: exit-handler" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.805Z" level=debug msg="Evaluating node exit-test-xpqdl.onExit: template: *v1alpha1.WorkflowStep (exit-handler), boundaryID: " namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.805Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:58.805Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:58.806Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (exit-handler)"
time="2024-05-16T17:33:58.806Z" level=debug msg="Executing node exit-test-xpqdl.onExit of Steps is Running" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.806Z" level=debug msg="Evaluating node exit-test-xpqdl.onExit[0].exit-step: template: *v1alpha1.WorkflowStep (test2), boundaryID: exit-test-xpqdl-1312219273" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.806Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:58.806Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:58.806Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.WorkflowStep (test2)"
time="2024-05-16T17:33:58.806Z" level=debug msg="Node exit-test-xpqdl.onExit[0].exit-step already completed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.806Z" level=info msg="Step group node exit-test-xpqdl-1058664605 deemed failed: child 'exit-test-xpqdl-1805001630' failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.806Z" level=info msg="node exit-test-xpqdl-1058664605 phase Running -> Failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.806Z" level=info msg="node exit-test-xpqdl-1058664605 message: child 'exit-test-xpqdl-1805001630' failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.806Z" level=info msg="node exit-test-xpqdl-1058664605 finished: 2024-05-16 17:33:58.806676507 +0000 UTC" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.806Z" level=info msg="step group exit-test-xpqdl-1058664605 was unsuccessful: child 'exit-test-xpqdl-1805001630' failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.806Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (test2)"
time="2024-05-16T17:33:58.806Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (test2)"
time="2024-05-16T17:33:58.806Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=project-il,name=exit-test-xpqdl)" depth=0 tmpl="*v1alpha1.NodeStatus (test2)"
time="2024-05-16T17:33:58.806Z" level=info msg="Outbound nodes of exit-test-xpqdl-1805001630 is [exit-test-xpqdl-1805001630]" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.806Z" level=info msg="Outbound nodes of exit-test-xpqdl-1312219273 is [exit-test-xpqdl-1805001630]" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.807Z" level=info msg="node exit-test-xpqdl-1312219273 phase Running -> Failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.807Z" level=info msg="node exit-test-xpqdl-1312219273 message: child 'exit-test-xpqdl-1805001630' failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.807Z" level=info msg="node exit-test-xpqdl-1312219273 finished: 2024-05-16 17:33:58.807042929 +0000 UTC" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.807Z" level=debug msg="Checking daemoned children of exit-test-xpqdl-1312219273" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.807Z" level=info msg="Updated phase Running -> Failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.807Z" level=info msg="Updated message  -> child 'exit-test-xpqdl-3770758008' failed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.807Z" level=info msg="Marking workflow completed" namespace=project-il workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.807Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064257\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowFailed' child 'exit-test-xpqdl-3770758008' failed"
time="2024-05-16T17:33:58.807Z" level=debug msg="Log changes patch: {\"metadata\":{\"labels\":{\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Failed\"}},\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"},{\"status\":\"True\",\"type\":\"Completed\"}],\"finishedAt\":\"2024-05-16T17:33:58Z\",\"message\":\"child 'exit-test-xpqdl-3770758008' failed\",\"nodes\":{\"exit-test-xpqdl-1058664605\":{\"finishedAt\":\"2024-05-16T17:33:58Z\",\"message\":\"child 'exit-test-xpqdl-1805001630' failed\",\"phase\":\"Failed\"},\"exit-test-xpqdl-1312219273\":{\"finishedAt\":\"2024-05-16T17:33:58Z\",\"message\":\"child 'exit-test-xpqdl-1805001630' failed\",\"outboundNodes\":[\"exit-test-xpqdl-1805001630\"],\"phase\":\"Failed\"},\"exit-test-xpqdl-1805001630\":{\"finishedAt\":\"2024-05-16T17:33:58Z\",\"hostNodeName\":\"node-10-120-81-51\",\"message\":\"Step exceeded its deadline\",\"phase\":\"Failed\"}},\"phase\":\"Failed\"}}"
time="2024-05-16T17:33:58.812Z" level=info msg="cleaning up pod" action=deletePod key=project-il/exit-test-xpqdl-1340600742-agent/deletePod
time="2024-05-16T17:33:58.827Z" level=info msg="Queueing Failed workflow project-il/exit-test-xpqdl for delete in 168h0m0s due to TTL"
time="2024-05-16T17:33:58.837Z" level=info msg="Workflow update successful" namespace=project-il phase=Failed resourceVersion=640064300 workflow=exit-test-xpqdl
time="2024-05-16T17:33:58.837Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064300\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node exit-test-xpqdl.onExit[0].exit-step: Step exceeded its deadline"
time="2024-05-16T17:33:58.837Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064300\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowNodeFailed' Failed node exit-test-xpqdl.onExit[0].exit-step: Step exceeded its deadline"
time="2024-05-16T17:33:58.837Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064300\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowNodeFailed' Failed node exit-test-xpqdl.onExit[0]: child 'exit-test-xpqdl-1805001630' failed"
time="2024-05-16T17:33:58.837Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"project-il\", Name:\"exit-test-xpqdl\", UID:\"6b1c5e05-bc2e-4e41-a002-33bc6ece8db8\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"640064300\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowNodeFailed' Failed node exit-test-xpqdl.onExit: child 'exit-test-xpqdl-1805001630' failed"
time="2024-05-16T17:33:58.851Z" level=info msg="cleaning up pod" action=labelPodCompleted key=project-il/exit-test-xpqdl-test2-1805001630/labelPodCompleted
time="2024-05-16T17:34:01.884Z" level=debug msg="Won't process Workflow since it's completed" key=project-il/exit-test-xpqdl

Logs from in your workflow's wait container

time="2024-05-16T17:33:53.392Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: artifacts/2024/05/16/exit-test-xpqdl/exit-test-xpqdl-test-3770758008/main.log"
time="2024-05-16T17:33:53.392Z" level=info msg="Creating minio client using static credentials" endpoint="seliis3obj.org:10444"
time="2024-05-16T17:33:53.392Z" level=info msg="Saving file to s3" bucket=projectdev-cluster01 endpoint="seliis3obj.org:10444" key=artifacts/2024/05/16/exit-test-xpqdl/exit-test-xpqdl-test-3770758008/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-05-16T17:33:53.470Z" level=info msg="Save artifact" artifactName=main-logs duration=77.666966ms error="<nil>" key=artifacts/2024/05/16/exit-test-xpqdl/exit-test-xpqdl-test-3770758008/main.log
time="2024-05-16T17:33:53.470Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-05-16T17:33:53.470Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-05-16T17:33:53.484Z" level=debug msg="Create workflowtaskresults 403"
time="2024-05-16T17:33:53.484Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:project-il:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"project-il\""
time="2024-05-16T17:33:53.515Z" level=debug msg="Patch pods 200"
time="2024-05-16T17:33:53.518Z" level=info msg="Alloc=8144 TotalAlloc=15019 Sys=28029 NumGC=4 Goroutines=8"
time="2024-05-16T17:33:57.561Z" level=info msg="Starting Workflow Executor" version=v3.4.10
time="2024-05-16T17:33:57.566Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2024-05-16T17:33:57.566Z" level=info msg="Executor initialized" deadline="2024-05-16 17:33:53 +0000 UTC" includeScriptOutput=false namespace=project-il podName=exit-test-xpqdl-test2-1805001630 template="{\"name\":\"test2\",\"inputs\":{},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"selidocker.org/proj-project/internal-images/testware/pdf-test-ps:1.0.558\",\"command\":[\"sh\",\"-c\"],\"args\":[\"echo \\\"exit handler executed!\\\"\"],\"resources\":{}},\"archiveLocation\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"seliis3obj.org:10444\",\"bucket\":\"projectdev-cluster01\",\"insecure\":false,\"accessKeySecret\":{\"name\":\"user-s3-auth\",\"key\":\"s3_access_key\"},\"secretKeySecret\":{\"name\":\"user-s3-auth\",\"key\":\"s3_secret_key\"},\"key\":\"artifacts/2024/05/16/exit-test-xpqdl/exit-test-xpqdl-test2-1805001630\"}}}" version="&Version{Version:v3.4.10,BuildDate:2023-08-15T17:25:13Z,GitCommit:bd6cd2555d1bb0e57a34ce74b0add36cb7fb6c76,GitTag:v3.4.10,GitTreeState:clean,GoVersion:go1.20.7,Compiler:gc,Platform:linux/amd64,}"
time="2024-05-16T17:33:57.567Z" level=info msg="Starting deadline monitor"
time="2024-05-16T17:33:57.567Z" level=info msg="Step exceeded its deadline"
time="2024-05-16T17:33:57.567Z" level=info msg="Killing containers"
time="2024-05-16T17:33:57.567Z" level=warning msg="Failed to kill [\"main\"]: open /var/run/argo/ctr/main/signal: no such file or directory"
time="2024-05-16T17:38:57.567Z" level=info msg="Alloc=6666 TotalAlloc=13539 Sys=27773 NumGC=6 Goroutines=6"
time="2024-05-16T17:43:57.568Z" level=info msg="Alloc=6686 TotalAlloc=13643 Sys=27773 NumGC=8 Goroutines=6"
@Joibel Joibel self-assigned this May 16, 2024
@Joibel Joibel added area/controller Controller issues, panics area/exit-handler labels May 16, 2024
@Joibel
Copy link
Member

Joibel commented May 16, 2024

I am pretty sure the bug is that this check should be aware of exit pods

argo-workflows/workflow/controller/operator.go

Line 1276 in adef075

if deadlineExceeded && (node.Phase == wfv1.NodePending || node.IsActiveSuspendNode()) {
using something like

argo-workflows/workflow/controller/exec_control.go

Line 42 in 71f1d86

_, onExitPod := pod.Labels[common.LabelKeyOnExit]

@jswxstw
Copy link
Member

jswxstw commented May 17, 2024

It also needs to consider the case when onExit is using Plugin/HTTP Template and it has some relevance to #13016
However, onExitNode was not taken into account in #13016, I can fix this issue in this PR to reduce code conflicts.

@Joibel
Copy link
Member

Joibel commented May 17, 2024

@jswxstw, do you want to own this issue then. I think you're volunteering to fix it?

@jswxstw
Copy link
Member

jswxstw commented May 17, 2024

@jswxstw, do you want to own this issue then. I think you're volunteering to fix it?

Yes, I’ll fix it.

@Joibel Joibel assigned jswxstw and unassigned Joibel May 17, 2024
jswxstw pushed a commit to jswxstw/argo-workflows that referenced this issue May 21, 2024
fix: Skip execution control for exit nodes. Fixes argoproj#13060
99f79a7 
Signed-off-by: jswxstw <jswxstw@gmail.com>
jswxstw pushed a commit to jswxstw/argo-workflows that referenced this issue May 21, 2024
fix: Skip execution control for exit nodes. Fixes argoproj#13060
5d5c40c 
Signed-off-by: jswxstw <jswxstw@gmail.com>
jswxstw pushed a commit to jswxstw/argo-workflows that referenced this issue May 21, 2024
fix: Skip execution control for exit nodes. Fixes argoproj#13060
21de6e7 
Signed-off-by: jswxstw <jswxstw@gmail.com>
@jswxstw jswxstw linked a pull request May 21, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jswxstw jswxstw

Labels
area/controller Controller issues, panics area/exit-handler type/bug
Projects
None yet
Milestone
No milestone
3 participants
@Joibel @jswxstw @encigem