fangpenlin ( Fang-Pen Lin )

I think I encountered similar issue on GKE. The argo version I am using is 2.10.0. Last night I submitted a workflow which has 1024 pods in parallel for processing data and uploading to Google Cloud Storage (via gsutil command rather than artifacts output). This morning I noticed that the whole workflow is done, but yet there's one file missing. I looked into the log and realized the controller somehow think the node is done in very short time. I tried to find the log of the pod but can't find anything. I think the pod doesn't even get a chance to be up fully running, but soon it's dead. Yet somehow argo controller thinks the pod is succeeded. Here's the log for the controller:

[
  {
    "textPayload": "time=\"2020-08-30T09:59:33Z\" level=info msg=\"Updating node data-2bhpn-4179820782 status Pending -> Succeeded\"\n",
    "insertId": "u36f3kmuxix7veq4p",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "dotdb3",
        "project_id": "dotdb-231204",
        "container_name": "controller",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "location": "us-central1",
        "namespace_name": "argo-0"
      }
    },
    "timestamp": "2020-08-30T09:59:33.794653184Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:59:39.464856256Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:59:25Z\" level=info msg=\"Skipped pod data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782) creation: already exists\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7ven1i",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "cluster_name": "dotdb3",
        "location": "us-central1",
        "namespace_name": "argo-0",
        "project_id": "dotdb-231204"
      }
    },
    "timestamp": "2020-08-30T09:59:25.474679262Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:59:29.444708649Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:59:14Z\" level=info msg=\"Skipped pod data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782) creation: already exists\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vei8g",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "argo-0",
        "container_name": "controller",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "project_id": "dotdb-231204",
        "location": "us-central1",
        "cluster_name": "dotdb3"
      }
    },
    "timestamp": "2020-08-30T09:59:14.325085578Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:59:19.443556733Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:59:03Z\" level=info msg=\"Skipped pod data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782) creation: already exists\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vee1l",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "container_name": "controller",
        "cluster_name": "dotdb3",
        "namespace_name": "argo-0",
        "location": "us-central1",
        "project_id": "dotdb-231204"
      }
    },
    "timestamp": "2020-08-30T09:59:03.944302801Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:59:09.462969696Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:59:00Z\" level=info msg=\"Updating node data-2bhpn-4179820782 message: Unschedulable: 0/20 nodes are available: 13 Insufficient cpu, 2 node(s) had taints that the pod didn't tolerate, 5 node(s) didn't match node selector.\"\n",
    "insertId": "u36f3kmuxix7ved3n",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "namespace_name": "argo-0",
        "location": "us-central1",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "cluster_name": "dotdb3",
        "project_id": "dotdb-231204"
      }
    },
    "timestamp": "2020-08-30T09:59:00.312067761Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:59:04.583608911Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:54Z\" level=info msg=\"Skipped pod data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782) creation: already exists\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vdz2p",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1",
        "container_name": "controller",
        "cluster_name": "dotdb3",
        "project_id": "dotdb-231204",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "namespace_name": "argo-0"
      }
    },
    "timestamp": "2020-08-30T09:58:54.287322757Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:59.500998938Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:52Z\" level=info msg=\"insignificant pod change\" key=argo-0/data-2bhpn-4179820782\n",
    "insertId": "u36f3kmuxix7vdz0k",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1",
        "container_name": "controller",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "project_id": "dotdb-231204",
        "namespace_name": "argo-0",
        "cluster_name": "dotdb3"
      }
    },
    "timestamp": "2020-08-30T09:58:52.757956615Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:54.466535608Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:44Z\" level=info msg=\"Skipped pod data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782) creation: already exists\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vdu6l",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "cluster_name": "dotdb3",
        "container_name": "controller",
        "namespace_name": "argo-0",
        "project_id": "dotdb-231204",
        "location": "us-central1"
      }
    },
    "timestamp": "2020-08-30T09:58:44.137674055Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:49.554273725Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:32Z\" level=info msg=\"Skipped pod data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782) creation: already exists\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vdp5o",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "location": "us-central1",
        "namespace_name": "argo-0",
        "cluster_name": "dotdb3",
        "project_id": "dotdb-231204"
      }
    },
    "timestamp": "2020-08-30T09:58:32.338253656Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:34.444975036Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:26Z\" level=info msg=\"Updating node data-2bhpn-4179820782 message: Unschedulable: 0/18 nodes are available: 13 Insufficient cpu, 5 node(s) didn't match node selector.\"\n",
    "insertId": "u36f3kmuxix7vdo7p",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1",
        "cluster_name": "dotdb3",
        "container_name": "controller",
        "namespace_name": "argo-0",
        "project_id": "dotdb-231204",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7"
      }
    },
    "timestamp": "2020-08-30T09:58:26.739056781Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:29.543231340Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:22Z\" level=info msg=\"insignificant pod change\" key=argo-0/data-2bhpn-4179820782\n",
    "insertId": "u36f3kmuxix7vdk2s",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "location": "us-central1",
        "project_id": "dotdb-231204",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "cluster_name": "dotdb3",
        "namespace_name": "argo-0"
      }
    },
    "timestamp": "2020-08-30T09:58:22.894848019Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:24.469135596Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:22Z\" level=info msg=\"Created pod: data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782)\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vdk2o",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "cluster_name": "dotdb3",
        "location": "us-central1",
        "project_id": "dotdb-231204",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "namespace_name": "argo-0"
      }
    },
    "timestamp": "2020-08-30T09:58:22.888713326Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:24.469135596Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:22Z\" level=info msg=\"Pod node data-2bhpn-4179820782 initialized Pending\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vdk2m",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "project_id": "dotdb-231204",
        "namespace_name": "argo-0",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "location": "us-central1",
        "cluster_name": "dotdb3"
      }
    },
    "timestamp": "2020-08-30T09:58:22.839015156Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:24.469135596Z"
  }
]

Normally each of these node would take 20 minutes to finish, but this node from start to finish only took 1 minute, certainly not right. I cannot reproduce it, and not much other log I can find. These are what I can provide.

juchaosong

comment created time in 2 months

startedscrapedia/scrapy-useragents

started time in 2 months

startedtwisted/twisted

started time in 2 months

startedJays2Kings/DS4Windows

started time in 2 months

startedopenairplay/open-airplay

started time in 2 months

push eventfangpenlin/fangpenlin

Fang-Pen Lin

commit sha 9db82296b04924286a0262e862d484ee0a185615

Update README.md

view details

push time in 2 months

push eventfangpenlin/fangpenlin

Fang-Pen Lin

commit sha 36004e6d5bcf37cbe9b3b553e35d19a4a8439369

Update README.md

view details

push time in 2 months

push eventfangpenlin/fangpenlin

Fang-Pen Lin

commit sha 4ed95c1f3fed7208ae0531a9880861314b7d951d

Update README.md

view details

push time in 2 months

push eventfangpenlin/fangpenlin

Fang-Pen Lin

commit sha ecaacac70bb8d4a3e2dea233937b7cccc3a5dc2b

Update README.md

view details

push time in 2 months

create barnchfangpenlin/fangpenlin

branch : master

created branch time in 2 months

created repositoryfangpenlin/fangpenlin

created time in 2 months

issue openedargoproj/argo

DAG with generated jobs from loop failure won't make the workflow fail

Summary

When you create a workflow with DAG which has generated jobs from loop, the failed job created by the loop won't make the whole workflow fail, instead, it will still get passed as Succeeded. And the final job depending on the generated tasks will be ignored.

Name:                reproduce-bug-9tpfr
Namespace:           default
ServiceAccount:      argo-workflow
Status:              Succeeded
Conditions:
 Completed           True
Created:             Thu Aug 13 20:47:23 -0700 (6 minutes ago)
Started:             Thu Aug 13 20:47:23 -0700 (6 minutes ago)
Finished:            Thu Aug 13 20:49:42 -0700 (4 minutes ago)
Duration:            2 minutes 19 seconds
ResourcesDuration:   36s*(1 cpu),36s*(100Mi memory)

STEP                          TEMPLATE       PODNAME                         DURATION  MESSAGE
 ✔ reproduce-bug-9tpfr        start
 ├-✔ gen-tasks(0)             gen-tasks      reproduce-bug-9tpfr-3595395365  3s
 ├-✔ process-tasks(0:0)(0)    process-tasks  reproduce-bug-9tpfr-3662458541  13s
 ├-✔ process-tasks(1:1)(0)    process-tasks  reproduce-bug-9tpfr-988357889   5s
 ├-✔ process-tasks(2:2)(0)    process-tasks  reproduce-bug-9tpfr-3450100829  4s
 ├-✖ process-tasks(3:3)       process-tasks                                            No more retries left
 | ├-✖ process-tasks(3:3)(0)  process-tasks  reproduce-bug-9tpfr-2008331609  13s       failed with exit code 1
 | ├-✖ process-tasks(3:3)(1)  process-tasks  reproduce-bug-9tpfr-2612472988  5s        failed with exit code 1
 | └-✖ process-tasks(3:3)(2)  process-tasks  reproduce-bug-9tpfr-4222579959  3s        failed with exit code 1
 ├-✔ process-tasks(4:4)(0)    process-tasks  reproduce-bug-9tpfr-3522592509  13s
 ├-✔ process-tasks(5:5)(0)    process-tasks  reproduce-bug-9tpfr-600389385   4s
 ├-✔ process-tasks(6:6)(0)    process-tasks  reproduce-bug-9tpfr-3207714925  4s
 ├-✔ process-tasks(7:7)(0)    process-tasks  reproduce-bug-9tpfr-585646929   3s
 ├-✔ process-tasks(8:8)(0)    process-tasks  reproduce-bug-9tpfr-1384515437  12s
 ├-✔ process-tasks(9:9)(0)    process-tasks  reproduce-bug-9tpfr-2475724017  4s

It should fail the whole workflow instead when any of the generated tasks from loop fails

Diagnostics

What version of Argo Workflows are you running?

v2.10.0-rc6

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  creationTimestamp: "2020-08-14T03:47:23Z"
  generateName: reproduce-bug-
  generation: 21
  labels:
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/phase: Succeeded
  name: reproduce-bug-9tpfr
  namespace: default
  resourceVersion: "9981935"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/default/workflows/reproduce-bug-9tpfr
  uid: dcdbbf44-dde0-11ea-b7b7-42010a800164
spec:
  arguments: {}
  entrypoint: start
  serviceAccountName: argo-workflow
  templates:
  - arguments: {}
    dag:
      tasks:
      - arguments: {}
        name: gen-tasks
        template: gen-tasks
      - arguments:
          parameters:
          - name: chunk
            value: '{{item}}'
        dependencies:
        - gen-tasks
        name: process-tasks
        template: process-tasks
        withParam: '{{tasks.gen-tasks.outputs.result}}'
      - arguments: {}
        dependencies:
        - process-tasks
        name: finish
        template: finish
    inputs: {}
    metadata: {}
    name: start
    outputs: {}
  - activeDeadlineSeconds: 300
    arguments: {}
    inputs: {}
    metadata: {}
    name: gen-tasks
    outputs: {}
    retryStrategy:
      backoff:
        duration: 15s
        factor: 2
      limit: 5
      retryPolicy: Always
    script:
      command:
      - bash
      image: python:3
      name: ""
      resources:
        requests:
          cpu: 250m
      source: |
        set -e
        python3 -c 'import os, json; print(json.dumps([str(i) for i in range(10)]))'
  - activeDeadlineSeconds: 1800
    arguments: {}
    inputs:
      parameters:
      - name: chunk
    metadata: {}
    name: process-tasks
    outputs: {}
    retryStrategy:
      backoff:
        duration: 15s
        factor: 2
      limit: 2
      retryPolicy: Always
    script:
      command:
      - bash
      image: python:3
      name: ""
      resources:
        requests:
          cpu: 100m
      source: |
        set -e
        chunk="{{inputs.parameters.chunk}}"
        if [[ $chunk == "3" ]]; then
          echo "failed"
          exit 1
        fi
        echo "process $chunk"
  - activeDeadlineSeconds: 300
    arguments: {}
    inputs: {}
    metadata: {}
    name: finish
    outputs: {}
    script:
      command:
      - sh
      image: busybox
      name: ""
      resources:
        requests:
          cpu: 100m
      source: |
        echo fin
status:
  conditions:
  - status: "True"
    type: Completed
  finishedAt: "2020-08-14T03:49:42Z"
  nodes:
    reproduce-bug-9tpfr:
      children:
      - reproduce-bug-9tpfr-1525049382
      displayName: reproduce-bug-9tpfr
      finishedAt: "2020-08-14T03:49:42Z"
      id: reproduce-bug-9tpfr
      name: reproduce-bug-9tpfr
      outboundNodes:
      - reproduce-bug-9tpfr-247809182
      phase: Succeeded
      startedAt: "2020-08-14T03:47:23Z"
      templateName: start
      templateScope: local/reproduce-bug-9tpfr
      type: DAG
    reproduce-bug-9tpfr-247809182:
      boundaryID: reproduce-bug-9tpfr
      displayName: finish
      finishedAt: "2020-08-14T03:49:42Z"
      id: reproduce-bug-9tpfr-247809182
      message: 'omitted: depends condition not met'
      name: reproduce-bug-9tpfr.finish
      phase: Omitted
      startedAt: "2020-08-14T03:49:42Z"
      templateName: finish
      templateScope: local/reproduce-bug-9tpfr
      type: Skipped
    reproduce-bug-9tpfr-546685502:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-3207714925
      displayName: process-tasks(6:6)
      finishedAt: "2020-08-14T03:47:37Z"
      id: reproduce-bug-9tpfr-546685502
      inputs:
        parameters:
        - name: chunk
          value: "6"
      name: reproduce-bug-9tpfr.process-tasks(6:6)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-585646929:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(7:7)(0)
      finishedAt: "2020-08-14T03:47:33Z"
      hostNodeName: gke-dotdb3-dotdb-pool-c3910e16-nb0l
      id: reproduce-bug-9tpfr-585646929
      inputs:
        parameters:
        - name: chunk
          value: "7"
      name: reproduce-bug-9tpfr.process-tasks(7:7)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-600389385:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(5:5)(0)
      finishedAt: "2020-08-14T03:47:33Z"
      hostNodeName: gke-dotdb3-dotdb-pool-c3910e16-nb0l
      id: reproduce-bug-9tpfr-600389385
      inputs:
        parameters:
        - name: chunk
          value: "5"
      name: reproduce-bug-9tpfr.process-tasks(5:5)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-850457006:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-3450100829
      displayName: process-tasks(2:2)
      finishedAt: "2020-08-14T03:47:36Z"
      id: reproduce-bug-9tpfr-850457006
      inputs:
        parameters:
        - name: chunk
          value: "2"
      name: reproduce-bug-9tpfr.process-tasks(2:2)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-988357889:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(1:1)(0)
      finishedAt: "2020-08-14T03:47:34Z"
      hostNodeName: gke-dotdb3-dotdb-pool-3779d805-hkcv
      id: reproduce-bug-9tpfr-988357889
      inputs:
        parameters:
        - name: chunk
          value: "1"
      name: reproduce-bug-9tpfr.process-tasks(1:1)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 2
        memory: 2
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-1384515437:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(8:8)(0)
      finishedAt: "2020-08-14T03:47:42Z"
      hostNodeName: gke-dotdb3-dotdb-pool-2e8afe0d-km90
      id: reproduce-bug-9tpfr-1384515437
      inputs:
        parameters:
        - name: chunk
          value: "8"
      name: reproduce-bug-9tpfr.process-tasks(8:8)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 6
        memory: 6
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-1525049382:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-3595395365
      displayName: gen-tasks
      finishedAt: "2020-08-14T03:47:29Z"
      id: reproduce-bug-9tpfr-1525049382
      name: reproduce-bug-9tpfr.gen-tasks
      outputs:
        exitCode: "0"
        result: '["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]'
      phase: Succeeded
      startedAt: "2020-08-14T03:47:23Z"
      templateName: gen-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-1602779214:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-3522592509
      displayName: process-tasks(4:4)
      finishedAt: "2020-08-14T03:47:45Z"
      id: reproduce-bug-9tpfr-1602779214
      inputs:
        parameters:
        - name: chunk
          value: "4"
      name: reproduce-bug-9tpfr.process-tasks(4:4)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-1670762753:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-2788826366
      - reproduce-bug-9tpfr-3412607194
      - reproduce-bug-9tpfr-850457006
      - reproduce-bug-9tpfr-3505932898
      - reproduce-bug-9tpfr-1602779214
      - reproduce-bug-9tpfr-4143798034
      - reproduce-bug-9tpfr-546685502
      - reproduce-bug-9tpfr-3415721642
      - reproduce-bug-9tpfr-3221614398
      - reproduce-bug-9tpfr-3518128714
      displayName: process-tasks
      finishedAt: "2020-08-14T03:49:42Z"
      id: reproduce-bug-9tpfr-1670762753
      name: reproduce-bug-9tpfr.process-tasks
      phase: Failed
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: TaskGroup
    reproduce-bug-9tpfr-2008331609:
      boundaryID: reproduce-bug-9tpfr
      displayName: process-tasks(3:3)(0)
      finishedAt: "2020-08-14T03:47:42Z"
      hostNodeName: gke-dotdb3-dotdb-pool-2e8afe0d-km90
      id: reproduce-bug-9tpfr-2008331609
      inputs:
        parameters:
        - name: chunk
          value: "3"
      message: failed with exit code 1
      name: reproduce-bug-9tpfr.process-tasks(3:3)(0)
      outputs:
        exitCode: "1"
      phase: Failed
      resourcesDuration:
        cpu: 6
        memory: 6
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-2475724017:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(9:9)(0)
      finishedAt: "2020-08-14T03:47:34Z"
      hostNodeName: gke-dotdb3-dotdb-pool-3779d805-hkcv
      id: reproduce-bug-9tpfr-2475724017
      inputs:
        parameters:
        - name: chunk
          value: "9"
      name: reproduce-bug-9tpfr.process-tasks(9:9)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-2612472988:
      boundaryID: reproduce-bug-9tpfr
      displayName: process-tasks(3:3)(1)
      finishedAt: "2020-08-14T03:48:05Z"
      hostNodeName: gke-dotdb3-dotdb-pool-2e8afe0d-km90
      id: reproduce-bug-9tpfr-2612472988
      inputs:
        parameters:
        - name: chunk
          value: "3"
      message: failed with exit code 1
      name: reproduce-bug-9tpfr.process-tasks(3:3)(1)
      outputs:
        exitCode: "1"
      phase: Failed
      resourcesDuration:
        cpu: 2
        memory: 2
      startedAt: "2020-08-14T03:48:00Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-2788826366:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-3662458541
      displayName: process-tasks(0:0)
      finishedAt: "2020-08-14T03:47:44Z"
      id: reproduce-bug-9tpfr-2788826366
      inputs:
        parameters:
        - name: chunk
          value: "0"
      name: reproduce-bug-9tpfr.process-tasks(0:0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-3207714925:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(6:6)(0)
      finishedAt: "2020-08-14T03:47:34Z"
      hostNodeName: gke-dotdb3-dotdb-pool-3779d805-hkcv
      id: reproduce-bug-9tpfr-3207714925
      inputs:
        parameters:
        - name: chunk
          value: "6"
      name: reproduce-bug-9tpfr.process-tasks(6:6)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 2
        memory: 2
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-3221614398:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-1384515437
      displayName: process-tasks(8:8)
      finishedAt: "2020-08-14T03:47:44Z"
      id: reproduce-bug-9tpfr-3221614398
      inputs:
        parameters:
        - name: chunk
          value: "8"
      name: reproduce-bug-9tpfr.process-tasks(8:8)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-3412607194:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-988357889
      displayName: process-tasks(1:1)
      finishedAt: "2020-08-14T03:47:36Z"
      id: reproduce-bug-9tpfr-3412607194
      inputs:
        parameters:
        - name: chunk
          value: "1"
      name: reproduce-bug-9tpfr.process-tasks(1:1)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-3415721642:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-585646929
      displayName: process-tasks(7:7)
      finishedAt: "2020-08-14T03:47:36Z"
      id: reproduce-bug-9tpfr-3415721642
      inputs:
        parameters:
        - name: chunk
          value: "7"
      name: reproduce-bug-9tpfr.process-tasks(7:7)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-3450100829:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(2:2)(0)
      finishedAt: "2020-08-14T03:47:33Z"
      hostNodeName: gke-dotdb3-dotdb-pool-3779d805-hkcv
      id: reproduce-bug-9tpfr-3450100829
      inputs:
        parameters:
        - name: chunk
          value: "2"
      name: reproduce-bug-9tpfr.process-tasks(2:2)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-3505932898:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-2008331609
      - reproduce-bug-9tpfr-2612472988
      - reproduce-bug-9tpfr-4222579959
      displayName: process-tasks(3:3)
      finishedAt: "2020-08-14T03:49:42Z"
      id: reproduce-bug-9tpfr-3505932898
      inputs:
        parameters:
        - name: chunk
          value: "3"
      message: No more retries left
      name: reproduce-bug-9tpfr.process-tasks(3:3)
      phase: Failed
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-3518128714:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-2475724017
      displayName: process-tasks(9:9)
      finishedAt: "2020-08-14T03:47:37Z"
      id: reproduce-bug-9tpfr-3518128714
      inputs:
        parameters:
        - name: chunk
          value: "9"
      name: reproduce-bug-9tpfr.process-tasks(9:9)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-3522592509:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(4:4)(0)
      finishedAt: "2020-08-14T03:47:42Z"
      hostNodeName: gke-dotdb3-dotdb-pool-2e8afe0d-km90
      id: reproduce-bug-9tpfr-3522592509
      inputs:
        parameters:
        - name: chunk
          value: "4"
      name: reproduce-bug-9tpfr.process-tasks(4:4)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 6
        memory: 6
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-3595395365:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-1670762753
      displayName: gen-tasks(0)
      finishedAt: "2020-08-14T03:47:26Z"
      hostNodeName: gke-dotdb3-dotdb-pool-3779d805-hkcv
      id: reproduce-bug-9tpfr-3595395365
      name: reproduce-bug-9tpfr.gen-tasks(0)
      outputs:
        exitCode: "0"
        result: '["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]'
      phase: Succeeded
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2020-08-14T03:47:23Z"
      templateName: gen-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-3662458541:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(0:0)(0)
      finishedAt: "2020-08-14T03:47:42Z"
      hostNodeName: gke-dotdb3-dotdb-pool-2e8afe0d-km90
      id: reproduce-bug-9tpfr-3662458541
      inputs:
        parameters:
        - name: chunk
          value: "0"
      name: reproduce-bug-9tpfr.process-tasks(0:0)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 6
        memory: 6
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-4143798034:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-600389385
      displayName: process-tasks(5:5)
      finishedAt: "2020-08-14T03:47:36Z"
      id: reproduce-bug-9tpfr-4143798034
      inputs:
        parameters:
        - name: chunk
          value: "5"
      name: reproduce-bug-9tpfr.process-tasks(5:5)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-4222579959:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(3:3)(2)
      finishedAt: "2020-08-14T03:48:40Z"
      hostNodeName: gke-dotdb3-dotdb-pool-3779d805-hkcv
      id: reproduce-bug-9tpfr-4222579959
      inputs:
        parameters:
        - name: chunk
          value: "3"
      message: failed with exit code 1
      name: reproduce-bug-9tpfr.process-tasks(3:3)(2)
      outputs:
        exitCode: "1"
      phase: Failed
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2020-08-14T03:48:37Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
  phase: Succeeded
  resourcesDuration:
    cpu: 36
    memory: 36
  startedAt: "2020-08-14T03:47:23Z"

time="2020-08-14T03:47:23Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:23Z" level=info msg="Updated phase  -> Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:23Z" level=info msg="DAG node reproduce-bug-9tpfr initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:23Z" level=info msg="All of node reproduce-bug-9tpfr.gen-tasks dependencies [] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:23Z" level=info msg="Retry node reproduce-bug-9tpfr-1525049382 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:23Z" level=info msg="Pod node reproduce-bug-9tpfr-3595395365 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:23Z" level=info msg="Created pod: reproduce-bug-9tpfr.gen-tasks(0) (reproduce-bug-9tpfr-3595395365)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:24Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980007 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:25Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:25Z" level=info msg="Updating node reproduce-bug-9tpfr-3595395365 message: PodInitializing"
time="2020-08-14T03:47:26Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-3595395365
time="2020-08-14T03:47:26Z" level=info msg="Skipped pod reproduce-bug-9tpfr.gen-tasks(0) (reproduce-bug-9tpfr-3595395365) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:27Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980037 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Setting node reproduce-bug-9tpfr-3595395365 outputs"
time="2020-08-14T03:47:29Z" level=info msg="Updating node reproduce-bug-9tpfr-3595395365 status Pending -> Succeeded"
time="2020-08-14T03:47:29Z" level=info msg="node reproduce-bug-9tpfr-1525049382 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="node reproduce-bug-9tpfr-1525049382 finished: 2020-08-14 03:47:29.356536963 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="TaskGroup node reproduce-bug-9tpfr-1670762753 initialized Running (message: )" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(0:0) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Retry node reproduce-bug-9tpfr-2788826366 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Pod node reproduce-bug-9tpfr-3662458541 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(1:1) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Retry node reproduce-bug-9tpfr-3412607194 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Pod node reproduce-bug-9tpfr-988357889 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(1:1)(0) (reproduce-bug-9tpfr-988357889)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(2:2) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Retry node reproduce-bug-9tpfr-850457006 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Pod node reproduce-bug-9tpfr-3450100829 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(2:2)(0) (reproduce-bug-9tpfr-3450100829)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(3:3) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Retry node reproduce-bug-9tpfr-3505932898 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Pod node reproduce-bug-9tpfr-2008331609 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(4:4) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Retry node reproduce-bug-9tpfr-1602779214 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Pod node reproduce-bug-9tpfr-3522592509 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(5:5) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Retry node reproduce-bug-9tpfr-4143798034 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Pod node reproduce-bug-9tpfr-600389385 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(5:5)(0) (reproduce-bug-9tpfr-600389385)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(6:6) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Retry node reproduce-bug-9tpfr-546685502 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Pod node reproduce-bug-9tpfr-3207714925 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(6:6)(0) (reproduce-bug-9tpfr-3207714925)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(7:7) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Retry node reproduce-bug-9tpfr-3415721642 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Pod node reproduce-bug-9tpfr-585646929 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(7:7)(0) (reproduce-bug-9tpfr-585646929)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(8:8) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Retry node reproduce-bug-9tpfr-3221614398 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Pod node reproduce-bug-9tpfr-1384515437 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(9:9) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Retry node reproduce-bug-9tpfr-3518128714 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Pod node reproduce-bug-9tpfr-2475724017 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(9:9)(0) (reproduce-bug-9tpfr-2475724017)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980134 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3595395365 completed"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-988357889 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-1384515437 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-2008331609 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-2475724017 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-3207714925 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-3450100829 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-3522592509 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-3662458541 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-585646929 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-600389385 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(1:1)(0) (reproduce-bug-9tpfr-988357889) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(2:2)(0) (reproduce-bug-9tpfr-3450100829) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(5:5)(0) (reproduce-bug-9tpfr-600389385) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(6:6)(0) (reproduce-bug-9tpfr-3207714925) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:33Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(7:7)(0) (reproduce-bug-9tpfr-585646929) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:33Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:33Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(9:9)(0) (reproduce-bug-9tpfr-2475724017) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:33Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980174 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:33Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-600389385
time="2020-08-14T03:47:33Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-585646929
time="2020-08-14T03:47:33Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-3450100829
time="2020-08-14T03:47:34Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3595395365 completed"
time="2020-08-14T03:47:34Z" level=info msg="Setting node reproduce-bug-9tpfr-3450100829 outputs"
time="2020-08-14T03:47:34Z" level=info msg="Updating node reproduce-bug-9tpfr-3450100829 status Pending -> Running"
time="2020-08-14T03:47:34Z" level=info msg="Setting node reproduce-bug-9tpfr-585646929 outputs"
time="2020-08-14T03:47:34Z" level=info msg="Updating node reproduce-bug-9tpfr-585646929 status Pending -> Running"
time="2020-08-14T03:47:34Z" level=info msg="Setting node reproduce-bug-9tpfr-600389385 outputs"
time="2020-08-14T03:47:34Z" level=info msg="Updating node reproduce-bug-9tpfr-600389385 status Pending -> Running"
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(1:1)(0) (reproduce-bug-9tpfr-988357889) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(6:6)(0) (reproduce-bug-9tpfr-3207714925) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(9:9)(0) (reproduce-bug-9tpfr-2475724017) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980201 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-2475724017
time="2020-08-14T03:47:34Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-988357889
time="2020-08-14T03:47:34Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-3207714925
time="2020-08-14T03:47:35Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:35Z" level=info msg="Setting node reproduce-bug-9tpfr-988357889 outputs"
time="2020-08-14T03:47:35Z" level=info msg="Updating node reproduce-bug-9tpfr-988357889 status Pending -> Succeeded"
time="2020-08-14T03:47:35Z" level=info msg="Updating node reproduce-bug-9tpfr-3450100829 status Running -> Succeeded"
time="2020-08-14T03:47:35Z" level=info msg="Updating node reproduce-bug-9tpfr-585646929 status Running -> Succeeded"
time="2020-08-14T03:47:35Z" level=info msg="Updating node reproduce-bug-9tpfr-600389385 status Running -> Succeeded"
time="2020-08-14T03:47:35Z" level=info msg="Setting node reproduce-bug-9tpfr-2475724017 outputs"
time="2020-08-14T03:47:35Z" level=info msg="Updating node reproduce-bug-9tpfr-2475724017 status Pending -> Running"
time="2020-08-14T03:47:35Z" level=info msg="Setting node reproduce-bug-9tpfr-3207714925 outputs"
time="2020-08-14T03:47:35Z" level=info msg="Updating node reproduce-bug-9tpfr-3207714925 status Pending -> Running"
time="2020-08-14T03:47:35Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:35Z" level=info msg="node reproduce-bug-9tpfr-3412607194 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-3412607194 finished: 2020-08-14 03:47:36.000227026 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-850457006 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-850457006 finished: 2020-08-14 03:47:36.001690292 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-4143798034 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-4143798034 finished: 2020-08-14 03:47:36.061716791 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-3415721642 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-3415721642 finished: 2020-08-14 03:47:36.062801551 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980235 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-988357889 completed"
time="2020-08-14T03:47:37Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3450100829 completed"
time="2020-08-14T03:47:37Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-585646929 completed"
time="2020-08-14T03:47:37Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-600389385 completed"
time="2020-08-14T03:47:37Z" level=info msg="Updating node reproduce-bug-9tpfr-2475724017 status Running -> Succeeded"
time="2020-08-14T03:47:37Z" level=info msg="Updating node reproduce-bug-9tpfr-3207714925 status Running -> Succeeded"
time="2020-08-14T03:47:37Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
E0814 03:47:37.305030       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980235", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(9:9)(0)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a036507c4568, ext:43052386523483, loc:(*time.Location)(0x2953ca0)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
E0814 03:47:37.314170       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980235", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(6:6)(0)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a036529296b3, ext:43052421540523, loc:(*time.Location)(0x2953ca0)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:37Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="node reproduce-bug-9tpfr-546685502 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="node reproduce-bug-9tpfr-546685502 finished: 2020-08-14 03:47:37.338582183 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
E0814 03:47:37.344631       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980235", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(6:6)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a0365454ae88, ext:43052451037818, loc:(*time.Location)(0x2953ca0)}}, Count:4, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:37Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="node reproduce-bug-9tpfr-3518128714 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="node reproduce-bug-9tpfr-3518128714 finished: 2020-08-14 03:47:37.372769276 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
E0814 03:47:37.377068       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980235", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(9:9)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a0365653a04c, ext:43052484523046, loc:(*time.Location)(0x2953ca0)}}, Count:5, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:37Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980257 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:38Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:38Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3450100829 completed"
time="2020-08-14T03:47:38Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-585646929 completed"
time="2020-08-14T03:47:38Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-600389385 completed"
time="2020-08-14T03:47:38Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-988357889 completed"
time="2020-08-14T03:47:38Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-2475724017 completed"
time="2020-08-14T03:47:39Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3207714925 completed"
time="2020-08-14T03:47:39Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:39Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:39Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:39Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:39Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:40Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:40Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:40Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:40Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:41Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:41Z" level=info msg="Updating node reproduce-bug-9tpfr-3662458541 status Pending -> Running"
time="2020-08-14T03:47:41Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:41Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:41Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:41Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980321 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:42Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-3522592509
time="2020-08-14T03:47:42Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-3662458541
time="2020-08-14T03:47:42Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-1384515437
time="2020-08-14T03:47:42Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:43Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-2008331609
time="2020-08-14T03:47:43Z" level=info msg="Setting node reproduce-bug-9tpfr-1384515437 outputs"
time="2020-08-14T03:47:43Z" level=info msg="Updating node reproduce-bug-9tpfr-1384515437 status Pending -> Running"
time="2020-08-14T03:47:43Z" level=info msg="Setting node reproduce-bug-9tpfr-2008331609 outputs"
time="2020-08-14T03:47:43Z" level=info msg="Updating node reproduce-bug-9tpfr-2008331609 status Pending -> Running"
time="2020-08-14T03:47:43Z" level=info msg="Setting node reproduce-bug-9tpfr-3522592509 outputs"
time="2020-08-14T03:47:43Z" level=info msg="Updating node reproduce-bug-9tpfr-3522592509 status Pending -> Running"
time="2020-08-14T03:47:43Z" level=info msg="Setting node reproduce-bug-9tpfr-3662458541 outputs"
time="2020-08-14T03:47:43Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980336 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:44Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:44Z" level=info msg="Updating node reproduce-bug-9tpfr-1384515437 status Running -> Succeeded"
time="2020-08-14T03:47:44Z" level=info msg="Updating node reproduce-bug-9tpfr-2008331609 status Running -> Failed"
time="2020-08-14T03:47:44Z" level=info msg="Updating node reproduce-bug-9tpfr-2008331609 message: failed with exit code 1"
time="2020-08-14T03:47:44Z" level=info msg="Updating node reproduce-bug-9tpfr-3662458541 status Running -> Succeeded"
time="2020-08-14T03:47:44Z" level=info msg="node reproduce-bug-9tpfr-2788826366 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:44Z" level=info msg="node reproduce-bug-9tpfr-2788826366 finished: 2020-08-14 03:47:44.530749349 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:44Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: Backoff for 15 seconds" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:44Z" level=info msg="node reproduce-bug-9tpfr-3221614398 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:44Z" level=info msg="node reproduce-bug-9tpfr-3221614398 finished: 2020-08-14 03:47:44.53177149 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
E0814 03:47:44.576680       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980336", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(8:8)(0)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a0381fca46cd, ext:43059643293861, loc:(*time.Location)(0x2953ca0)}}, Count:6, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:44Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980353 workflow=reproduce-bug-9tpfr
E0814 03:47:44.626797       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980336", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(0:0)(0)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a038236458c7, ext:43059703722712, loc:(*time.Location)(0x2953ca0)}}, Count:7, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
E0814 03:47:44.653302       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980336", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(0:0)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03825ed8795, ext:43059746267572, loc:(*time.Location)(0x2953ca0)}}, Count:8, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
E0814 03:47:44.690093       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980336", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(8:8)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03826f2f633, ext:43059763400748, loc:(*time.Location)(0x2953ca0)}}, Count:9, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:45Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:45Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-2008331609 completed"
time="2020-08-14T03:47:45Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3662458541 completed"
time="2020-08-14T03:47:45Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-1384515437 completed"
time="2020-08-14T03:47:45Z" level=info msg="Updating node reproduce-bug-9tpfr-3522592509 status Running -> Succeeded"
E0814 03:47:45.711454       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980353", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(4:4)(0)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a0386a3cd788, ext:43060818574184, loc:(*time.Location)(0x2953ca0)}}, Count:10, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:45Z" level=info msg="node reproduce-bug-9tpfr-1602779214 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:45Z" level=info msg="node reproduce-bug-9tpfr-1602779214 finished: 2020-08-14 03:47:45.717442949 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
E0814 03:47:45.720901       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980353", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(4:4)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a0386ac7eeab, ext:43060827689616, loc:(*time.Location)(0x2953ca0)}}, Count:11, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:45Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980365 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:46Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:46Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-1384515437 completed"
time="2020-08-14T03:47:46Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-2008331609 completed"
time="2020-08-14T03:47:46Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3662458541 completed"
time="2020-08-14T03:47:46Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3522592509 completed"
time="2020-08-14T03:48:00Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:00Z" level=info msg="node has maxDuration set, setting executionDeadline to: Mon Jan 01 00:00:00 +0000 (a long while ago)" namespace=default node="reproduce-bug-9tpfr.process-tasks(3:3)" workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:00Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: " namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:00Z" level=info msg="1 child nodes of reproduce-bug-9tpfr.process-tasks(3:3) failed. Trying again..." namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:00Z" level=info msg="Pod node reproduce-bug-9tpfr-2612472988 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:00Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(3:3)(1) (reproduce-bug-9tpfr-2612472988)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:00Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980515 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:01Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:02Z" level=info msg="Updating node reproduce-bug-9tpfr-2612472988 message: PodInitializing"
time="2020-08-14T03:48:02Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(1) (reproduce-bug-9tpfr-2612472988) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:02Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980527 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:03Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:03Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(1) (reproduce-bug-9tpfr-2612472988) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:04Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:05Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(1) (reproduce-bug-9tpfr-2612472988) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:05Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-2612472988
time="2020-08-14T03:48:05Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:06Z" level=info msg="Setting node reproduce-bug-9tpfr-2612472988 outputs"
time="2020-08-14T03:48:06Z" level=info msg="Updating node reproduce-bug-9tpfr-2612472988 status Pending -> Running"
time="2020-08-14T03:48:06Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980562 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:07Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:07Z" level=info msg="Updating node reproduce-bug-9tpfr-2612472988 status Running -> Failed"
time="2020-08-14T03:48:07Z" level=info msg="Updating node reproduce-bug-9tpfr-2612472988 message: failed with exit code 1"
time="2020-08-14T03:48:07Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: Backoff for 30 seconds" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:07Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980575 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:08Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:08Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-2612472988 completed"
time="2020-08-14T03:48:37Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:37Z" level=info msg="node has maxDuration set, setting executionDeadline to: Mon Jan 01 00:00:00 +0000 (a long while ago)" namespace=default node="reproduce-bug-9tpfr.process-tasks(3:3)" workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:37Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: " namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:37Z" level=info msg="2 child nodes of reproduce-bug-9tpfr.process-tasks(3:3) failed. Trying again..." namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:37Z" level=info msg="Pod node reproduce-bug-9tpfr-4222579959 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:37Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(3:3)(2) (reproduce-bug-9tpfr-4222579959)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:37Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980821 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:38Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:38Z" level=info msg="Updating node reproduce-bug-9tpfr-4222579959 message: PodInitializing"
time="2020-08-14T03:48:38Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(2) (reproduce-bug-9tpfr-4222579959) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:38Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980833 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:39Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:39Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(2) (reproduce-bug-9tpfr-4222579959) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:40Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:40Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-4222579959
time="2020-08-14T03:48:40Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(2) (reproduce-bug-9tpfr-4222579959) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:41Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:41Z" level=info msg="Setting node reproduce-bug-9tpfr-4222579959 outputs"
time="2020-08-14T03:48:41Z" level=info msg="Updating node reproduce-bug-9tpfr-4222579959 status Pending -> Running"
time="2020-08-14T03:48:41Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980871 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:42Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:42Z" level=info msg="Updating node reproduce-bug-9tpfr-4222579959 status Running -> Failed"
time="2020-08-14T03:48:42Z" level=info msg="Updating node reproduce-bug-9tpfr-4222579959 message: failed with exit code 1"
time="2020-08-14T03:48:42Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: Backoff for 1 minute 0 seconds" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:42Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980880 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:43Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:43Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-4222579959 completed"
time="2020-08-14T03:49:42Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node has maxDuration set, setting executionDeadline to: Mon Jan 01 00:00:00 +0000 (a long while ago)" namespace=default node="reproduce-bug-9tpfr.process-tasks(3:3)" workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: " namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="No more retries left. Failing..." namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr-3505932898 phase Running -> Failed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: No more retries left" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr-3505932898 finished: 2020-08-14 03:49:42.72020564 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr-1670762753 phase Running -> Failed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr-1670762753 finished: 2020-08-14 03:49:42.720482407 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="Skipped node reproduce-bug-9tpfr-247809182 initialized Omitted (message: omitted: depends condition not met)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="Outbound nodes of reproduce-bug-9tpfr set to [reproduce-bug-9tpfr-247809182]" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr finished: 2020-08-14 03:49:42.720741695 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="Checking daemoned children of reproduce-bug-9tpfr" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="Updated phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="Marking workflow completed" namespace=default workflow=reproduce-bug-9tpfr
E0814 03:49:42.782781       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980880", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a055ab00e9c8, ext:43177831423928, loc:(*time.Location)(0x2953ca0)}}, Count:12, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:49:42Z" level=info msg="Checking daemoned children of " namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="Workflow update successful" namespace=default phase=Succeeded resourceVersion=9981935 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:43Z" level=info msg="reproduce-bug-9tpfr released all acquired locks"
time="2020-08-14T03:49:44Z" level=info msg="Workflow update successful" namespace=default phase=Succeeded resourceVersion=9981935 workflow=reproduce-bug-9tpfr

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

created time in 2 months

startednackjicholson/aiosql

started time in 2 months

startedhazelcast/hazelcast-jet

started time in 2 months

issue openedargoproj/argo

Output artifacts are uploaded even when the container failed

Summary

Output artifacts are uploaded even when the container failed.

Artifacts shouldn't be uploaded if the container failed.

Not sure is this a feature by design or a bug, but as it's very surprising to me, and took me very long time to debug, so I would report it as a bug. The background, I have a process keeps writing data to a file, and I add the file to artifacts to be uploaded once it finished. I also set a livenessProbe to ensure that worker is healthy. Sometimes, the process got killed and the process exited with code 137. And I realize the output file is uploaded anyway. I have an upfront skipping check for the existence of the file in my GCP bucket. The side effect of uploading the artifacts even when container failed is to have an incomplete file in the cloud storage and my retry task will think it's done already and skip.

I tried my best to find an option for turning off uploading artifacts no matter what, but cannot find any so far. I think in many cases uploading artifacts when container fails still make sense, like keep the error log in the cloud storage. It's very easy to workaround, I can just upload the file with a command line in the container script, but just wonder maybe it would be nice to have an option to disable it? So that I can still utilize the benefits of artifact output. I think it could be something like a when keyword for a task but for artifacts instead?

Diagnostics

What version of Argo Workflows are you running?

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: reproduce-bug-
spec:
  entrypoint: upload 
  templates:
  - name: upload
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        echo "hi" > /var/my-file
        sleep 10
        exit 1
      resources:
        requests:
          cpu: 250m
    outputs:
      artifacts:
      - name: my-file
        path: /var/my-file
        optional: true
        archive:
          none: {}
        s3:
          endpoint: storage.googleapis.com
          bucket: my-bucket-data
          key: _tmp/my-file
          accessKeySecret:
            name: argo-secrets
            key: gcs.s3.access.key
          secretKeySecret:
            name: argo-secrets
            key: gcs.s3.secret.key

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

created time in 2 months

pull request commentenvoy/lambda-convert

yaml file for Github Actions.

LGTM as well 👍

mei-envoy

comment created time in 2 months

startedbevyengine/bevy

started time in 2 months

issue commentargoproj/argo

DAG hanging

just happened to run into the same bug, where my generator script is returning an empty json payload, here's the yaml I used for reproducing it:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: reproduce-bug-
spec:
  entrypoint: start
  templates:
  - name: start
    dag:
      tasks:
      - name: first-step
        template: first-step
      - name: second-step
        dependencies: [first-step]
        template: second-step
        arguments:
          parameters:
          - name: chunk
            value: "{{item}}"
        withParam: "{{tasks.first-step.outputs.result}}"
      - name: third-step
        dependencies: [second-step]
        template: third-step

  - name: first-step
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        python3 -c 'import os, json; print(json.dumps([]))'
      resources:
        requests:
          cpu: 250m

  - name: second-step
    inputs:
      parameters:
      - name: chunk
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        echo hi {{$inputs.parameters.chunk}}
      resources:
        requests:
          cpu: 100m

  - name: third-step
    script:
      image: busybox
      command: [sh]
      source: |
        echo fin
      resources:
        requests:
          cpu: 100m

I return empty json on purpose, as I have a finish maker detection step in my real workflow, I wanted to skip the whole DAG if the finish marker file is already there in place in cloud storage. And JSON result is required even if when condition didn't pass. And because of that, turned out returning empty json will make DAG hangs forever. So I changed it to return a dummy value and skip in the downstream with when condition as well.

I guess maybe it makes sense to have a feature to skip the whole DAG without even providing JSON result. Anyway, just added a possible case people may want to return empty JSON on purpose on top of it.

eestewart

comment created time in 2 months

issue closedargoproj/argo

Argo API returning "Error: Request entity too large: limit is 3145728" when workflow is huge

Summary

The argo API should be able to return payload bigger than 3MB given if the whole workflow graph is huge

Diagnostics

v2.10.0-rc6

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: reproduce-bug-
spec:
  # keep workflows for 1d (86,400 seconds)
  ttlStrategy:
    secondsAfterCompletion: 86400
  # delete all pods as soon as they complete
  podGC:
    strategy: OnPodCompletion
  serviceAccountName: argo-workflow
  entrypoint: start
  arguments:
    parameters:
    - name: index
      value: "0000"

  templates:
  - name: start
    inputs:
      parameters:
      - name: index
    dag:
      tasks:
      - name: first-step
        template: first-step
        arguments:
          parameters:
          - name: index
            value: "{{inputs.parameters.index}}"
      - name: second-step
        dependencies: [first-step]
        template: second-step
        arguments:
          parameters:
          - name: index
            value: "{{inputs.parameters.index}}"
          - name: chunk
            value: "{{item}}"
        withParam: "{{tasks.first-step.outputs.result}}"
      - name: third-step
        dependencies: [second-step]
        template: third-step
        arguments:
          parameters:
          - name: index
            value: "{{inputs.parameters.index}}"
      - name: forth-step
        dependencies: [third-step]
        template: start
        arguments:
          parameters:
          - name: index
            value: "{{tasks.third-step.outputs.result}}"
        when: "{{tasks.third-step.outputs.result}} != EXIT"

  - name: first-step
    inputs:
      parameters:
      - name: index
    # allow to work for 5 minutes
    activeDeadlineSeconds: 300
    retryStrategy:
      limit: 5
      retryPolicy: "Always"
      backoff:
        duration: "15s"
        factor: 2
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        python3 -c 'import os, json; print(json.dumps([str(i) for i in range(300)]))'
      env:
      - name: LC_ALL
        value: C.UTF-8
      - name: LANG
        value: C.UTF-8
      resources:
        requests:
          cpu: 250m

  - name: second-step
    inputs:
      parameters:
      - name: index
      - name: chunk
    # allow to work for 30 minutes
    activeDeadlineSeconds: 1800
    retryStrategy:
      limit: 2
      retryPolicy: "Always"
      backoff:
        duration: "15s"
        factor: 2
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        echo hi {{$inputs.parameters.index}} {{$inputs.parameters.chunk}}
      env:
      - name: LC_ALL
        value: C.UTF-8
      - name: LANG
        value: C.UTF-8
      resources:
        requests:
          cpu: 100m

  - name: third-step
    inputs:
      parameters:
      - name: index
    # allow to work for 5 minutes
    activeDeadlineSeconds: 300
    script:
      image: python:3.7.8
      command: [python3]
      source: |
        current_index = int("{{inputs.parameters.index}}")
        next_index = current_index + 1
        if next_index>= 1024:
          print("EXIT")
        else:
          print(f"{next_index:04d}")
      env:
      - name: LC_ALL
        value: C.UTF-8
      - name: LANG
        value: C.UTF-8
      resources:
        requests:
          cpu: 100m

When you are trying to run a workflow that has very deep recursive times and very wide repeating tasks in the loop. Given my example, it's 1024 * 300, i.e around 300k tasks in this workflow. After a while, when enough tasks processed by argo, and you can open the argo web UI to see what's going on in the workflow page. You will notice error like this

Error: Request entity too large: limit is 3145728

Tried argo command like

argo get @latest

And the response could be something like this

Name:                reproduce-bug-2dr4r
Namespace:           default
ServiceAccount:      argo-workflow
Status:              Error
Message:             Request entity too large: limit is 3145728
Conditions:
 Completed           True
Created:             Tue Aug 11 02:41:06 -0700 (8 hours ago)
Started:             Tue Aug 11 02:41:06 -0700 (8 hours ago)
Finished:            Tue Aug 11 08:07:10 -0700 (3 hours ago)
Duration:            5 hours 26 minutes

I guess argo's http server for HTTP have a very small request size limit set at 3MB. I looked into the config but didn't found any option for controlling HTTP response size, did I miss something? Or maybe it makes sense to add an option for controlling this value to help serving big workflows?

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

closed time in 2 months

fangpenlin

issue commentargoproj/argo

Argo API returning "Error: Request entity too large: limit is 3145728" when workflow is huge

So, I ran with persistence option on with off loading large workflow on, doesn't seem to see the same problem. However, the web UI failed to load at some point, and my PostgreSQL seems like crashing. Also it's just too painful to have thousands of jobs in one workflow. So I turn the whole workflow into Workflow of workflows. So far it seems like working beautifully. Closing this issue for now, if people running into similar problem, I would recommend workflow of workflows instead of crazy size workflow.

fangpenlin

comment created time in 2 months

issue commentargoproj/argo

Argo API returning "Error: Request entity too large: limit is 3145728" when workflow is huge

@alexec ah, no wonder. the error message is a bit misleading though. I will turn on persistence / off load and try again.

the reproducing sample code I provided is simplified from my actual workflow, I didn't have time to run it and see if it can reproduce, I guess some part I removed in the code makes it not big enough to trigger the same issue.

anyway, thanks for the help, will update if the problem is gone.

fangpenlin

comment created time in 2 months

issue openedargoproj/argo

Argo API returning "Error: Request entity too large: limit is 3145728" when workflow is huge

Summary

The argo API should be able to return payload bigger than 3MB given if the whole workflow graph is huge

Diagnostics

v2.10.0-rc6

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: reproduce-bug-
spec:
  # keep workflows for 1d (86,400 seconds)
  ttlStrategy:
    secondsAfterCompletion: 86400
  # delete all pods as soon as they complete
  podGC:
    strategy: OnPodCompletion
  serviceAccountName: argo-workflow
  entrypoint: start
  arguments:
    parameters:
    - name: index
      value: "0000"

  templates:
  - name: start
    inputs:
      parameters:
      - name: index
    dag:
      tasks:
      - name: first-step
        template: first-step
        arguments:
          parameters:
          - name: index
            value: "{{inputs.parameters.index}}"
      - name: second-step
        dependencies: [first-step]
        template: second-step
        arguments:
          parameters:
          - name: index
            value: "{{inputs.parameters.index}}"
          - name: chunk
            value: "{{item}}"
        withParam: "{{tasks.first-step.outputs.result}}"
      - name: third-step
        dependencies: [second-step]
        template: third-step
        arguments:
          parameters:
          - name: index
            value: "{{inputs.parameters.index}}"
      - name: forth-step
        dependencies: [third-step]
        template: start
        arguments:
          parameters:
          - name: index
            value: "{{tasks.third-step.outputs.result}}"
        when: "{{tasks.third-step.outputs.result}} != EXIT"

  - name: first-step
    inputs:
      parameters:
      - name: index
    # allow to work for 5 minutes
    activeDeadlineSeconds: 300
    retryStrategy:
      limit: 5
      retryPolicy: "Always"
      backoff:
        duration: "15s"
        factor: 2
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        python3 -c 'import os, json; print(json.dumps([str(i) for i in range(300)]))'
      env:
      - name: LC_ALL
        value: C.UTF-8
      - name: LANG
        value: C.UTF-8
      resources:
        requests:
          cpu: 250m

  - name: second-step
    inputs:
      parameters:
      - name: index
      - name: chunk
    # allow to work for 30 minutes
    activeDeadlineSeconds: 1800
    retryStrategy:
      limit: 2
      retryPolicy: "Always"
      backoff:
        duration: "15s"
        factor: 2
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        echo hi {{$inputs.parameters.index}} {{$inputs.parameters.chunk}}
      env:
      - name: LC_ALL
        value: C.UTF-8
      - name: LANG
        value: C.UTF-8
      resources:
        requests:
          cpu: 100m

  - name: third-step
    inputs:
      parameters:
      - name: index
    # allow to work for 5 minutes
    activeDeadlineSeconds: 300
    script:
      image: python:3.7.8
      command: [python3]
      source: |
        current_index = int("{{inputs.parameters.index}}")
        next_index = current_index + 1
        if next_index>= 1024:
          print("EXIT")
        else:
          print(f"{next_index:04d}")
      env:
      - name: LC_ALL
        value: C.UTF-8
      - name: LANG
        value: C.UTF-8
      resources:
        requests:
          cpu: 100m

When you are trying to run a workflow that has very deep recursive times and very wide repeating tasks in the loop. Given my example, it's 1024 * 300, i.e around 300k tasks in this workflow. After a while, when enough tasks processed by argo, and you can open the argo web UI to see what's going on in the workflow page. You will notice error like this

Error: Request entity too large: limit is 3145728

Tried argo command like

argo get @latest

And the response could be something like this

Name:                reproduce-bug-2dr4r
Namespace:           default
ServiceAccount:      argo-workflow
Status:              Error
Message:             Request entity too large: limit is 3145728
Conditions:
 Completed           True
Created:             Tue Aug 11 02:41:06 -0700 (8 hours ago)
Started:             Tue Aug 11 02:41:06 -0700 (8 hours ago)
Finished:            Tue Aug 11 08:07:10 -0700 (3 hours ago)
Duration:            5 hours 26 minutes

I guess argo's http server for HTTP have a very small request size limit set at 3MB. I looked into the config but didn't found any option for controlling HTTP response size, did I miss something? Or maybe it makes sense to add an option for controlling this value to help serving big workflows?

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

created time in 2 months

issue commentargoproj/argo

Unable to load GCS artifact which is a not empty directory

I can confirm this bug. I looked into the code and I think I know what's going on here. The GCS artifact loader is trying to list files with the given key as prefix

https://github.com/argoproj/argo/blob/cb3536f9d1dd64258c1c3d737bb115bdab923e58/workflow/artifacts/gcs/gcs.go#L84

And try to create the intermediate folders if it needs to

https://github.com/argoproj/argo/blob/cb3536f9d1dd64258c1c3d737bb115bdab923e58/workflow/artifacts/gcs/gcs.go#L104

The problem is, if you are trying to pull a file from a bucket with common prefix shared between different files like this

result
result.metadata

It seems like the loader will try to first create a local file result.tmp, and later on, the TrimPrefix making the code thinks result.metadata is a sub-folder file .metadata under result. So it ends up trying to create a folder result.tmp. But is already a file we created previously, so it failed.

puchake