profile
viewpoint
Fang-Pen Lin fangpenlin Niantic San Francisco https://fangpenlin.com One-Person-Army Software Engineer

fangpenlin/avataaars-generator 1388

Simple generator React app for avataaars

fangpenlin/avataaars 444

React component for avataaars

fangpenlin/bugbuzz-python 106

Fall in love with debugging

fangpenlin/ansible-docker-demo 47

A simple ansible project for demonstrating how to build a docker image

fangpenlin/ansible-inside-docker-demo 16

This repo demonstrates how to pack an Ansible playbook into a docker image

fangpenlin/crane 11

Dockerfile builder

fangpenlin/avoid_disaster 5

Script backups easily to S3 using Python

fangpenlin/bugbuzz-dashboard 3

Dashboard for bugbuzz

startedstrimzi/strimzi-ui

started time in 14 hours

startedlinkedin/cruise-control

started time in a day

startedscholzj/zoo-entrance

started time in 2 days

startedconfluentinc/schema-registry

started time in 2 days

startedjedisct1/wasm-crypto

started time in 4 days

startedjedisct1/libsodium.js

started time in 4 days

startedDiceBear/avatars

started time in 5 days

startedhynek/argon2-cffi

started time in 5 days

startedtonyg/js-nacl

started time in 5 days

startedWebAssembly/binaryen

started time in 5 days

startedstreamnative/kop

started time in 5 days

startedantelle/argon2-browser

started time in 6 days

startedsegmentio/topicctl

started time in 8 days

startedstrimzi/strimzi-kafka-operator

started time in 8 days

startedbryanyang0528/ksql-python

started time in 14 days

startedconfluentinc/ksql

started time in 15 days

startedrycus86/prometheus_flask_exporter

started time in 19 days

startedbenoitc/gunicorn

started time in 19 days

startedstorax/kubedoom

started time in 20 days

push eventfangpenlin/fangpenlin

Fang-Pen Lin

commit sha 839c44066fa9db9a7b56b9b8284f54f0fed27d27

Update README.md

view details

push time in a month

push eventfangpenlin/fangpenlin

Fang-Pen Lin

commit sha c348bfc41939cce505e804a9f634f09406c3d68e

Update README.md

view details

push time in a month

push eventfangpenlin/fangpenlin

Fang-Pen Lin

commit sha 0e8cc8da0df14b0a8c76a2b7dba222351928ee7f

Update README.md

view details

push time in a month

push eventfangpenlin/fangpenlin

Fang-Pen Lin

commit sha 43d62ab99a21486096c08ad4e213787dd9fdc7ee

Update README.md

view details

push time in a month

startedrs/jplot

started time in a month

startedrs/jaggr

started time in a month

startedrakyll/hey

started time in a month

created tagfangpenlin/rocket-launcher-sampleapp

tag0.0.4

Sample app generated from rocket-launcher project template

created time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen Lin

commit sha 623f67bb651e906b86306c0374c07307a6541756

Generated from rocket-launcher d43b9d9cd3ccfeece04e92690e4ba68d6dc13054

view details

push time in a month

push eventfangpenlin/rocket-launcher

Fang-Pen

commit sha d43b9d9cd3ccfeece04e92690e4ba68d6dc13054

Don't add verbose to pipenv install

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen Lin

commit sha 741d0ad9131601a541a59959d97a09cc861b54c1

Generated from rocket-launcher 62774a5b30cfa22d23a4764a3185d02c64a8abe5

view details

push time in a month

push eventfangpenlin/rocket-launcher

Fang-Pen

commit sha 62774a5b30cfa22d23a4764a3185d02c64a8abe5

Ignore venv

view details

push time in a month

startedTheAllenChou/unity-physics-constraints

started time in a month

created tagfangpenlin/rocket-launcher-sampleapp

tag0.0.3

Sample app generated from rocket-launcher project template

created time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen Lin

commit sha 956ca748e20b0243d8aef3a75e2dadfde18b00d5

Generated from rocket-launcher 662ba08f636ad386f7d8373cc4f984adb1796939

view details

push time in a month

push eventfangpenlin/rocket-launcher

Fang-Pen

commit sha 662ba08f636ad386f7d8373cc4f984adb1796939

Add dockerfile

view details

push time in a month

created tagfangpenlin/rocket-launcher-sampleapp

tag0.0.2

Sample app generated from rocket-launcher project template

created time in a month

created tagfangpenlin/rocket-launcher-sampleapp

tag0.0.1

Sample app generated from rocket-launcher project template

created time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen Lin

commit sha ab7313bb026665a355d8d21e4ca442444060f8f4

Generated from rocket-launcher febc7ca4a8f910ec8a531d773583893461894219

view details

push time in a month

push eventfangpenlin/rocket-launcher

Fang-Pen

commit sha febc7ca4a8f910ec8a531d773583893461894219

Fix circle CI

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen Lin

commit sha ad0d9d58d6abb0c9639a14ac8b1c1a1f3194dc6b

Generated from rocket-launcher 7b09e284803d125638375d54a23c4ab1a1463ecb

view details

push time in a month

push eventfangpenlin/rocket-launcher

Fang-Pen

commit sha 7b09e284803d125638375d54a23c4ab1a1463ecb

Add ci code for docker publication

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen Lin

commit sha d3a4efca480183245fe509de447f30a3415d9b48

Generated from rocket-launcher d23363e3865cee8f3d824e54149e68813a67d801

view details

push time in a month

push eventfangpenlin/rocket-launcher

Fang-Pen

commit sha d23363e3865cee8f3d824e54149e68813a67d801

Remove outdated initdb

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen Lin

commit sha ed08d3a3dfa1571a3e6263dcb6299d359991a03b

Generated from rocket-launcher 9c277ab3b741ac11c7aeb48c9e13ce707bc11357

view details

push time in a month

push eventfangpenlin/rocket-launcher

Fang-Pen

commit sha 9c277ab3b741ac11c7aeb48c9e13ce707bc11357

Fix code style

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen Lin

commit sha 05a243f51a688a04cefbecb59efb19624b6362f0

Generated from rocket-launcher da7cb2c88a07004622181fe38069c11fe0ab3d2c

view details

push time in a month

push eventfangpenlin/rocket-launcher

Fang-Pen

commit sha da7cb2c88a07004622181fe38069c11fe0ab3d2c

Bring over changes in sample app

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha e04d3bdde6af4c6e4ad95a303e633a4c1d5eda74

Add test for register

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha cdfd873f5722600771d9b86108e99270ea6b40d7

Expires reset password link

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha ddb8ba65f731a660f36775d5678f87b942d47d26

Add test for reset password

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha aa6c74dd21ad210eb57d1ee5e83a6ceb1b16079a

Reset password

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha cd84100c9455d06932a443a69330bacbe620ced3

Remove not needed slash

view details

Fang-Pen

commit sha 0f3d50cbd04c7f66567da9df51b353971e40f5a8

Add part of forgot password

view details

Fang-Pen

commit sha 2bc3a7f3e435adcdf3c97bec7a599592a06b9657

Add flask mail, send rest password email

view details

Fang-Pen

commit sha 23b2dfb8ce9add2b7f03357c47969b27b4a4159b

Add test for reset password

view details

Fang-Pen

commit sha 31e2b69d0699398abd30fd446b06ae156dc99077

Fix timezone issue

view details

push time in a month

startedraryelcostasouza/pyTranscriber

started time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha 87228f9dcea87ef41a0b80b1ab9b3a203a531b1c

Add missing identity_changed

view details

Fang-Pen

commit sha 10d1986af1a22a5a12864caccef20fcb4004ca0c

Refactor login and logout

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha 2d8d9113208dc567fbd7596d459abedf615190f7

Fix broken tests

view details

Fang-Pen

commit sha f1dcfcbd23c2e987cd61b6db8aaabebda9ce83f2

Remove unused stuff

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha d12edff6b349311667962d77c4563ccdcab81e93

Update model

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha 963ae73f1969637c5f410c38088d0ea08ab387c7

Get rid of unused stuff

view details

Fang-Pen

commit sha d8daf58ce78b2012e482271153a83120a0fc3349

Add factory boy and tests for login

view details

Fang-Pen

commit sha 1519e035eff872574c18de2bc33995a47c45461d

Simplified login tests

view details

Fang-Pen

commit sha 5cfe656368b55a56a0204f8a7edafec5d418e2bd

No need for encoding now

view details

Fang-Pen

commit sha a92f01086e4f8a4ea66be6b2a4fa27732e8ae45c

Remove all not needed encoding header

view details

Fang-Pen

commit sha 4f72e10c6b0b30263342478b64496691bf696821

Clean code

view details

Fang-Pen

commit sha ec2a880eebc4e8be1bdb05707ca22b45220773c2

Add tests for forms

view details

Fang-Pen

commit sha 8ec456e818ebf2934fedd4970186ba68a0eb69c1

Clean code

view details

push time in a month

startedFactoryBoy/factory_boy

started time in a month

startedpytest-dev/pytest-factoryboy

started time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha 9fdbb698d2dec9fc06450c1d1a8496dddd0ccece

Fix some minor issues

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha 9107c3c12e3322070dfc4b603a33bcd59051a002

Add font-awesome

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha fa84ec15202982f55523c188ea4a90fce3fee3f7

Add principal

view details

push time in a month

push eventfangpenlin/rocket-launcher-sampleapp

Fang-Pen

commit sha 53c14270c532754cbb05f3c69497873cf4284702

Update project

view details

push time in a month

startedflask-admin/flask-admin

started time in a month

startedkelseyhightower/nocode

started time in 2 months

startedvitessio/vitess

started time in 2 months

push eventfangpenlin/avataaars-generator

Fang-Pen Lin

commit sha 9254849c7ac4dffa0bfae2b38cd5bb0f617a147f

Update README.md

view details

push time in 2 months

startedzalando/postgres-operator

started time in 2 months

startedan-tao/drogon

started time in 2 months

issue commentargoproj/argo

Pod's phase is failed, but the workflow node status is succeeded.

I think I encountered similar issue on GKE. The argo version I am using is 2.10.0. Last night I submitted a workflow which has 1024 pods in parallel for processing data and uploading to Google Cloud Storage (via gsutil command rather than artifacts output). This morning I noticed that the whole workflow is done, but yet there's one file missing. I looked into the log and realized the controller somehow think the node is done in very short time. I tried to find the log of the pod but can't find anything. I think the pod doesn't even get a chance to be up fully running, but soon it's dead. Yet somehow argo controller thinks the pod is succeeded. Here's the log for the controller:

[
  {
    "textPayload": "time=\"2020-08-30T09:59:33Z\" level=info msg=\"Updating node data-2bhpn-4179820782 status Pending -> Succeeded\"\n",
    "insertId": "u36f3kmuxix7veq4p",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "cluster_name": "dotdb3",
        "project_id": "dotdb-231204",
        "container_name": "controller",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "location": "us-central1",
        "namespace_name": "argo-0"
      }
    },
    "timestamp": "2020-08-30T09:59:33.794653184Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:59:39.464856256Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:59:25Z\" level=info msg=\"Skipped pod data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782) creation: already exists\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7ven1i",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "cluster_name": "dotdb3",
        "location": "us-central1",
        "namespace_name": "argo-0",
        "project_id": "dotdb-231204"
      }
    },
    "timestamp": "2020-08-30T09:59:25.474679262Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:59:29.444708649Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:59:14Z\" level=info msg=\"Skipped pod data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782) creation: already exists\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vei8g",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "namespace_name": "argo-0",
        "container_name": "controller",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "project_id": "dotdb-231204",
        "location": "us-central1",
        "cluster_name": "dotdb3"
      }
    },
    "timestamp": "2020-08-30T09:59:14.325085578Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:59:19.443556733Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:59:03Z\" level=info msg=\"Skipped pod data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782) creation: already exists\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vee1l",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "container_name": "controller",
        "cluster_name": "dotdb3",
        "namespace_name": "argo-0",
        "location": "us-central1",
        "project_id": "dotdb-231204"
      }
    },
    "timestamp": "2020-08-30T09:59:03.944302801Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:59:09.462969696Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:59:00Z\" level=info msg=\"Updating node data-2bhpn-4179820782 message: Unschedulable: 0/20 nodes are available: 13 Insufficient cpu, 2 node(s) had taints that the pod didn't tolerate, 5 node(s) didn't match node selector.\"\n",
    "insertId": "u36f3kmuxix7ved3n",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "namespace_name": "argo-0",
        "location": "us-central1",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "cluster_name": "dotdb3",
        "project_id": "dotdb-231204"
      }
    },
    "timestamp": "2020-08-30T09:59:00.312067761Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:59:04.583608911Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:54Z\" level=info msg=\"Skipped pod data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782) creation: already exists\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vdz2p",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1",
        "container_name": "controller",
        "cluster_name": "dotdb3",
        "project_id": "dotdb-231204",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "namespace_name": "argo-0"
      }
    },
    "timestamp": "2020-08-30T09:58:54.287322757Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:59.500998938Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:52Z\" level=info msg=\"insignificant pod change\" key=argo-0/data-2bhpn-4179820782\n",
    "insertId": "u36f3kmuxix7vdz0k",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1",
        "container_name": "controller",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "project_id": "dotdb-231204",
        "namespace_name": "argo-0",
        "cluster_name": "dotdb3"
      }
    },
    "timestamp": "2020-08-30T09:58:52.757956615Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:54.466535608Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:44Z\" level=info msg=\"Skipped pod data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782) creation: already exists\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vdu6l",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "cluster_name": "dotdb3",
        "container_name": "controller",
        "namespace_name": "argo-0",
        "project_id": "dotdb-231204",
        "location": "us-central1"
      }
    },
    "timestamp": "2020-08-30T09:58:44.137674055Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:49.554273725Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:32Z\" level=info msg=\"Skipped pod data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782) creation: already exists\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vdp5o",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "location": "us-central1",
        "namespace_name": "argo-0",
        "cluster_name": "dotdb3",
        "project_id": "dotdb-231204"
      }
    },
    "timestamp": "2020-08-30T09:58:32.338253656Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:34.444975036Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:26Z\" level=info msg=\"Updating node data-2bhpn-4179820782 message: Unschedulable: 0/18 nodes are available: 13 Insufficient cpu, 5 node(s) didn't match node selector.\"\n",
    "insertId": "u36f3kmuxix7vdo7p",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1",
        "cluster_name": "dotdb3",
        "container_name": "controller",
        "namespace_name": "argo-0",
        "project_id": "dotdb-231204",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7"
      }
    },
    "timestamp": "2020-08-30T09:58:26.739056781Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:29.543231340Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:22Z\" level=info msg=\"insignificant pod change\" key=argo-0/data-2bhpn-4179820782\n",
    "insertId": "u36f3kmuxix7vdk2s",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "location": "us-central1",
        "project_id": "dotdb-231204",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "cluster_name": "dotdb3",
        "namespace_name": "argo-0"
      }
    },
    "timestamp": "2020-08-30T09:58:22.894848019Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:24.469135596Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:22Z\" level=info msg=\"Created pod: data-2bhpn.scan-shard-delta(560:0560)(0) (data-2bhpn-4179820782)\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vdk2o",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "cluster_name": "dotdb3",
        "location": "us-central1",
        "project_id": "dotdb-231204",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "namespace_name": "argo-0"
      }
    },
    "timestamp": "2020-08-30T09:58:22.888713326Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:24.469135596Z"
  },
  {
    "textPayload": "time=\"2020-08-30T09:58:22Z\" level=info msg=\"Pod node data-2bhpn-4179820782 initialized Pending\" namespace=argo-0 workflow=data-2bhpn\n",
    "insertId": "u36f3kmuxix7vdk2m",
    "resource": {
      "type": "k8s_container",
      "labels": {
        "container_name": "controller",
        "project_id": "dotdb-231204",
        "namespace_name": "argo-0",
        "pod_name": "argo-0-workflow-controller-75d5d566-4h9t7",
        "location": "us-central1",
        "cluster_name": "dotdb3"
      }
    },
    "timestamp": "2020-08-30T09:58:22.839015156Z",
    "severity": "ERROR",
    "logName": "projects/dotdb-231204/logs/stderr",
    "receiveTimestamp": "2020-08-30T09:58:24.469135596Z"
  }
]

Normally each of these node would take 20 minutes to finish, but this node from start to finish only took 1 minute, certainly not right. I cannot reproduce it, and not much other log I can find. These are what I can provide.

juchaosong

comment created time in 2 months

startedscrapedia/scrapy-useragents

started time in 2 months

startedtwisted/twisted

started time in 2 months

startedJays2Kings/DS4Windows

started time in 2 months

startedopenairplay/open-airplay

started time in 2 months

push eventfangpenlin/fangpenlin

Fang-Pen Lin

commit sha 9db82296b04924286a0262e862d484ee0a185615

Update README.md

view details

push time in 2 months

push eventfangpenlin/fangpenlin

Fang-Pen Lin

commit sha 36004e6d5bcf37cbe9b3b553e35d19a4a8439369

Update README.md

view details

push time in 2 months

push eventfangpenlin/fangpenlin

Fang-Pen Lin

commit sha 4ed95c1f3fed7208ae0531a9880861314b7d951d

Update README.md

view details

push time in 2 months

push eventfangpenlin/fangpenlin

Fang-Pen Lin

commit sha ecaacac70bb8d4a3e2dea233937b7cccc3a5dc2b

Update README.md

view details

push time in 2 months

create barnchfangpenlin/fangpenlin

branch : master

created branch time in 2 months

created repositoryfangpenlin/fangpenlin

created time in 2 months

issue openedargoproj/argo

DAG with generated jobs from loop failure won't make the workflow fail

Summary

When you create a workflow with DAG which has generated jobs from loop, the failed job created by the loop won't make the whole workflow fail, instead, it will still get passed as Succeeded. And the final job depending on the generated tasks will be ignored.

Name:                reproduce-bug-9tpfr
Namespace:           default
ServiceAccount:      argo-workflow
Status:              Succeeded
Conditions:
 Completed           True
Created:             Thu Aug 13 20:47:23 -0700 (6 minutes ago)
Started:             Thu Aug 13 20:47:23 -0700 (6 minutes ago)
Finished:            Thu Aug 13 20:49:42 -0700 (4 minutes ago)
Duration:            2 minutes 19 seconds
ResourcesDuration:   36s*(1 cpu),36s*(100Mi memory)

STEP                          TEMPLATE       PODNAME                         DURATION  MESSAGE
 ✔ reproduce-bug-9tpfr        start
 ├-✔ gen-tasks(0)             gen-tasks      reproduce-bug-9tpfr-3595395365  3s
 ├-✔ process-tasks(0:0)(0)    process-tasks  reproduce-bug-9tpfr-3662458541  13s
 ├-✔ process-tasks(1:1)(0)    process-tasks  reproduce-bug-9tpfr-988357889   5s
 ├-✔ process-tasks(2:2)(0)    process-tasks  reproduce-bug-9tpfr-3450100829  4s
 ├-✖ process-tasks(3:3)       process-tasks                                            No more retries left
 | ├-✖ process-tasks(3:3)(0)  process-tasks  reproduce-bug-9tpfr-2008331609  13s       failed with exit code 1
 | ├-✖ process-tasks(3:3)(1)  process-tasks  reproduce-bug-9tpfr-2612472988  5s        failed with exit code 1
 | └-✖ process-tasks(3:3)(2)  process-tasks  reproduce-bug-9tpfr-4222579959  3s        failed with exit code 1
 ├-✔ process-tasks(4:4)(0)    process-tasks  reproduce-bug-9tpfr-3522592509  13s
 ├-✔ process-tasks(5:5)(0)    process-tasks  reproduce-bug-9tpfr-600389385   4s
 ├-✔ process-tasks(6:6)(0)    process-tasks  reproduce-bug-9tpfr-3207714925  4s
 ├-✔ process-tasks(7:7)(0)    process-tasks  reproduce-bug-9tpfr-585646929   3s
 ├-✔ process-tasks(8:8)(0)    process-tasks  reproduce-bug-9tpfr-1384515437  12s
 ├-✔ process-tasks(9:9)(0)    process-tasks  reproduce-bug-9tpfr-2475724017  4s

It should fail the whole workflow instead when any of the generated tasks from loop fails

Diagnostics

What version of Argo Workflows are you running?

v2.10.0-rc6

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  creationTimestamp: "2020-08-14T03:47:23Z"
  generateName: reproduce-bug-
  generation: 21
  labels:
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/phase: Succeeded
  name: reproduce-bug-9tpfr
  namespace: default
  resourceVersion: "9981935"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/default/workflows/reproduce-bug-9tpfr
  uid: dcdbbf44-dde0-11ea-b7b7-42010a800164
spec:
  arguments: {}
  entrypoint: start
  serviceAccountName: argo-workflow
  templates:
  - arguments: {}
    dag:
      tasks:
      - arguments: {}
        name: gen-tasks
        template: gen-tasks
      - arguments:
          parameters:
          - name: chunk
            value: '{{item}}'
        dependencies:
        - gen-tasks
        name: process-tasks
        template: process-tasks
        withParam: '{{tasks.gen-tasks.outputs.result}}'
      - arguments: {}
        dependencies:
        - process-tasks
        name: finish
        template: finish
    inputs: {}
    metadata: {}
    name: start
    outputs: {}
  - activeDeadlineSeconds: 300
    arguments: {}
    inputs: {}
    metadata: {}
    name: gen-tasks
    outputs: {}
    retryStrategy:
      backoff:
        duration: 15s
        factor: 2
      limit: 5
      retryPolicy: Always
    script:
      command:
      - bash
      image: python:3
      name: ""
      resources:
        requests:
          cpu: 250m
      source: |
        set -e
        python3 -c 'import os, json; print(json.dumps([str(i) for i in range(10)]))'
  - activeDeadlineSeconds: 1800
    arguments: {}
    inputs:
      parameters:
      - name: chunk
    metadata: {}
    name: process-tasks
    outputs: {}
    retryStrategy:
      backoff:
        duration: 15s
        factor: 2
      limit: 2
      retryPolicy: Always
    script:
      command:
      - bash
      image: python:3
      name: ""
      resources:
        requests:
          cpu: 100m
      source: |
        set -e
        chunk="{{inputs.parameters.chunk}}"
        if [[ $chunk == "3" ]]; then
          echo "failed"
          exit 1
        fi
        echo "process $chunk"
  - activeDeadlineSeconds: 300
    arguments: {}
    inputs: {}
    metadata: {}
    name: finish
    outputs: {}
    script:
      command:
      - sh
      image: busybox
      name: ""
      resources:
        requests:
          cpu: 100m
      source: |
        echo fin
status:
  conditions:
  - status: "True"
    type: Completed
  finishedAt: "2020-08-14T03:49:42Z"
  nodes:
    reproduce-bug-9tpfr:
      children:
      - reproduce-bug-9tpfr-1525049382
      displayName: reproduce-bug-9tpfr
      finishedAt: "2020-08-14T03:49:42Z"
      id: reproduce-bug-9tpfr
      name: reproduce-bug-9tpfr
      outboundNodes:
      - reproduce-bug-9tpfr-247809182
      phase: Succeeded
      startedAt: "2020-08-14T03:47:23Z"
      templateName: start
      templateScope: local/reproduce-bug-9tpfr
      type: DAG
    reproduce-bug-9tpfr-247809182:
      boundaryID: reproduce-bug-9tpfr
      displayName: finish
      finishedAt: "2020-08-14T03:49:42Z"
      id: reproduce-bug-9tpfr-247809182
      message: 'omitted: depends condition not met'
      name: reproduce-bug-9tpfr.finish
      phase: Omitted
      startedAt: "2020-08-14T03:49:42Z"
      templateName: finish
      templateScope: local/reproduce-bug-9tpfr
      type: Skipped
    reproduce-bug-9tpfr-546685502:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-3207714925
      displayName: process-tasks(6:6)
      finishedAt: "2020-08-14T03:47:37Z"
      id: reproduce-bug-9tpfr-546685502
      inputs:
        parameters:
        - name: chunk
          value: "6"
      name: reproduce-bug-9tpfr.process-tasks(6:6)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-585646929:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(7:7)(0)
      finishedAt: "2020-08-14T03:47:33Z"
      hostNodeName: gke-dotdb3-dotdb-pool-c3910e16-nb0l
      id: reproduce-bug-9tpfr-585646929
      inputs:
        parameters:
        - name: chunk
          value: "7"
      name: reproduce-bug-9tpfr.process-tasks(7:7)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-600389385:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(5:5)(0)
      finishedAt: "2020-08-14T03:47:33Z"
      hostNodeName: gke-dotdb3-dotdb-pool-c3910e16-nb0l
      id: reproduce-bug-9tpfr-600389385
      inputs:
        parameters:
        - name: chunk
          value: "5"
      name: reproduce-bug-9tpfr.process-tasks(5:5)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-850457006:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-3450100829
      displayName: process-tasks(2:2)
      finishedAt: "2020-08-14T03:47:36Z"
      id: reproduce-bug-9tpfr-850457006
      inputs:
        parameters:
        - name: chunk
          value: "2"
      name: reproduce-bug-9tpfr.process-tasks(2:2)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-988357889:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(1:1)(0)
      finishedAt: "2020-08-14T03:47:34Z"
      hostNodeName: gke-dotdb3-dotdb-pool-3779d805-hkcv
      id: reproduce-bug-9tpfr-988357889
      inputs:
        parameters:
        - name: chunk
          value: "1"
      name: reproduce-bug-9tpfr.process-tasks(1:1)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 2
        memory: 2
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-1384515437:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(8:8)(0)
      finishedAt: "2020-08-14T03:47:42Z"
      hostNodeName: gke-dotdb3-dotdb-pool-2e8afe0d-km90
      id: reproduce-bug-9tpfr-1384515437
      inputs:
        parameters:
        - name: chunk
          value: "8"
      name: reproduce-bug-9tpfr.process-tasks(8:8)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 6
        memory: 6
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-1525049382:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-3595395365
      displayName: gen-tasks
      finishedAt: "2020-08-14T03:47:29Z"
      id: reproduce-bug-9tpfr-1525049382
      name: reproduce-bug-9tpfr.gen-tasks
      outputs:
        exitCode: "0"
        result: '["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]'
      phase: Succeeded
      startedAt: "2020-08-14T03:47:23Z"
      templateName: gen-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-1602779214:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-3522592509
      displayName: process-tasks(4:4)
      finishedAt: "2020-08-14T03:47:45Z"
      id: reproduce-bug-9tpfr-1602779214
      inputs:
        parameters:
        - name: chunk
          value: "4"
      name: reproduce-bug-9tpfr.process-tasks(4:4)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-1670762753:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-2788826366
      - reproduce-bug-9tpfr-3412607194
      - reproduce-bug-9tpfr-850457006
      - reproduce-bug-9tpfr-3505932898
      - reproduce-bug-9tpfr-1602779214
      - reproduce-bug-9tpfr-4143798034
      - reproduce-bug-9tpfr-546685502
      - reproduce-bug-9tpfr-3415721642
      - reproduce-bug-9tpfr-3221614398
      - reproduce-bug-9tpfr-3518128714
      displayName: process-tasks
      finishedAt: "2020-08-14T03:49:42Z"
      id: reproduce-bug-9tpfr-1670762753
      name: reproduce-bug-9tpfr.process-tasks
      phase: Failed
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: TaskGroup
    reproduce-bug-9tpfr-2008331609:
      boundaryID: reproduce-bug-9tpfr
      displayName: process-tasks(3:3)(0)
      finishedAt: "2020-08-14T03:47:42Z"
      hostNodeName: gke-dotdb3-dotdb-pool-2e8afe0d-km90
      id: reproduce-bug-9tpfr-2008331609
      inputs:
        parameters:
        - name: chunk
          value: "3"
      message: failed with exit code 1
      name: reproduce-bug-9tpfr.process-tasks(3:3)(0)
      outputs:
        exitCode: "1"
      phase: Failed
      resourcesDuration:
        cpu: 6
        memory: 6
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-2475724017:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(9:9)(0)
      finishedAt: "2020-08-14T03:47:34Z"
      hostNodeName: gke-dotdb3-dotdb-pool-3779d805-hkcv
      id: reproduce-bug-9tpfr-2475724017
      inputs:
        parameters:
        - name: chunk
          value: "9"
      name: reproduce-bug-9tpfr.process-tasks(9:9)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-2612472988:
      boundaryID: reproduce-bug-9tpfr
      displayName: process-tasks(3:3)(1)
      finishedAt: "2020-08-14T03:48:05Z"
      hostNodeName: gke-dotdb3-dotdb-pool-2e8afe0d-km90
      id: reproduce-bug-9tpfr-2612472988
      inputs:
        parameters:
        - name: chunk
          value: "3"
      message: failed with exit code 1
      name: reproduce-bug-9tpfr.process-tasks(3:3)(1)
      outputs:
        exitCode: "1"
      phase: Failed
      resourcesDuration:
        cpu: 2
        memory: 2
      startedAt: "2020-08-14T03:48:00Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-2788826366:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-3662458541
      displayName: process-tasks(0:0)
      finishedAt: "2020-08-14T03:47:44Z"
      id: reproduce-bug-9tpfr-2788826366
      inputs:
        parameters:
        - name: chunk
          value: "0"
      name: reproduce-bug-9tpfr.process-tasks(0:0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-3207714925:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(6:6)(0)
      finishedAt: "2020-08-14T03:47:34Z"
      hostNodeName: gke-dotdb3-dotdb-pool-3779d805-hkcv
      id: reproduce-bug-9tpfr-3207714925
      inputs:
        parameters:
        - name: chunk
          value: "6"
      name: reproduce-bug-9tpfr.process-tasks(6:6)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 2
        memory: 2
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-3221614398:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-1384515437
      displayName: process-tasks(8:8)
      finishedAt: "2020-08-14T03:47:44Z"
      id: reproduce-bug-9tpfr-3221614398
      inputs:
        parameters:
        - name: chunk
          value: "8"
      name: reproduce-bug-9tpfr.process-tasks(8:8)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-3412607194:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-988357889
      displayName: process-tasks(1:1)
      finishedAt: "2020-08-14T03:47:36Z"
      id: reproduce-bug-9tpfr-3412607194
      inputs:
        parameters:
        - name: chunk
          value: "1"
      name: reproduce-bug-9tpfr.process-tasks(1:1)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-3415721642:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-585646929
      displayName: process-tasks(7:7)
      finishedAt: "2020-08-14T03:47:36Z"
      id: reproduce-bug-9tpfr-3415721642
      inputs:
        parameters:
        - name: chunk
          value: "7"
      name: reproduce-bug-9tpfr.process-tasks(7:7)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-3450100829:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(2:2)(0)
      finishedAt: "2020-08-14T03:47:33Z"
      hostNodeName: gke-dotdb3-dotdb-pool-3779d805-hkcv
      id: reproduce-bug-9tpfr-3450100829
      inputs:
        parameters:
        - name: chunk
          value: "2"
      name: reproduce-bug-9tpfr.process-tasks(2:2)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-3505932898:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-2008331609
      - reproduce-bug-9tpfr-2612472988
      - reproduce-bug-9tpfr-4222579959
      displayName: process-tasks(3:3)
      finishedAt: "2020-08-14T03:49:42Z"
      id: reproduce-bug-9tpfr-3505932898
      inputs:
        parameters:
        - name: chunk
          value: "3"
      message: No more retries left
      name: reproduce-bug-9tpfr.process-tasks(3:3)
      phase: Failed
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-3518128714:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-2475724017
      displayName: process-tasks(9:9)
      finishedAt: "2020-08-14T03:47:37Z"
      id: reproduce-bug-9tpfr-3518128714
      inputs:
        parameters:
        - name: chunk
          value: "9"
      name: reproduce-bug-9tpfr.process-tasks(9:9)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:30Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-3522592509:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(4:4)(0)
      finishedAt: "2020-08-14T03:47:42Z"
      hostNodeName: gke-dotdb3-dotdb-pool-2e8afe0d-km90
      id: reproduce-bug-9tpfr-3522592509
      inputs:
        parameters:
        - name: chunk
          value: "4"
      name: reproduce-bug-9tpfr.process-tasks(4:4)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 6
        memory: 6
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-3595395365:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-1670762753
      displayName: gen-tasks(0)
      finishedAt: "2020-08-14T03:47:26Z"
      hostNodeName: gke-dotdb3-dotdb-pool-3779d805-hkcv
      id: reproduce-bug-9tpfr-3595395365
      name: reproduce-bug-9tpfr.gen-tasks(0)
      outputs:
        exitCode: "0"
        result: '["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]'
      phase: Succeeded
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2020-08-14T03:47:23Z"
      templateName: gen-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-3662458541:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(0:0)(0)
      finishedAt: "2020-08-14T03:47:42Z"
      hostNodeName: gke-dotdb3-dotdb-pool-2e8afe0d-km90
      id: reproduce-bug-9tpfr-3662458541
      inputs:
        parameters:
        - name: chunk
          value: "0"
      name: reproduce-bug-9tpfr.process-tasks(0:0)(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      resourcesDuration:
        cpu: 6
        memory: 6
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
    reproduce-bug-9tpfr-4143798034:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-600389385
      displayName: process-tasks(5:5)
      finishedAt: "2020-08-14T03:47:36Z"
      id: reproduce-bug-9tpfr-4143798034
      inputs:
        parameters:
        - name: chunk
          value: "5"
      name: reproduce-bug-9tpfr.process-tasks(5:5)
      outputs:
        exitCode: "0"
      phase: Succeeded
      startedAt: "2020-08-14T03:47:29Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Retry
    reproduce-bug-9tpfr-4222579959:
      boundaryID: reproduce-bug-9tpfr
      children:
      - reproduce-bug-9tpfr-247809182
      displayName: process-tasks(3:3)(2)
      finishedAt: "2020-08-14T03:48:40Z"
      hostNodeName: gke-dotdb3-dotdb-pool-3779d805-hkcv
      id: reproduce-bug-9tpfr-4222579959
      inputs:
        parameters:
        - name: chunk
          value: "3"
      message: failed with exit code 1
      name: reproduce-bug-9tpfr.process-tasks(3:3)(2)
      outputs:
        exitCode: "1"
      phase: Failed
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2020-08-14T03:48:37Z"
      templateName: process-tasks
      templateScope: local/reproduce-bug-9tpfr
      type: Pod
  phase: Succeeded
  resourcesDuration:
    cpu: 36
    memory: 36
  startedAt: "2020-08-14T03:47:23Z"

time="2020-08-14T03:47:23Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:23Z" level=info msg="Updated phase  -> Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:23Z" level=info msg="DAG node reproduce-bug-9tpfr initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:23Z" level=info msg="All of node reproduce-bug-9tpfr.gen-tasks dependencies [] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:23Z" level=info msg="Retry node reproduce-bug-9tpfr-1525049382 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:23Z" level=info msg="Pod node reproduce-bug-9tpfr-3595395365 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:23Z" level=info msg="Created pod: reproduce-bug-9tpfr.gen-tasks(0) (reproduce-bug-9tpfr-3595395365)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:24Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980007 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:25Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:25Z" level=info msg="Updating node reproduce-bug-9tpfr-3595395365 message: PodInitializing"
time="2020-08-14T03:47:26Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-3595395365
time="2020-08-14T03:47:26Z" level=info msg="Skipped pod reproduce-bug-9tpfr.gen-tasks(0) (reproduce-bug-9tpfr-3595395365) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:27Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980037 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Setting node reproduce-bug-9tpfr-3595395365 outputs"
time="2020-08-14T03:47:29Z" level=info msg="Updating node reproduce-bug-9tpfr-3595395365 status Pending -> Succeeded"
time="2020-08-14T03:47:29Z" level=info msg="node reproduce-bug-9tpfr-1525049382 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="node reproduce-bug-9tpfr-1525049382 finished: 2020-08-14 03:47:29.356536963 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="TaskGroup node reproduce-bug-9tpfr-1670762753 initialized Running (message: )" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(0:0) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Retry node reproduce-bug-9tpfr-2788826366 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Pod node reproduce-bug-9tpfr-3662458541 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(1:1) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Retry node reproduce-bug-9tpfr-3412607194 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Pod node reproduce-bug-9tpfr-988357889 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(1:1)(0) (reproduce-bug-9tpfr-988357889)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(2:2) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Retry node reproduce-bug-9tpfr-850457006 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Pod node reproduce-bug-9tpfr-3450100829 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(2:2)(0) (reproduce-bug-9tpfr-3450100829)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(3:3) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Retry node reproduce-bug-9tpfr-3505932898 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Pod node reproduce-bug-9tpfr-2008331609 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(4:4) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Retry node reproduce-bug-9tpfr-1602779214 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Pod node reproduce-bug-9tpfr-3522592509 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(5:5) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Retry node reproduce-bug-9tpfr-4143798034 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:29Z" level=info msg="Pod node reproduce-bug-9tpfr-600389385 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(5:5)(0) (reproduce-bug-9tpfr-600389385)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(6:6) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Retry node reproduce-bug-9tpfr-546685502 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Pod node reproduce-bug-9tpfr-3207714925 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(6:6)(0) (reproduce-bug-9tpfr-3207714925)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(7:7) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Retry node reproduce-bug-9tpfr-3415721642 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Pod node reproduce-bug-9tpfr-585646929 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(7:7)(0) (reproduce-bug-9tpfr-585646929)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(8:8) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Retry node reproduce-bug-9tpfr-3221614398 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Pod node reproduce-bug-9tpfr-1384515437 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="All of node reproduce-bug-9tpfr.process-tasks(9:9) dependencies [gen-tasks] completed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Retry node reproduce-bug-9tpfr-3518128714 initialized Running" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Pod node reproduce-bug-9tpfr-2475724017 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(9:9)(0) (reproduce-bug-9tpfr-2475724017)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:30Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980134 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3595395365 completed"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-988357889 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-1384515437 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-2008331609 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-2475724017 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-3207714925 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-3450100829 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-3522592509 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-3662458541 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-585646929 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Updating node reproduce-bug-9tpfr-600389385 message: PodInitializing"
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(1:1)(0) (reproduce-bug-9tpfr-988357889) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(2:2)(0) (reproduce-bug-9tpfr-3450100829) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(5:5)(0) (reproduce-bug-9tpfr-600389385) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:32Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(6:6)(0) (reproduce-bug-9tpfr-3207714925) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:33Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(7:7)(0) (reproduce-bug-9tpfr-585646929) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:33Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:33Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(9:9)(0) (reproduce-bug-9tpfr-2475724017) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:33Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980174 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:33Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-600389385
time="2020-08-14T03:47:33Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-585646929
time="2020-08-14T03:47:33Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-3450100829
time="2020-08-14T03:47:34Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3595395365 completed"
time="2020-08-14T03:47:34Z" level=info msg="Setting node reproduce-bug-9tpfr-3450100829 outputs"
time="2020-08-14T03:47:34Z" level=info msg="Updating node reproduce-bug-9tpfr-3450100829 status Pending -> Running"
time="2020-08-14T03:47:34Z" level=info msg="Setting node reproduce-bug-9tpfr-585646929 outputs"
time="2020-08-14T03:47:34Z" level=info msg="Updating node reproduce-bug-9tpfr-585646929 status Pending -> Running"
time="2020-08-14T03:47:34Z" level=info msg="Setting node reproduce-bug-9tpfr-600389385 outputs"
time="2020-08-14T03:47:34Z" level=info msg="Updating node reproduce-bug-9tpfr-600389385 status Pending -> Running"
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(1:1)(0) (reproduce-bug-9tpfr-988357889) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(6:6)(0) (reproduce-bug-9tpfr-3207714925) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(9:9)(0) (reproduce-bug-9tpfr-2475724017) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980201 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:34Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-2475724017
time="2020-08-14T03:47:34Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-988357889
time="2020-08-14T03:47:34Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-3207714925
time="2020-08-14T03:47:35Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:35Z" level=info msg="Setting node reproduce-bug-9tpfr-988357889 outputs"
time="2020-08-14T03:47:35Z" level=info msg="Updating node reproduce-bug-9tpfr-988357889 status Pending -> Succeeded"
time="2020-08-14T03:47:35Z" level=info msg="Updating node reproduce-bug-9tpfr-3450100829 status Running -> Succeeded"
time="2020-08-14T03:47:35Z" level=info msg="Updating node reproduce-bug-9tpfr-585646929 status Running -> Succeeded"
time="2020-08-14T03:47:35Z" level=info msg="Updating node reproduce-bug-9tpfr-600389385 status Running -> Succeeded"
time="2020-08-14T03:47:35Z" level=info msg="Setting node reproduce-bug-9tpfr-2475724017 outputs"
time="2020-08-14T03:47:35Z" level=info msg="Updating node reproduce-bug-9tpfr-2475724017 status Pending -> Running"
time="2020-08-14T03:47:35Z" level=info msg="Setting node reproduce-bug-9tpfr-3207714925 outputs"
time="2020-08-14T03:47:35Z" level=info msg="Updating node reproduce-bug-9tpfr-3207714925 status Pending -> Running"
time="2020-08-14T03:47:35Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:35Z" level=info msg="node reproduce-bug-9tpfr-3412607194 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-3412607194 finished: 2020-08-14 03:47:36.000227026 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-850457006 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-850457006 finished: 2020-08-14 03:47:36.001690292 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-4143798034 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-4143798034 finished: 2020-08-14 03:47:36.061716791 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-3415721642 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="node reproduce-bug-9tpfr-3415721642 finished: 2020-08-14 03:47:36.062801551 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:36Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980235 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-988357889 completed"
time="2020-08-14T03:47:37Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3450100829 completed"
time="2020-08-14T03:47:37Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-585646929 completed"
time="2020-08-14T03:47:37Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-600389385 completed"
time="2020-08-14T03:47:37Z" level=info msg="Updating node reproduce-bug-9tpfr-2475724017 status Running -> Succeeded"
time="2020-08-14T03:47:37Z" level=info msg="Updating node reproduce-bug-9tpfr-3207714925 status Running -> Succeeded"
time="2020-08-14T03:47:37Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
E0814 03:47:37.305030       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980235", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(9:9)(0)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a036507c4568, ext:43052386523483, loc:(*time.Location)(0x2953ca0)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
E0814 03:47:37.314170       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980235", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(6:6)(0)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a036529296b3, ext:43052421540523, loc:(*time.Location)(0x2953ca0)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:37Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="node reproduce-bug-9tpfr-546685502 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="node reproduce-bug-9tpfr-546685502 finished: 2020-08-14 03:47:37.338582183 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
E0814 03:47:37.344631       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980235", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(6:6)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a0365454ae88, ext:43052451037818, loc:(*time.Location)(0x2953ca0)}}, Count:4, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:37Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="node reproduce-bug-9tpfr-3518128714 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:37Z" level=info msg="node reproduce-bug-9tpfr-3518128714 finished: 2020-08-14 03:47:37.372769276 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
E0814 03:47:37.377068       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980235", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(9:9)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a0365653a04c, ext:43052484523046, loc:(*time.Location)(0x2953ca0)}}, Count:5, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:37Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980257 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:38Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:38Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3450100829 completed"
time="2020-08-14T03:47:38Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-585646929 completed"
time="2020-08-14T03:47:38Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-600389385 completed"
time="2020-08-14T03:47:38Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-988357889 completed"
time="2020-08-14T03:47:38Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-2475724017 completed"
time="2020-08-14T03:47:39Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3207714925 completed"
time="2020-08-14T03:47:39Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:39Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:39Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:39Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:39Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:40Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(0:0)(0) (reproduce-bug-9tpfr-3662458541) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:40Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:40Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:40Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:41Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:41Z" level=info msg="Updating node reproduce-bug-9tpfr-3662458541 status Pending -> Running"
time="2020-08-14T03:47:41Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(0) (reproduce-bug-9tpfr-2008331609) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:41Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(4:4)(0) (reproduce-bug-9tpfr-3522592509) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:41Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(8:8)(0) (reproduce-bug-9tpfr-1384515437) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:41Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980321 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:42Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-3522592509
time="2020-08-14T03:47:42Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-3662458541
time="2020-08-14T03:47:42Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-1384515437
time="2020-08-14T03:47:42Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:43Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-2008331609
time="2020-08-14T03:47:43Z" level=info msg="Setting node reproduce-bug-9tpfr-1384515437 outputs"
time="2020-08-14T03:47:43Z" level=info msg="Updating node reproduce-bug-9tpfr-1384515437 status Pending -> Running"
time="2020-08-14T03:47:43Z" level=info msg="Setting node reproduce-bug-9tpfr-2008331609 outputs"
time="2020-08-14T03:47:43Z" level=info msg="Updating node reproduce-bug-9tpfr-2008331609 status Pending -> Running"
time="2020-08-14T03:47:43Z" level=info msg="Setting node reproduce-bug-9tpfr-3522592509 outputs"
time="2020-08-14T03:47:43Z" level=info msg="Updating node reproduce-bug-9tpfr-3522592509 status Pending -> Running"
time="2020-08-14T03:47:43Z" level=info msg="Setting node reproduce-bug-9tpfr-3662458541 outputs"
time="2020-08-14T03:47:43Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980336 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:44Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:44Z" level=info msg="Updating node reproduce-bug-9tpfr-1384515437 status Running -> Succeeded"
time="2020-08-14T03:47:44Z" level=info msg="Updating node reproduce-bug-9tpfr-2008331609 status Running -> Failed"
time="2020-08-14T03:47:44Z" level=info msg="Updating node reproduce-bug-9tpfr-2008331609 message: failed with exit code 1"
time="2020-08-14T03:47:44Z" level=info msg="Updating node reproduce-bug-9tpfr-3662458541 status Running -> Succeeded"
time="2020-08-14T03:47:44Z" level=info msg="node reproduce-bug-9tpfr-2788826366 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:44Z" level=info msg="node reproduce-bug-9tpfr-2788826366 finished: 2020-08-14 03:47:44.530749349 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:44Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: Backoff for 15 seconds" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:44Z" level=info msg="node reproduce-bug-9tpfr-3221614398 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:44Z" level=info msg="node reproduce-bug-9tpfr-3221614398 finished: 2020-08-14 03:47:44.53177149 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
E0814 03:47:44.576680       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980336", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(8:8)(0)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a0381fca46cd, ext:43059643293861, loc:(*time.Location)(0x2953ca0)}}, Count:6, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:44Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980353 workflow=reproduce-bug-9tpfr
E0814 03:47:44.626797       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980336", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(0:0)(0)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a038236458c7, ext:43059703722712, loc:(*time.Location)(0x2953ca0)}}, Count:7, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
E0814 03:47:44.653302       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980336", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(0:0)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03825ed8795, ext:43059746267572, loc:(*time.Location)(0x2953ca0)}}, Count:8, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
E0814 03:47:44.690093       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980336", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(8:8)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03826f2f633, ext:43059763400748, loc:(*time.Location)(0x2953ca0)}}, Count:9, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:45Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:45Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-2008331609 completed"
time="2020-08-14T03:47:45Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3662458541 completed"
time="2020-08-14T03:47:45Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-1384515437 completed"
time="2020-08-14T03:47:45Z" level=info msg="Updating node reproduce-bug-9tpfr-3522592509 status Running -> Succeeded"
E0814 03:47:45.711454       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980353", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(4:4)(0)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a0386a3cd788, ext:43060818574184, loc:(*time.Location)(0x2953ca0)}}, Count:10, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:45Z" level=info msg="node reproduce-bug-9tpfr-1602779214 phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:45Z" level=info msg="node reproduce-bug-9tpfr-1602779214 finished: 2020-08-14 03:47:45.717442949 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
E0814 03:47:45.720901       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980353", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr.process-tasks(4:4)", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a0386ac7eeab, ext:43060827689616, loc:(*time.Location)(0x2953ca0)}}, Count:11, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:47:45Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980365 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:46Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:47:46Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-1384515437 completed"
time="2020-08-14T03:47:46Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-2008331609 completed"
time="2020-08-14T03:47:46Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3662458541 completed"
time="2020-08-14T03:47:46Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-3522592509 completed"
time="2020-08-14T03:48:00Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:00Z" level=info msg="node has maxDuration set, setting executionDeadline to: Mon Jan 01 00:00:00 +0000 (a long while ago)" namespace=default node="reproduce-bug-9tpfr.process-tasks(3:3)" workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:00Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: " namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:00Z" level=info msg="1 child nodes of reproduce-bug-9tpfr.process-tasks(3:3) failed. Trying again..." namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:00Z" level=info msg="Pod node reproduce-bug-9tpfr-2612472988 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:00Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(3:3)(1) (reproduce-bug-9tpfr-2612472988)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:00Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980515 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:01Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:02Z" level=info msg="Updating node reproduce-bug-9tpfr-2612472988 message: PodInitializing"
time="2020-08-14T03:48:02Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(1) (reproduce-bug-9tpfr-2612472988) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:02Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980527 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:03Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:03Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(1) (reproduce-bug-9tpfr-2612472988) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:04Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:05Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(1) (reproduce-bug-9tpfr-2612472988) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:05Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-2612472988
time="2020-08-14T03:48:05Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:06Z" level=info msg="Setting node reproduce-bug-9tpfr-2612472988 outputs"
time="2020-08-14T03:48:06Z" level=info msg="Updating node reproduce-bug-9tpfr-2612472988 status Pending -> Running"
time="2020-08-14T03:48:06Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980562 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:07Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:07Z" level=info msg="Updating node reproduce-bug-9tpfr-2612472988 status Running -> Failed"
time="2020-08-14T03:48:07Z" level=info msg="Updating node reproduce-bug-9tpfr-2612472988 message: failed with exit code 1"
time="2020-08-14T03:48:07Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: Backoff for 30 seconds" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:07Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980575 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:08Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:08Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-2612472988 completed"
time="2020-08-14T03:48:37Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:37Z" level=info msg="node has maxDuration set, setting executionDeadline to: Mon Jan 01 00:00:00 +0000 (a long while ago)" namespace=default node="reproduce-bug-9tpfr.process-tasks(3:3)" workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:37Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: " namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:37Z" level=info msg="2 child nodes of reproduce-bug-9tpfr.process-tasks(3:3) failed. Trying again..." namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:37Z" level=info msg="Pod node reproduce-bug-9tpfr-4222579959 initialized Pending" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:37Z" level=info msg="Created pod: reproduce-bug-9tpfr.process-tasks(3:3)(2) (reproduce-bug-9tpfr-4222579959)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:37Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980821 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:38Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:38Z" level=info msg="Updating node reproduce-bug-9tpfr-4222579959 message: PodInitializing"
time="2020-08-14T03:48:38Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(2) (reproduce-bug-9tpfr-4222579959) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:38Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980833 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:39Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:39Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(2) (reproduce-bug-9tpfr-4222579959) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:40Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:40Z" level=info msg="insignificant pod change" key=default/reproduce-bug-9tpfr-4222579959
time="2020-08-14T03:48:40Z" level=info msg="Skipped pod reproduce-bug-9tpfr.process-tasks(3:3)(2) (reproduce-bug-9tpfr-4222579959) creation: already exists" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:41Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:41Z" level=info msg="Setting node reproduce-bug-9tpfr-4222579959 outputs"
time="2020-08-14T03:48:41Z" level=info msg="Updating node reproduce-bug-9tpfr-4222579959 status Pending -> Running"
time="2020-08-14T03:48:41Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980871 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:42Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:42Z" level=info msg="Updating node reproduce-bug-9tpfr-4222579959 status Running -> Failed"
time="2020-08-14T03:48:42Z" level=info msg="Updating node reproduce-bug-9tpfr-4222579959 message: failed with exit code 1"
time="2020-08-14T03:48:42Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: Backoff for 1 minute 0 seconds" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:42Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=9980880 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:43Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:48:43Z" level=info msg="Labeled pod default/reproduce-bug-9tpfr-4222579959 completed"
time="2020-08-14T03:49:42Z" level=info msg="Processing workflow" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node has maxDuration set, setting executionDeadline to: Mon Jan 01 00:00:00 +0000 (a long while ago)" namespace=default node="reproduce-bug-9tpfr.process-tasks(3:3)" workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: " namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="No more retries left. Failing..." namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr-3505932898 phase Running -> Failed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr-3505932898 message: No more retries left" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr-3505932898 finished: 2020-08-14 03:49:42.72020564 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr-1670762753 phase Running -> Failed" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr-1670762753 finished: 2020-08-14 03:49:42.720482407 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="Skipped node reproduce-bug-9tpfr-247809182 initialized Omitted (message: omitted: depends condition not met)" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="Outbound nodes of reproduce-bug-9tpfr set to [reproduce-bug-9tpfr-247809182]" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="node reproduce-bug-9tpfr finished: 2020-08-14 03:49:42.720741695 +0000 UTC" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="Checking daemoned children of reproduce-bug-9tpfr" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="Updated phase Running -> Succeeded" namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="Marking workflow completed" namespace=default workflow=reproduce-bug-9tpfr
E0814 03:49:42.782781       1 event.go:263] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"reproduce-bug-9tpfr.162b05c9901aa7df", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Workflow", Namespace:"default", Name:"reproduce-bug-9tpfr", UID:"dcdbbf44-dde0-11ea-b7b7-42010a800164", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"9980880", FieldPath:""}, Reason:"WorkflowNodeSucceeded", Message:"(combined from similar events): Succeeded node reproduce-bug-9tpfr", Source:v1.EventSource{Component:"workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a03605cb37df, ext:43051207148013, loc:(*time.Location)(0x2953ca0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfc5a055ab00e9c8, ext:43177831423928, loc:(*time.Location)(0x2953ca0)}}, Count:12, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "reproduce-bug-9tpfr.162b05c9901aa7df" is forbidden: User "system:serviceaccount:argo:argo" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!)
time="2020-08-14T03:49:42Z" level=info msg="Checking daemoned children of " namespace=default workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:42Z" level=info msg="Workflow update successful" namespace=default phase=Succeeded resourceVersion=9981935 workflow=reproduce-bug-9tpfr
time="2020-08-14T03:49:43Z" level=info msg="reproduce-bug-9tpfr released all acquired locks"
time="2020-08-14T03:49:44Z" level=info msg="Workflow update successful" namespace=default phase=Succeeded resourceVersion=9981935 workflow=reproduce-bug-9tpfr


<!-- Issue Author: Don't delete this message to encourage other users to support your issue! --> Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

created time in 2 months

startednackjicholson/aiosql

started time in 2 months

startedhazelcast/hazelcast-jet

started time in 2 months

issue openedargoproj/argo

Output artifacts are uploaded even when the container failed

Summary

Output artifacts are uploaded even when the container failed.

Artifacts shouldn't be uploaded if the container failed.

Not sure is this a feature by design or a bug, but as it's very surprising to me, and took me very long time to debug, so I would report it as a bug. The background, I have a process keeps writing data to a file, and I add the file to artifacts to be uploaded once it finished. I also set a livenessProbe to ensure that worker is healthy. Sometimes, the process got killed and the process exited with code 137. And I realize the output file is uploaded anyway. I have an upfront skipping check for the existence of the file in my GCP bucket. The side effect of uploading the artifacts even when container failed is to have an incomplete file in the cloud storage and my retry task will think it's done already and skip.

I tried my best to find an option for turning off uploading artifacts no matter what, but cannot find any so far. I think in many cases uploading artifacts when container fails still make sense, like keep the error log in the cloud storage. It's very easy to workaround, I can just upload the file with a command line in the container script, but just wonder maybe it would be nice to have an option to disable it? So that I can still utilize the benefits of artifact output. I think it could be something like a when keyword for a task but for artifacts instead?

Diagnostics

What version of Argo Workflows are you running?

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: reproduce-bug-
spec:
  entrypoint: upload 
  templates:
  - name: upload
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        echo "hi" > /var/my-file
        sleep 10
        exit 1
      resources:
        requests:
          cpu: 250m
    outputs:
      artifacts:
      - name: my-file
        path: /var/my-file
        optional: true
        archive:
          none: {}
        s3:
          endpoint: storage.googleapis.com
          bucket: my-bucket-data
          key: _tmp/my-file
          accessKeySecret:
            name: argo-secrets
            key: gcs.s3.access.key
          secretKeySecret:
            name: argo-secrets
            key: gcs.s3.secret.key


<!-- Issue Author: Don't delete this message to encourage other users to support your issue! --> Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

created time in 2 months

pull request commentenvoy/lambda-convert

yaml file for Github Actions.

LGTM as well 👍

mei-envoy

comment created time in 2 months

startedbevyengine/bevy

started time in 2 months

issue commentargoproj/argo

DAG hanging

just happened to run into the same bug, where my generator script is returning an empty json payload, here's the yaml I used for reproducing it:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: reproduce-bug-
spec:
  entrypoint: start
  templates:
  - name: start
    dag:
      tasks:
      - name: first-step
        template: first-step
      - name: second-step
        dependencies: [first-step]
        template: second-step
        arguments:
          parameters:
          - name: chunk
            value: "{{item}}"
        withParam: "{{tasks.first-step.outputs.result}}"
      - name: third-step
        dependencies: [second-step]
        template: third-step

  - name: first-step
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        python3 -c 'import os, json; print(json.dumps([]))'
      resources:
        requests:
          cpu: 250m

  - name: second-step
    inputs:
      parameters:
      - name: chunk
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        echo hi {{$inputs.parameters.chunk}}
      resources:
        requests:
          cpu: 100m

  - name: third-step
    script:
      image: busybox
      command: [sh]
      source: |
        echo fin
      resources:
        requests:
          cpu: 100m

I return empty json on purpose, as I have a finish maker detection step in my real workflow, I wanted to skip the whole DAG if the finish marker file is already there in place in cloud storage. And JSON result is required even if when condition didn't pass. And because of that, turned out returning empty json will make DAG hangs forever. So I changed it to return a dummy value and skip in the downstream with when condition as well.

I guess maybe it makes sense to have a feature to skip the whole DAG without even providing JSON result. Anyway, just added a possible case people may want to return empty JSON on purpose on top of it.

eestewart

comment created time in 2 months

issue closedargoproj/argo

Argo API returning "Error: Request entity too large: limit is 3145728" when workflow is huge

Summary

The argo API should be able to return payload bigger than 3MB given if the whole workflow graph is huge

Diagnostics

v2.10.0-rc6

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: reproduce-bug-
spec:
  # keep workflows for 1d (86,400 seconds)
  ttlStrategy:
    secondsAfterCompletion: 86400
  # delete all pods as soon as they complete
  podGC:
    strategy: OnPodCompletion
  serviceAccountName: argo-workflow
  entrypoint: start
  arguments:
    parameters:
    - name: index
      value: "0000"

  templates:
  - name: start
    inputs:
      parameters:
      - name: index
    dag:
      tasks:
      - name: first-step
        template: first-step
        arguments:
          parameters:
          - name: index
            value: "{{inputs.parameters.index}}"
      - name: second-step
        dependencies: [first-step]
        template: second-step
        arguments:
          parameters:
          - name: index
            value: "{{inputs.parameters.index}}"
          - name: chunk
            value: "{{item}}"
        withParam: "{{tasks.first-step.outputs.result}}"
      - name: third-step
        dependencies: [second-step]
        template: third-step
        arguments:
          parameters:
          - name: index
            value: "{{inputs.parameters.index}}"
      - name: forth-step
        dependencies: [third-step]
        template: start
        arguments:
          parameters:
          - name: index
            value: "{{tasks.third-step.outputs.result}}"
        when: "{{tasks.third-step.outputs.result}} != EXIT"

  - name: first-step
    inputs:
      parameters:
      - name: index
    # allow to work for 5 minutes
    activeDeadlineSeconds: 300
    retryStrategy:
      limit: 5
      retryPolicy: "Always"
      backoff:
        duration: "15s"
        factor: 2
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        python3 -c 'import os, json; print(json.dumps([str(i) for i in range(300)]))'
      env:
      - name: LC_ALL
        value: C.UTF-8
      - name: LANG
        value: C.UTF-8
      resources:
        requests:
          cpu: 250m

  - name: second-step
    inputs:
      parameters:
      - name: index
      - name: chunk
    # allow to work for 30 minutes
    activeDeadlineSeconds: 1800
    retryStrategy:
      limit: 2
      retryPolicy: "Always"
      backoff:
        duration: "15s"
        factor: 2
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        echo hi {{$inputs.parameters.index}} {{$inputs.parameters.chunk}}
      env:
      - name: LC_ALL
        value: C.UTF-8
      - name: LANG
        value: C.UTF-8
      resources:
        requests:
          cpu: 100m

  - name: third-step
    inputs:
      parameters:
      - name: index
    # allow to work for 5 minutes
    activeDeadlineSeconds: 300
    script:
      image: python:3.7.8
      command: [python3]
      source: |
        current_index = int("{{inputs.parameters.index}}")
        next_index = current_index + 1
        if next_index>= 1024:
          print("EXIT")
        else:
          print(f"{next_index:04d}")
      env:
      - name: LC_ALL
        value: C.UTF-8
      - name: LANG
        value: C.UTF-8
      resources:
        requests:
          cpu: 100m


When you are trying to run a workflow that has very deep recursive times and very wide repeating tasks in the loop. Given my example, it's 1024 * 300, i.e around 300k tasks in this workflow. After a while, when enough tasks processed by argo, and you can open the argo web UI to see what's going on in the workflow page. You will notice error like this

Error: Request entity too large: limit is 3145728

Tried argo command like

argo get @latest

And the response could be something like this

Name:                reproduce-bug-2dr4r
Namespace:           default
ServiceAccount:      argo-workflow
Status:              Error
Message:             Request entity too large: limit is 3145728
Conditions:
 Completed           True
Created:             Tue Aug 11 02:41:06 -0700 (8 hours ago)
Started:             Tue Aug 11 02:41:06 -0700 (8 hours ago)
Finished:            Tue Aug 11 08:07:10 -0700 (3 hours ago)
Duration:            5 hours 26 minutes

I guess argo's http server for HTTP have a very small request size limit set at 3MB. I looked into the config but didn't found any option for controlling HTTP response size, did I miss something? Or maybe it makes sense to add an option for controlling this value to help serving big workflows?


<!-- Issue Author: Don't delete this message to encourage other users to support your issue! --> Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

closed time in 2 months

fangpenlin

issue commentargoproj/argo

Argo API returning "Error: Request entity too large: limit is 3145728" when workflow is huge

So, I ran with persistence option on with off loading large workflow on, doesn't seem to see the same problem. However, the web UI failed to load at some point, and my PostgreSQL seems like crashing. Also it's just too painful to have thousands of jobs in one workflow. So I turn the whole workflow into Workflow of workflows. So far it seems like working beautifully. Closing this issue for now, if people running into similar problem, I would recommend workflow of workflows instead of crazy size workflow.

fangpenlin

comment created time in 2 months

issue commentargoproj/argo

Argo API returning "Error: Request entity too large: limit is 3145728" when workflow is huge

@alexec ah, no wonder. the error message is a bit misleading though. I will turn on persistence / off load and try again.

the reproducing sample code I provided is simplified from my actual workflow, I didn't have time to run it and see if it can reproduce, I guess some part I removed in the code makes it not big enough to trigger the same issue.

anyway, thanks for the help, will update if the problem is gone.

fangpenlin

comment created time in 2 months

issue openedargoproj/argo

Argo API returning "Error: Request entity too large: limit is 3145728" when workflow is huge

Summary

The argo API should be able to return payload bigger than 3MB given if the whole workflow graph is huge

Diagnostics

v2.10.0-rc6

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: reproduce-bug-
spec:
  # keep workflows for 1d (86,400 seconds)
  ttlStrategy:
    secondsAfterCompletion: 86400
  # delete all pods as soon as they complete
  podGC:
    strategy: OnPodCompletion
  serviceAccountName: argo-workflow
  entrypoint: start
  arguments:
    parameters:
    - name: index
      value: "0000"

  templates:
  - name: start
    inputs:
      parameters:
      - name: index
    dag:
      tasks:
      - name: first-step
        template: first-step
        arguments:
          parameters:
          - name: index
            value: "{{inputs.parameters.index}}"
      - name: second-step
        dependencies: [first-step]
        template: second-step
        arguments:
          parameters:
          - name: index
            value: "{{inputs.parameters.index}}"
          - name: chunk
            value: "{{item}}"
        withParam: "{{tasks.first-step.outputs.result}}"
      - name: third-step
        dependencies: [second-step]
        template: third-step
        arguments:
          parameters:
          - name: index
            value: "{{inputs.parameters.index}}"
      - name: forth-step
        dependencies: [third-step]
        template: start
        arguments:
          parameters:
          - name: index
            value: "{{tasks.third-step.outputs.result}}"
        when: "{{tasks.third-step.outputs.result}} != EXIT"

  - name: first-step
    inputs:
      parameters:
      - name: index
    # allow to work for 5 minutes
    activeDeadlineSeconds: 300
    retryStrategy:
      limit: 5
      retryPolicy: "Always"
      backoff:
        duration: "15s"
        factor: 2
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        python3 -c 'import os, json; print(json.dumps([str(i) for i in range(300)]))'
      env:
      - name: LC_ALL
        value: C.UTF-8
      - name: LANG
        value: C.UTF-8
      resources:
        requests:
          cpu: 250m

  - name: second-step
    inputs:
      parameters:
      - name: index
      - name: chunk
    # allow to work for 30 minutes
    activeDeadlineSeconds: 1800
    retryStrategy:
      limit: 2
      retryPolicy: "Always"
      backoff:
        duration: "15s"
        factor: 2
    script:
      image: python:3
      command: [bash]
      source: |
        set -e
        echo hi {{$inputs.parameters.index}} {{$inputs.parameters.chunk}}
      env:
      - name: LC_ALL
        value: C.UTF-8
      - name: LANG
        value: C.UTF-8
      resources:
        requests:
          cpu: 100m

  - name: third-step
    inputs:
      parameters:
      - name: index
    # allow to work for 5 minutes
    activeDeadlineSeconds: 300
    script:
      image: python:3.7.8
      command: [python3]
      source: |
        current_index = int("{{inputs.parameters.index}}")
        next_index = current_index + 1
        if next_index>= 1024:
          print("EXIT")
        else:
          print(f"{next_index:04d}")
      env:
      - name: LC_ALL
        value: C.UTF-8
      - name: LANG
        value: C.UTF-8
      resources:
        requests:
          cpu: 100m


When you are trying to run a workflow that has very deep recursive times and very wide repeating tasks in the loop. Given my example, it's 1024 * 300, i.e around 300k tasks in this workflow. After a while, when enough tasks processed by argo, and you can open the argo web UI to see what's going on in the workflow page. You will notice error like this

Error: Request entity too large: limit is 3145728

Tried argo command like

argo get @latest

And the response could be something like this

Name:                reproduce-bug-2dr4r
Namespace:           default
ServiceAccount:      argo-workflow
Status:              Error
Message:             Request entity too large: limit is 3145728
Conditions:
 Completed           True
Created:             Tue Aug 11 02:41:06 -0700 (8 hours ago)
Started:             Tue Aug 11 02:41:06 -0700 (8 hours ago)
Finished:            Tue Aug 11 08:07:10 -0700 (3 hours ago)
Duration:            5 hours 26 minutes

I guess argo's http server for HTTP have a very small request size limit set at 3MB. I looked into the config but didn't found any option for controlling HTTP response size, did I miss something? Or maybe it makes sense to add an option for controlling this value to help serving big workflows?


<!-- Issue Author: Don't delete this message to encourage other users to support your issue! --> Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

created time in 2 months

issue commentargoproj/argo

Unable to load GCS artifact which is a not empty directory

I can confirm this bug. I looked into the code and I think I know what's going on here. The GCS artifact loader is trying to list files with the given key as prefix

https://github.com/argoproj/argo/blob/cb3536f9d1dd64258c1c3d737bb115bdab923e58/workflow/artifacts/gcs/gcs.go#L84

And try to create the intermediate folders if it needs to

https://github.com/argoproj/argo/blob/cb3536f9d1dd64258c1c3d737bb115bdab923e58/workflow/artifacts/gcs/gcs.go#L104

The problem is, if you are trying to pull a file from a bucket with common prefix shared between different files like this

  • result
  • result.metadata

It seems like the loader will try to first create a local file result.tmp, and later on, the TrimPrefix making the code thinks result.metadata is a sub-folder file .metadata under result. So it ends up trying to create a folder result.tmp. But is already a file we created previously, so it failed.

puchake

comment created time in 3 months

startedargoproj/argo-cd

started time in 3 months

startedargoproj/argo-ui

started time in 3 months

startedcloudhut/kowl

started time in 3 months

starteddanielqsj/kafka_exporter

started time in 3 months

startedprometheus/jmx_exporter

started time in 3 months

startedojdkbuild/ojdkbuild

started time in 3 months

startedargoproj/argo

started time in 3 months

more