Yeah, +1 from me on the value of auto-provisioning an LB for the ingress gateway on infrastructures that support it. I've also found that to be convenient with GKE clusters and with PKS on GCP and vSphere with NSX-T. I do think the property names could be adjusted to be clearer and better organized (maybe ingress_gateway.enable_load_balancer and ingress_gateway.static_ip -- no reason to mention Istio as it's an implementation detail).

Thanks, @christopherclark !

Add meeting video for Sept 1 2020

Thanks, @christopherclark !

Thanks, @piyalibanerjee !

@mnitchev You mentioned that the Eirini team had introduced a task-completer component to work around this issue? Is that related to https://www.pivotaltracker.com/story/show/172175910?

Thanks, @staylor14 ! Merged.

This list isn't exhaustive, and is primarily off the top of my head. We can have more complete updates gathered from CLI, VAT, and Service Enablement next time.

@selzoc interesting, I did notice that there are two non-web processes for the app (executable-jar and task) and that they don't seem to have routes mapped to them. Maybe Eirini is choking on those ones?

Original issue on cf-for-k8s: https://github.com/cloudfoundry/cf-for-k8s/issues/287

Based on associated slack threads (https://cloudfoundry.slack.com/archives/CH9LF6V1P/p1594916021423300 and https://cloudfoundry.slack.com/archives/CH9LF6V1P/p1594928317432200), CC's behavior in cf-for-k8s seems to be the cause, so I'm opening an issue on the capi-k8s-release repo for the CAPI team to track.

As another data point, I see the same error when pushing the java app from the CF acceptance tests with the v7 CLI.

Describe the bug

Pushing the spring-music sample CF app against an up-to-date cf-for-k8s installation with the v7 CLI builds the image successfully, but fails to start the app. Running cf start spring-music also fails as a command, but the app does then start in the background.

Tracing the cf push command shows that Cloud Controller is returning a 500 on a POST to the /v3/apps/:app_guid/actions/restart endpoint.

Pushing this app with the v6 CLI does succeed, though.

FYI @cloudfoundry/cf-capi

To Reproduce

1. Install cf-for-k8s from latest master-branch build
2. Target an org and space in the installation with the latest v7 CF CLI.
3. Clone spring-music and build its JAR file locally with gradle.
4. Push the spring-music app and observe that it fails with a 500 status-code error from a CC API call after building the image.

<details> <summary>CF push failure</summary>

$ CF_TRACE=true cf push spring-music
(snip)
REQUEST: [2020-07-16T08:59:34-07:00]
PATCH /v3/apps/5b2a9a19-dd50-4c95-974b-38e4be1db608/relationships/current_droplet HTTP/1.1
Host: api.c4k.haas-424.pez.vmware.com
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: cf/7.0.1+fb3f929c2.2020-06-24 (go1.13.8; amd64 darwin)
{
  "data": {
    "guid": "7a1b8445-741f-48aa-a900-deb1d465d464"
  }
}


RESPONSE: [2020-07-16T08:59:34-07:00]
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Thu, 16 Jul 2020 15:59:34 GMT
Referrer-Policy: strict-origin-when-cross-origin
Server: istio-envoy
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Envoy-Upstream-Service-Time: 102
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Runtime: 0.085240
X-Vcap-Request-Id: 21283f64-dc81-4705-ba93-4756eab4a24e::34e144b5-9453-4db3-ad75-8e9c2c2055df
X-Xss-Protection: 1; mode=block
{
  "data": {
    "guid": "7a1b8445-741f-48aa-a900-deb1d465d464"
  },
  "links": {
    "related": {
      "href": "https://api.c4k.haas-424.pez.vmware.com/v3/apps/5b2a9a19-dd50-4c95-974b-38e4be1db608/droplets/current"
    },
    "self": {
      "href": "https://api.c4k.haas-424.pez.vmware.com/v3/apps/5b2a9a19-dd50-4c95-974b-38e4be1db608/relationships/current_droplet"
    }
  }
}

Waiting for app spring-music to start...

REQUEST: [2020-07-16T08:59:34-07:00]
POST /v3/apps/5b2a9a19-dd50-4c95-974b-38e4be1db608/actions/restart HTTP/1.1
Host: api.c4k.haas-424.pez.vmware.com
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
User-Agent: cf/7.0.1+fb3f929c2.2020-06-24 (go1.13.8; amd64 darwin)

RESPONSE: [2020-07-16T08:59:35-07:00]
HTTP/1.1 500 Internal Server Error
Content-Type: application/json; charset=utf-8
Date: Thu, 16 Jul 2020 15:59:35 GMT
Referrer-Policy: strict-origin-when-cross-origin
Server: istio-envoy
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Envoy-Upstream-Service-Time: 482
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Runtime: 0.453171
X-Vcap-Request-Id: 2956c47b-b491-4c50-ac6a-d2ecf5e89715::053bff97-eea9-497c-80e1-2cd25b533bf7
X-Xss-Protection: 1; mode=block
{
  "errors": [
    {
      "code": 10001,
      "detail": "An unknown error occurred.",
      "title": "UnknownError"
    }
  ]
}

</details>

<details> <summary>Stacktrace in CC logs</summary>

$ kapp logs -a cf -m cf-api-server% -c cf-api-server -f
(snip)
cf-api-server-7b7bd77f98-xscg5 > cf-api-server | {"timestamp":"2020-07-16T15:59:35.459408695Z","message":"Request failed: 500: {\"errors\"=>[{\"title\"=>\"UnknownError\", \"detail\"=>\"An unknown error occurred.\", \"code\"=>10001, \"test_mode_info\"=>{\"detail\"=>\"HTTPClient::KeepAliveDisconnected: \", \"title\"=>\"CF-KeepAliveDisconnected\", \"backtrace\"=>[\"/usr/local/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:808:in `block in parse_header'\", \"/usr/local/lib/ruby/2.5.0/timeout.rb:93:in `block in timeout'\", \"/usr/local/lib/ruby/2.5.0/timeout.rb:103:in `timeout'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:801:in `parse_header'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:784:in `read_header'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:561:in `get_header'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1299:in `do_get_header'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1245:in `do_get_block'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1019:in `block in do_request'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1138:in `rescue in protect_keep_alive_disconnected'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1131:in `protect_keep_alive_disconnected'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1014:in `do_request'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:856:in `request'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:775:in `put'\", \"/cloud_controller_ng/lib/cloud_controller/opi/apps_client.rb:17:in `desire_app'\", \"/cloud_controller_ng/lib/cloud_controller/diego/desire_app_handler.rb:10:in `create_or_update_app'\", \"/cloud_controller_ng/lib/cloud_controller/diego/messenger.rb:28:in `send_desire_request'\", \"/cloud_controller_ng/lib/cloud_controller/diego/runner.rb:23:in `block in start'\", \"/cloud_controller_ng/lib/cloud_controller/diego/runner.rb:41:in `with_logging'\", \"/cloud_controller_ng/lib/cloud_controller/diego/runner.rb:23:in `start'\", \"/cloud_controller_ng/lib/cloud_controller/backends/copilot_runner_delegator.rb:17:in `start'\", \"/cloud_controller_ng/app/actions/process_restart.rb:23:in `block in restart'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/sequel-5.22.0/lib/sequel/database/transactions.rb:229:in `block in transaction'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/sequel-5.22.0/lib/sequel/connection_pool/threaded.rb:88:in `hold'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/sequel-5.22.0/lib/sequel/database/connecting.rb:270:in `synchronize'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/sequel-5.22.0/lib/sequel/database/transactions.rb:199:in `transaction'\", \"/cloud_controller_ng/app/actions/process_restart.rb:8:in `restart'\", \"/cloud_controller_ng/app/actions/app_restart.rb:18:in `block (2 levels) in restart'\", \"/cloud_controller_ng/app/actions/app_restart.rb:17:in `each'\", \"/cloud_controller_ng/app/actions/app_restart.rb:17:in `block in restart'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/sequel-5.22.0/lib/sequel/database/transactions.rb:258:in `_transaction'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/sequel-5.22.0/lib/sequel/database/transactions.rb:233:in `block in transaction'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/sequel-5.22.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/sequel-5.22.0/lib/sequel/database/connecting.rb:270:in `synchronize'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/sequel-5.22.0/lib/sequel/database/transactions.rb:199:in `transaction'\", \"/cloud_controller_ng/app/actions/app_restart.rb:13:in `restart'\", \"/cloud_controller_ng/app/controllers/v3/apps_controller.rb:206:in `restart'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/abstract_controller/base.rb:194:in `process_action'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/rendering.rb:30:in `process_action'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:132:in `run_callbacks'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/abstract_controller/callbacks.rb:41:in `process_action'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/rescue.rb:22:in `process_action'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/notifications.rb:168:in `block in instrument'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/notifications.rb:168:in `instrument'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/abstract_controller/base.rb:134:in `process'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionview-5.2.4.3/lib/action_view/rendering.rb:32:in `process'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal.rb:191:in `dispatch'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_controller/metal.rb:252:in `dispatch'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/routing/route_set.rb:34:in `serve'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/journey/router.rb:35:in `each'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/journey/router.rb:35:in `serve'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/routing/route_set.rb:840:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/http/content_security_policy.rb:18:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:98:in `run_callbacks'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/activesupport-5.2.4.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/executor.rb:14:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'\", \"/cloud_controller_ng/middleware/request_logs.rb:38:in `call'\", \"/cloud_controller_ng/middleware/security_context_setter.rb:19:in `call'\", \"/cloud_controller_ng/middleware/vcap_request_id.rb:15:in `call'\", \"/cloud_controller_ng/middleware/cors.rb:49:in `call_app'\", \"/cloud_controller_ng/middleware/cors.rb:14:in `call'\", \"/cloud_controller_ng/middleware/request_metrics.rb:12:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/thin-1.7.2/lib/thin/connection.rb:86:in `block in pre_process'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/thin-1.7.2/lib/thin/connection.rb:84:in `catch'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/thin-1.7.2/lib/thin/connection.rb:84:in `pre_process'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/thin-1.7.2/lib/thin/connection.rb:50:in `block in process'\", \"/usr/local/lib/ruby/gems/2.5.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in `block in spawn_threadpool'\"]}}]}","log_level":"error","source":"cc.api","data":{"request_guid":"2956c47b-b491-4c50-ac6a-d2ecf5e89715::053bff97-eea9-497c-80e1-2cd25b533bf7"},"thread_id":47227938764260,"fiber_id":47227937214680,"process_id":1,"file":"/cloud_controller_ng/app/controllers/v3/errors_controller.rb","lineno":11,"method":"internal_error"}

</details>

Expected behavior

cf push should exit with status 0 and the app should be running afterwards.

Additional context

cf-for-k8s SHA

https://github.com/cloudfoundry/cf-for-k8s/commit/6a4bf75106f1c3b5625579f4c91df94b5533c0ad

Deploy instructions

ytt -f config -f ~/workspace/environments/haas-424/cf-for-k8s-values | kapp deploy -a cf -y -f-

Cluster information

TKGI v1.8 (k8s 1.17.5) on vSphere with NSX-T container networking.

CLI versions

paste output of the following commands

1. ytt --version: ytt version 0.28.0
2. kapp --version: kapp version 0.30.0
3. kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-07-04T14:53:16Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5+vmware.1", GitCommit:"f8b685623a975d20a9b35685c24611c44d464b3b", GitTreeState:"clean", BuildDate:"2020-05-01T21:32:30Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

4. cf version: cf version 7.0.1+fb3f929c2.2020-06-24

As an update, the Eirini story to support CF_INSTANCE_INDEX specifically is at https://www.pivotaltracker.com/story/show/173150663. Could we get that prioritized? This env var is part of the CF "downwards" API, such as it is, and we are aware of existing categories of CF apps whose instances behave differently based on whether or not the instance index is 0. (/cc @zrob as I think he has some more specifics on this that I can't recall at the moment.)