profile
viewpoint

Ask questionshelm upgrade fails with spec.clusterIP: Invalid value: "": field is immutable

When issue helm upgrade, it shows errors like, ("my-service" change from "clusterIP: None" to "type: LoadBalancer" without field clusterIP)

Error: UPGRADE FAILED: Service "my-service" is invalid: spec.clusterIP: Invalid value: "": field is immutable 

However, all other pods with new version are still going to be restarted, except that "my-service" Type does not change to new type "LoadBalancer"

I understand that why upgrade failed because helm does not support changing on some certain fields. But why helm still upgrades other services/pods by restarting it. Should helm does nothing if there is any error during the upgrade? I excepted helm to treat the whole set of services as a package to either upgrade all or none, but seems my expectation might be wrong.

And if we ever end up in such situation, then what we should to get out the situation? like how to upgrade "my-service" to have new type?

And if I use --dry-run option, helm does not show any errors.

Is this consider a bug or expected, i.e. upgrade throws some error but some service still gets upgraded.

<!-- If you need help or think you have found a bug, please help us with your issue by entering the following information (otherwise you can delete this text): -->

Output of helm version:

Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.27", GitCommit:"145f9e21a4515947d6fb10819e5a336aff1b6959", GitTreeState:"clean", BuildDate:"2020-02-21T18:01:40Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): GKE and Minkube

helm/helm

Answer questions forainychen

Hi, here are the reproduce steps Having two services yaml file as below.

nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

prometheus.yaml

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: prometheus
spec:
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - image: prom/prometheus
        name: prometheus
        ports:
        - containerPort: 9090
        imagePullPolicy: Always
      hostname: prometheus
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  selector:
    app: prometheus
  clusterIP: None
  ports:
  - name: headless
    port: 9090
    targetPort: 0

Then put there two files in helm1/templates/ then install. It shows prometheus service uses clusterIP and nginx version is 1.14.2

# helm upgrade --install test helm1
Release "test" does not exist. Installing it now.
NAME: test
LAST DEPLOYED: Tue Apr 21 20:42:55 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP    35d
prometheus   ClusterIP   None         <none>        9090/TCP   7s

# kubectl describe deployment nginx |grep Image
    Image:        nginx:1.14.2

Now update the section for nginx.yaml to new version 1.16

        image: nginx:1.16

and prometheus.yaml by changing it to LoadBalancer.

spec:
  selector:
    app: prometheus
  ports:
  - name: "9090"
    port: 9090
    protocol: TCP
    targetPort: 9090
  type: LoadBalancer

Now put them as helm2 and do the upgrade. Then you can see the upgrade throw some errors, but the nginx service goes through, by upgrade to a new version, but the prometheus is not upgraded as it is still using Cluster IP.

# helm upgrade --install test helm2
Error: UPGRADE FAILED: cannot patch "prometheus" with kind Service: Service "prometheus" is invalid: spec.clusterIP: Invalid value: "": field is immutable

# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP    35d
prometheus   ClusterIP   None         <none>        9090/TCP   5m34s

# kubectl describe deployment nginx |grep Image
    Image:        nginx:1.16

helm list shows

# helm list
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS	CHART                                    	APP VERSION
test	default  	2       	2020-04-21 20:48:20.133644429 -0700 PDT	failed	

helm history

# helm history test
REVISION	UPDATED                 	STATUS  	CHART       APP VERSION	DESCRIPTION                                                                                                                                               
1       	Tue Apr 21 20:42:55 2020	deployed	helm-helm	1.0.0.6    	Install complete                                                                                                                                          
2       	Tue Apr 21 20:48:20 2020	failed  	helm-helm	1.0.0.6    	Upgrade "test" failed: cannot patch "prometheus" with kind Service: Service "prometheus" is invalid: spec.clusterIP: Invalid value: "": field is immutable

useful!

Related questions

Error: validation: chart.metadata is required when using --repo - helm hot 4
"Error: Transport is closing" message when attempting to install hot 2
Error: open .helm\repository\cache\local-index.yaml hot 2
"helm dep build" fails if requirements.yaml contains local dependencies and remote one hot 2
Helm upgrade fails the release after adding a new resource hot 2
Error: could not find a ready tiller pod hot 2
Error: apiVersion 'v2' is not valid. The value must be "v1" hot 2
Unable to start Tiller pod, bind address already in use hot 2
Upgrading releases with immutable resources (e.g. persitentVolume) hot 2
helm 3: Not able to push chart to docker hub hot 1
`helm upgrade --recreate-pods` flag should do a soft restart - helm hot 1
Helm3: No 'init', doesn't use existing ~/.helm hot 1
No kind Job is registered for version batch/v1 hot 1
`Error: rendered manifests contain a resource that already exists` but nothing shows up on `helm list --all` - helm hot 1
error calling eq: invalid type for comparison hot 1
Github User Rank List