profile
viewpoint
Andrew Sy Kim andrewsykim @VMware Open Source at VMware | Working on Kubernetes

andrewsykim/appium 0

Automation for iOS and Android Apps.

andrewsykim/autoscaler 0

Autoscaling components for Kubernetes

andrewsykim/awsecr-creds 0

Allow for AWS ECR credentials to be refreshed inside your Kubernetes cluster via ImagePullSecrets

andrewsykim/cilium 0

HTTP, gRPC, and Kafka Aware Security and Networking for Containers with BPF and XDP

andrewsykim/cloud-provider 0

cloud-provider defines the shared interfaces which Kubernetes cloud providers implement. These interfaces allow various controllers to integrate with any cloud provider in a pluggable fashion.

andrewsykim/cluster-api 0

Home for the Cluster Management API work, a subproject of sig-cluster-lifecycle

andrewsykim/cluster-api-bootstrap-provider-kubeadm 0

Cluster API Bootstrap Provider Kubeadm: giving your cluster API clusters a kubeadm kick

pull request commentkubernetes/kubernetes

cloud/node-controller use InstanceMetadataByProviderID

/retest

gongguan

comment created time in 9 hours

pull request commentkubernetes/kubernetes

cloud/node-controller use InstanceMetadataByProviderID

/retest

gongguan

comment created time in 9 hours

pull request commentkubernetes/kubernetes

Disable sysctlRouteLocalnet to address CVE-2020-8558

(some failures expected)

lbernail

comment created time in 9 hours

pull request commentkubernetes/kubernetes

Disable sysctlRouteLocalnet to address CVE-2020-8558

/test pull-kubernetes-e2e-gci-gce-ipvs

lbernail

comment created time in 9 hours

Pull request review commentkubernetes/kubernetes

Disable sysctlRouteLocalnet to address CVE-2020-8558

 func NewProxier(ipt utiliptables.Interface, 	kernelHandler KernelHandler, ) (*Proxier, error) { 	// Set the route_localnet sysctl we need for

nit: update comment with context?

lbernail

comment created time in 9 hours

issue commentkubernetes/kubernetes

One Node loose private and external ip address

@mohideen what's your CNI?

shahbour

comment created time in 10 hours

issue commentkubernetes/kubernetes

net.ipv4.conf.all.route_localnet=1 opens security issue

+1 from me as long as we test it and ensure it doesn't break any existing behavior.

There were previous feature requests to actually allow localhost:nodeport but given the security implications it sounds like we should avoid it.

uablrek

comment created time in 10 hours

Pull request review commentkubernetes/cloud-provider-vsphere

Initial implementation for YAML based config

+/*+ Copyright 2020 The Kubernetes Authors.++ Licensed under the Apache License, Version 2.0 (the "License");+ you may not use this file except in compliance with the License.+ You may obtain a copy of the License at++     http://www.apache.org/licenses/LICENSE-2.0++ Unless required by applicable law or agreed to in writing, software+ distributed under the License is distributed on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ See the License for the specific language governing permissions and+ limitations under the License.+*/++package config++/*+	TODO:+	When the INI based cloud-config is deprecated, this file should be renamed+	from types_yaml.go to types.go and the structs within this file should be named:++	LBConfigYAML -> LBConfig+	LoadBalancerClassConfigYAML -> LoadBalancerClassConfig+	LoadBalancerClassConfigYAML -> LoadBalancerClassConfig+	NsxtConfigYAML -> NsxtConfig+*/++// LBConfigYAML  is used to read and store information from the cloud configuration file+type LBConfigYAML struct {+	LoadBalancer      LoadBalancerConfigYAML                  `yaml:"loadbalancer"`+	LoadBalancerClass map[string]*LoadBalancerClassConfigYAML `yaml:"loadbalancerclass"`+	NSXT              NsxtConfigYAML                          `yaml:"nsxt"`+}++// LoadBalancerConfigYAML contains the configuration for the load balancer itself+type LoadBalancerConfigYAML struct {+	Size             string            `yaml:"size"`+	LBServiceID      string            `yaml:"lb-service-id"`+	Tier1GatewayPath string            `yaml:"tier1-gateway-path"`+	AdditionalTags   map[string]string `yaml:"tags"`++	// this struct use to inherit from LoadBalancerClassConfigYAML, but the YAML parser+	// wasnt able to indirectly parse inherited fields+	IPPoolName        string `yaml:"ip-pool-name"`

ip-pool-name makes sense for INI but I think best practice for yaml is to use camel case?

dvonthenen

comment created time in 11 hours

Pull request review commentkubernetes/cloud-provider-vsphere

Initial implementation for YAML based config

+/*+Copyright 2018 The Kubernetes Authors.

2020

dvonthenen

comment created time in 11 hours

Pull request review commentkubernetes/cloud-provider-vsphere

Initial implementation for YAML based config

+/*+Copyright 2018 The Kubernetes Authors.

2020

dvonthenen

comment created time in 11 hours

Pull request review commentkubernetes/cloud-provider-vsphere

Initial implementation for YAML based config

 host = nsxt-server 	assertEquals("LoadBalancer.tcpAppProfileName", config.LoadBalancer.TCPAppProfileName, "default-tcp-lb-app-profile") 	assertEquals("LoadBalancer.udpAppProfileName", config.LoadBalancer.UDPAppProfileName, "default-udp-lb-app-profile") 	assertEquals("LoadBalancer.size", config.LoadBalancer.Size, "MEDIUM")-	if len(config.LoadBalancerClasses) != 2 {-		t.Errorf("expected two LoadBalancerClass subsections, but got %d", len(config.LoadBalancerClasses))+	if len(config.LoadBalancerClass) != 2 {+		t.Errorf("expected two LoadBalancerClass subsections, but got %d", len(config.LoadBalancerClass)) 	}-	assertEquals("LoadBalancerClass.public.ipPoolName", config.LoadBalancerClasses["public"].IPPoolName, "poolPublic")-	assertEquals("LoadBalancerClass.private.tcpAppProfileName", config.LoadBalancerClasses["private"].TCPAppProfileName, "tcp2")-	assertEquals("LoadBalancerClass.private.udpAppProfileName", config.LoadBalancerClasses["private"].UDPAppProfileName, "udp2")+	assertEquals("LoadBalancerClass.public.ipPoolName", config.LoadBalancerClass["public"].IPPoolName, "poolPublic")+	assertEquals("LoadBalancerClass.private.tcpAppProfileName", config.LoadBalancerClass["private"].TCPAppProfileName, "tcp2")+	assertEquals("LoadBalancerClass.private.udpAppProfileName", config.LoadBalancerClass["private"].UDPAppProfileName, "udp2") 	if len(config.LoadBalancer.AdditionalTags) != 2 || config.LoadBalancer.AdditionalTags["tag1"] != "value1" || config.LoadBalancer.AdditionalTags["tag2"] != "value 2" { 		t.Errorf("unexpected additionalTags %v", config.LoadBalancer.AdditionalTags) 	}-	assertEquals("NSX-T.user", config.NSXT.User, "admin")-	assertEquals("NSX-T.password", config.NSXT.Password, "secret")-	assertEquals("NSX-T.host", config.NSXT.Host, "nsxt-server")+	assertEquals("NSXT.user", config.NSXT.User, "admin")+	assertEquals("NSXT.password", config.NSXT.Password, "secret")+	assertEquals("NSXT.host", config.NSXT.Host, "nsxt-server") } -func TestReadConfigOnVMC(t *testing.T) {+func TestReadINIConfigOnVMC(t *testing.T) { 	contents := ` [LoadBalancer]-ipPoolID = 123-456+ip-pool-id = 123-456

For posterity: this is not a breaking change since we haven't release a version supporting this yet.

cc @mandelsoft @MartinWeindel

dvonthenen

comment created time in 11 hours

Pull request review commentkubernetes/cloud-provider-vsphere

Initial implementation for YAML based config

+/*+Copyright 2019 New The Kubernetes Authors.

2020 cause new file?

dvonthenen

comment created time in 11 hours

Pull request review commentkubernetes/cloud-provider-vsphere

Initial implementation for YAML based config

 func (vs *VSphere) HasClusterID() bool { }  // Initializes vSphere from vSphere CloudProvider Configuration-func buildVSphereFromConfig(cfg *CPIConfig) (*VSphere, error) {+func buildVSphereFromConfig(cfg *ccfg.CPIConfig, lbcfg *lcfg.LBConfig) (*VSphere, error) { 	nm := newNodeManager(cfg, nil)-	lb, err := loadbalancer.NewLBProvider(&cfg.LBConfig)++	lb, err := loadbalancer.NewLBProvider(lbcfg) 	if err != nil { 		return nil, err 	}-	if _, ok := os.LookupEnv("ENABLE_ALPHA_NSXT_LB"); !ok {-		if lb != nil {-			klog.Infof("To enable NSX-T load balancer support you need to set the env variable ENABLE_ALPHA_NSXT_LB")-			lb = nil-		}-	} else {+	if _, ok := os.LookupEnv("ENABLE_ALPHA_NSXT_LB"); ok { 		if lb == nil {-			return nil, fmt.Errorf("To enable NSX-T load balancer support you need to configure section LoadBalancer")+			klog.Warning("To enable NSX-T load balancer support you need to configure section LoadBalancer")+		} else {+			klog.Infof("NSX-T load balancer support enabled. This feature is alpha, use in production at your own risk.")+			// redirect vapi logging from the NSX-T GO SDK to klog+			log.SetLogger(NewKlogBridge()) 		}-	}-	if lb == nil {-		klog.Infof("NSX-T load balancer support disabled") 	} else {-		klog.Infof("NSX-T load balancer support enabled. This feature is alpha, use in production at your own risk.")-		// redirect vapi logging from the NSX-T GO SDK to klog-		log.SetLogger(NewKlogBridge())+		// explicitly nil the LB interface if ENABLE_ALPHA_NSXT_LB is not set even if the LBConfig is valid

Previous this was

if lb != nil {
    klog.Infof("To enable NSX-T load balancer support you need to set the env variable ENABLE_ALPHA_NSXT_LB")
    lb = nil
}

Do we want to keep that log message?

dvonthenen

comment created time in 11 hours

Pull request review commentkubernetes/cloud-provider-vsphere

Initial implementation for YAML based config

 limitations under the License. package vsphere  import (-	"fmt" 	"io"+	"io/ioutil" 	"os" 	"runtime"  	v1 "k8s.io/api/core/v1"-	cloudprovider "k8s.io/cloud-provider" 	"k8s.io/klog" -	"github.com/vmware/vsphere-automation-sdk-go/runtime/log"+	cloudprovider "k8s.io/cloud-provider" +	"github.com/vmware/vsphere-automation-sdk-go/runtime/log"

non-blocking nit: we should group github.com imports separately.

dvonthenen

comment created time in 11 hours

Pull request review commentkubernetes/cloud-provider-vsphere

Initial implementation for YAML based config

 Steps that will be covered in order to setup zones for the vSphere CPI, vSphere  > ***Note:*** The CSI and CPI drivers have their own vsphere.conf files. The following modifications need to be made in both configurations. -The zones implementation depends on 2 sets of vSphere tags to be used on objects, such as datacenters or clusters. The first is a `region` tag and the second is a `zone` tag. vSphere tags are very simply put key/value pairs that can be assigned to objects and instead of using fixed keys to denote a `region` or a `zone`, we give the end-user the ability to come up with their own keys for a `region` and `zone` in the form of vSphere Tag Catagory. It just allows for a level of indirection in case you already have regions and zones setup in your configuration. Once a key/label or vSphere Tag Category is selected for each, create a `[Labels]` section in the `vsphere.conf` then assign tag names for both `region` and `zone`.+The zones implementation depends on 2 sets of vSphere tags to be used on objects, such as datacenters or clusters. The first is a `region` tag and the second is a `zone` tag. vSphere tags are very simply put key/value pairs that can be assigned to objects and instead of using fixed keys to denote a `region` or a `zone`, we give the end-user the ability to come up with their own keys for a `region` and `zone` in the form of vSphere Tag Catagory. It just allows for a level of indirection in case you already have regions and zones setup in your configuration. Once a key/label or vSphere Tag Category is selected for each, create a `labels:` section in the `vsphere.conf` then assign tag names for both `region` and `zone`.  In the example `vsphere.conf` below, `k8s-region` and `k8s-zone` was selected:  ```bash-[Global]-# properties in this section will be used for all specified vCenters unless overridden in VirtualCenter section.--user = "vCenter username for cloud provider"-password = "password"-port = "443" #Optional-insecure-flag = "1" #set to 1 if the vCenter uses a self-signed cert-datacenters = "list of datacenters where Kubernetes node VMs are present"--[VirtualCenter "1.2.3.4"]-# Override specific properties for this Virtual Center.-        user = "vCenter username for cloud provider"-        password = "password"-        # port, insecure-flag, datacenters will be used from Global section.--[VirtualCenter "10.0.0.1"]-# Override specific properties for this Virtual Center.-        port = "448"-        insecure-flag = "0"-        # user, password, datacenters will be used from Global section.--[Labels]-region = k8s-region-zone = k8s-zone+# Global properties in this section will be used for all specified vCenters unless overriden in VirtualCenter section.+global:+  user: YourVCenterUser+  password: YourVCenterPass+  port: "443"

Guessing this has to be a string for compatiblity?

dvonthenen

comment created time in 11 hours

Pull request review commentkubernetes/cloud-provider-vsphere

Update docs for VMTools exclude-nics filtering

 Server Version: 18.06.0-ce Cgroup Driver: systemd ``` +### VMTools NIC/Device Filtering++A number of [CNI](https://github.com/containernetworking/cni) implementations (such Calico, Antrea, and etc) introduce networking artifacts that interfere with the normal operation of vSphere's internal reporting for network/device interfaces. To address this issue, an `exclude-nics` filter for VMTools needs to be applied in order to prevent these artifacts from getting reported to vSphere and causing problems with network/device associations to vNICs on virtual machines.++The recommended `exclude-nics` filter is as follows for `/etc/vmware-tools/tools.conf`:++```bash+[guestinfo]+primary-nics=eth0+exclude-nics=antrea-*,ovs-system,br*,flannel*,veth*,docker*,virbr*,vxlan_sys_*,genev_sys_*,gre_sys_*,stt_sys_*,????????-??????

I think the ????????-?????? filter is gonna be confusing to a lot of users, should we add more words explaining this?

dvonthenen

comment created time in 12 hours

pull request commentkubernetes/kubernetes

Automated cherry pick of #92584: kube-proxy ipvs masquerade hairpin traffic

For the session affinity tests I think you also need to backport https://github.com/kubernetes/kubernetes/pull/89854 :confused:

aojea

comment created time in 12 hours

pull request commentkubernetes/kubernetes

Automated cherry pick of #92584: kube-proxy ipvs masquerade hairpin traffic

@aojea seeing actual test failures here

aojea

comment created time in 12 hours

pull request commentkubernetes/kubernetes

cloud/node-controller use InstanceMetadataByProviderID

/milestone v1.19

(should have had the milestone from the start)

gongguan

comment created time in 13 hours

Pull request review commentkubernetes/kubernetes

only log cloud provider deprecation warning for in-tree components

 func CreateNodeDialer(s completedServerRunOptions) (tunneler.Tunneler, *http.Tra 	if len(s.SSHUser) > 0 { 		// Get ssh key distribution func, if supported 		var installSSHKey tunneler.InstallSSHKey++		cloudprovider.DeprecationWarningForProvider(s.CloudProvider.CloudProvider)

Gonna fix this in a follow-up PR

andrewsykim

comment created time in a day

Pull request review commentkubernetes/kubernetes

Move cmd/controller-manager to k8s.io/controller-manager

 import (  	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" 	"k8s.io/apimachinery/pkg/runtime"+	serviceconfigv1alpha1 "k8s.io/controller-manager/config/v1alpha1"

+1

cici37

comment created time in a day

Pull request review commentkubernetes/enhancements

Different protocols in the same Service definition with type=LoadBalancer

+---+title: Different protocols in the same Service definition with type=LoadBalancer+authors:+  - "@janosi"+owning-sig: sig-network+participating-sigs:+  - sig-cloud-provider+reviewers:+  - "@thockin"+  - "@dcbw"+  - "@andrewsykim"+approvers:+  - "@thockin"+editor: TBD+creation-date: 2020-01-03+last-updated: 2020-03-29+status: provisional+see-also:+replaces:+superseded-by:+---++# different protocols in the same service definition with type=loadbalancer++## Table of Contents++<!-- toc -->+- [Release Signoff Checklist](#release-signoff-checklist)+- [Summary](#summary)+- [Motivation](#motivation)+  - [Goals](#goals)+  - [Non-Goals](#non-goals)+- [Proposal](#proposal)+  - [User Stories](#user-stories)+    - [Story 1](#story-1)+    - [Story 2](#story-2)+  - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints)+    - [Alibaba](#alibaba)+    - [AWS](#aws)+    - [Azure](#azure)+    - [DigitalOcean](#digitalocean)+    - [GCE](#gce)+    - [IBM Cloud](#ibm-cloud)+    - [OpenStack](#openstack)+    - [Oracle Cloud](#oracle-cloud)+    - [Tencent Cloud](#tencent-cloud)+  - [Risks and Mitigations](#risks-and-mitigations)+    - [Billing perspective](#billing-perspective)+    - [API change and upgrade/downgrade situations](#api-change-and-upgradedowngrade-situations)+- [Design Details](#design-details)+  - [Option Control Alternatives](#option-control-alternatives)+    - [Annotation in the Service definition](#annotation-in-the-service-definition)+    - [Merging Services in CPI](#merging-services-in-cpi)+    - [Proposed solution](#proposed-solution)+  - [Test Plan](#test-plan)+  - [Graduation Criteria](#graduation-criteria)+  - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)+    - [Downgrade Strategy](#downgrade-strategy)+  - [Version Skew Strategy](#version-skew-strategy)+- [Implementation History](#implementation-history)+- [Drawbacks [optional]](#drawbacks-optional)+- [Alternatives [optional]](#alternatives-optional)+- [Infrastructure Needed [optional]](#infrastructure-needed-optional)+<!-- /toc -->++## Release Signoff Checklist++- [ ] kubernetes/enhancements issue in release milestone, which links to KEP (this should be a link to the KEP location in kubernetes/enhancements, not the initial KEP PR) https://github.com/kubernetes/enhancements/issues/1435+- [ ] KEP approvers have set the KEP status to `implementable`+- [ ] Design details are appropriately documented+- [ ] Test plan is in place, giving consideration to SIG Architecture and SIG Testing input+- [ ] Graduation criteria is in place+- [ ] "Implementation History" section is up-to-date for milestone+- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]+- [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes++## Summary++This feature enables the creation of a LoadBalancer Service that has different port definitions with different protocols. ++## Motivation++The ultimate goal of this feature is to support users that want to expose their applications via a single IP address but different L4 protocols with a cloud provided load-balancer. +The following issue and PR shows considerable interest from users that would benefit from this feature:+https://github.com/kubernetes/kubernetes/issues/23880+https://github.com/kubernetes/kubernetes/pull/75831++The current restriction that rejects a Service creation request if it has two different protocol definitions was introduced because some cloud implementations may charge their load balancer users on a "per protocol" basis. That is, the current logic is to prevent negative surprises with the load balancer bills. The current implementation enforces a more explicit statement or consent form the end user for the usage of two different protocols on the same load balancer. For example GCE expects that the user creates two load balancer Service definitions for the two different protocols. ++But such workarounds or solutions do not exist in all cloud implementations. According to the feedback from the end users it would be more beneficial to remove this restriction from the Kubernetes code. This KEP is to investigate how the removal of that restriction would affect the billing of load balancer resources in the different clouds, i.e. whether it is safe or not to allow the usage of different protocol values in the same Service definition.++### Goals++The goals of this KEP are:++- To analyze the impact of this feature with regard to current implementations of cloud-provider load-balancers+- define how the activation of this feature could be made configurable if certain cloud-provider load-balancer implementations do not want to provide this feature++### Non-Goals+++## Proposal++The first thing proposed here is to relax the API validation in Kubernetes that currently rejects Service definitions with different protocols if their type is LoadBalancer. Kubernetes would not reject Service definitions like this from that point:+```yaml+apiVersion: v1+kind: Service+metadata:+  name: mixed-protocol+spec:+  type: LoadBalancer+  ports:+    - name: dns-udp+      port: 53+      protocol: UDP+    - name: dns-tcp+      port: 53+      protocol: TCP+  selector:+    app: my-dns-server+  ```++Once that limit is removed those Service definitions will reach the Cloud Provider LB controller implementations. The logic of the particular Cloud Provider LB controller and of course the actual capabilities and architecture of the backing Cloud Load Balancer services determine how the actual exposure of the application really manifests. For this reason it is important to understand the capabilities of those backing services and to design this feature accordingly.++### User Stories++#### Story 1++As a Kubernetes cluster user I want to have the capability of exposing a DNS server for TCP and UDP based requests on the same Load Balancer IP address. See [RFC7766](https://tools.ietf.org/html/rfc7766).  In order to achieve this I want to define different `ports` with mixed protocols in my Service definition of `type: LoadBalancer`++#### Story 2++As as Kubernetes cluster user I want to have the capability of exposing my SIP Server for TCP and UDP based requests on the same Load Balacer IP address and port. This requirement comes from [RFC3261 Section 18.2.1](https://tools.ietf.org/html/rfc3261#section-18.2.1)++### Implementation Details/Notes/Constraints++#### Alibaba++The Alibaba Cloud Provider Interface Implementation (CPI) supports TCP, UDP and HTTPS in Service definitions and can configure the SLB listeners with the protocols defined in the Service.++The Alibaba SLB supports TCP, UDP and HTTPS listeners. A listener must be configured for each protocol, and then those listeners can be assigned to the same SLB instance. ++The number of listeners does not affect SLB pricing.+https://www.alibabacloud.com/help/doc-detail/74809.htm++A user can ask for an internal TCP/UDP Load Balancer via a K8s Service definition that also has the annotation `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"`. Internal SLBs are free.++Summary: Alibaba CPI and SLB seem to support mixed protocol Services, and the pricing is not based on the number of protocols per Service.++#### AWS++The AWS CPI does not support mixed protocols in the Service definition since it only allows TCP for load balancers. The AWS CPI looks for annotations on the Service to determine whether TCP, TLS or HTTP(S) listener should be created in the AWS ELB for a configured Service port.++AWS Classic LB supports TCP,TLS, and HTTP(S) protocols behind the same IP address. ++AWS Network LB supports TCP/TLS and UDP protocols behind the same IP address. As we can see, UDP cannot be utilized currently, due to the limitation in the AWS CPI.++The usage of TCP+HTTP or UDP+HTTP on the same LB instace behind the same IP address is not possible in AWS.++From a pricing perspective the AWS NLB and the CLB have the following models:+https://aws.amazon.com/elasticloadbalancing/pricing/+Both are primarily usage based rather than charging based on the number of protocol, however NLBs have separate pricing unit quotas for TCP, UDP and TLS.++A user can ask for an internal Load Balancer via a K8s Service definition that also has the annotation `service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0`. So far the author could not find any difference in the usage and pricing of those when compared to the external LBs - except the pre-requisite of a private subnet on which the LB can be deployed.++Summary: AWS CPI is the current bottleneck with its TCP-only limitation. As long as it is there the implementation of this feature will have no effect on the AWS bills.++#### Azure++Azure CPI LB documentation: https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/docs/services/README.md++The Azure CPI already supports the usage of both UDP and TCP protocols in the same Service definition. It is achieved with the CPI specific annotation `service.beta.kubernetes.io/azure-load-balancer-mixed-protocols`. If this key has value of `true` in the Service definition, the Azure CPI adds the other protocol value (UDP or TCP) to its internal Service representation. Consequently, it also manages twice the amount of load balancer rules for the specific frontend. ++Only TCP and UDP are supported in the current mixed protocol configuration.++The Azure Load Balancer supports only TCP and UDP as protocols. HTTP support would require the usage of the Azure Application Gateway. I.e. HTTP and L4 protocols cannot be used on the same LB instance/IP address++Basic Azure Load Balancers are free.++Pricing of the Standard Azure Load Balancer is based on load-balancing rules and outbound rules.  There is a flat price up to 5 rules, on top of which every new forwarding rule has additional cost. +https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-overview#pricing++A user can ask for an internal Azure Load Balancer via a K8s Service definition that also has the annotation `service.beta.kubernetes.io/azure-load-balancer-internal: true`. There is not any limitation on the usage of mixed service protocols with internal LBs in the Azure CPI.++Summary: Azure already has mixed protocol support for TCP and UDP, and it already affects the bills of the users. The implementation of this feature may require some work in Azure CPI.++#### DigitalOcean++The DigitalOcean CPI does not support mixed protocols in the Service definition, it accepts TCP only for load balancer Services. It is possible to ask for HTTP(S) and HTTP2 ports with Service annotations.++Summary: The DO CPI is the current bottleneck with its TCP-only limitation. As long as it is there the implementation of this feature will have no effect on the DO bills.++#### GCE++The GCE CPI supports only TCP and UDP protocols in Services.++GCE/GKE creates Network Load Balancers based on the K8s Services with type=LoadBalancer. In GCE there are "forwarding rules" that define how the incoming traffic shall be forwarded to the compute instances. A single forwarding rule can support either TCP or UDP but not both. In order to have both TCP and UDP forwarding rules we have to create separate forwarding rule instances for those. Two or more forwarding rules can share the same external IP if+- the network load balancer type is External+- the external IP address is not ephemeral but static++There is a workaround in GCE, please see this comment from the original issue: https://github.com/kubernetes/kubernetes/issues/23880#issuecomment-269054735 If the external IP is static the user can create two Service instances, one for UDP and another for TCP and the user has to specify that static external IP address in those Service definitions in `loadBalancerIP`.++HTTP protocol support: there is a different LB type in GCE for HTTP traffic: HTTP(S) LB. I.e. just like in the case of AWS it is not possible to have e.g. TCP+HTTP or UDP+HTTP behind the same LB/IP address.++Forwarding rule pricing is per rule: there is a flat price up to 5 forwarding rule instances, on top of which every new forwarding rule has additional cost. ++[GCP forwarding_rules_charges](https://cloud.google.com/compute/network-pricing#forwarding_rules_charges) suggest that the same K8s Service definition would result in the creation of 2 forwarding rules in GCP. This has the same fixed price up to 5 forwarding rule instances, and each additional rule results in extra cost.++A user can ask for an internal TCP/UDP Load Balancer via a K8s Service definition that also has the annotation `cloud.google.com/load-balancer-type: "Internal"`. Forwarding rules are also part of the GCE Internal TCP/UDP Load Balancer architecture, but in case of Internal TCP/UDP Load Balancer it is not supported to define different forwarding rules with different protocols for the same IP address. That is, for Services with type=LoadBalancer and with annotation `cloud.google.com/load-balancer-type: "Internal"` this feature would not be supported.++Summary: The implementation of this feature can affect the bills of GCP users. However the following perspectives are also observed:+- if a user wants to have UDP and TCP ports behind the same NLB two Services with must be defined, one for TCP and one for UDP. As the pricing is based on the number of forwarding rules this setup also means the same pricing as with the single Service instance.+- if a user is happy with 2 NLB (Service) instances for TCP and UDP still the user has two more forwarding rules to be billed - i.e. it has the same effect on pricing as if those TCP and UDP endpoints were behind the same NLB instance++That is, if a user wants to use different protocols on the same LB that can be achieved with 2 Service definitions with the current GCP services now. It is not the number of LBs or Service definitions that matters. This phenomenon is already there with the current practice, and the enabling of mixed protocols will not change it to the worse.++#### IBM Cloud++The IBM Cloud creates VPC Load Balancer in a VPC based cluster and NLB in a classic cloud based cluster. ++The IBM Cloud CPI implementation for the classic cluster supports TCP and UDP protocol values in K8s Services, and it supports different protocol values in a Service. ++The IBM Cloud CPI implementation for the VPC clusters supports only TCP. ++The VPC Load Balancer supports TCP and HTTP, and it is possible to create TCP and HTTP listeners for the same LB instance. UDP is not supported. ++The VPC LB pricing is time and data amount based, i.e. the number of protocols on the same LB instance does not affect it.++The NLB supports TCP and UDP on the same NLB instance. The usage of NLB does not have pricing effects, it is part of the IKS basic package.++Summary: once this feature is implemented in the K8s API Server the IBM Cloud VPC LB can still use only TCP ports from a Service definition. NLB can use TCP and UDP ports from a single Service definition.++#### OpenStack++The OpenStack CPI supports TCP, UDP, HTTP(S) in Service definitions and can configure the Octavia listeners with the protocols defined in the Service.+OpenStack Octavia supports TCP, UDP and HTTP(S) on listeners, an own listener must be configured for each protocol, and different listeners can be used on the same LB instance.++There was a bug in Octavia versions <5.0.0: it was not possible to use the same port number (5e.g. 53) with different protocols (e.g. TCP and UDP) on the same LB instance. It has been fixed in 5.0.0, which is available since the "T" release of OpenStack.++Summary: the OpenStack based clouds that use Octavia v2 as their LBaaS seems to support this feature once implemented. It is true that the "T" release is the newest one, so upgrade may take a while. On the other hand OpenStack documentation mentions, that a newer Octavia version can be used with previous releases of other OpenStack projects, i.e. it can be the case that the upgrade effort is on Octavia side in an OpenStack cloud. +Pricing is up to the pricing model of the OpenStack providers.++#### Oracle Cloud++The Oracle CPI does not support UDP in the K8s Service definition. It supports only TCP and HTTP. It supports mixed TCP and HTTP ports in the same Service definition.++The Oracle Cloud Load Balancer supports TCP, HTTP(S) protocols. ++The pricing is based on time and capacity: https://www.oracle.com/cloud/networking/load-balancing-pricing.html I.e. the amount of protocols on a sinlge LB instance does not affect the pricing.++Summary: Oracle CPI and LB seem to support mixed protocol Services, and the pricing is not based on the number of protocols per Service.++#### Tencent Cloud++The Tencent Cloud CPI supports TCP, UDP and HTTP in Service definitions. It maps "HTTP" protocol value in a Service definition to "TCP" before creating a listener on the CLB.+The Tencent Cloud CPI can manage both of their LB solutions: "Classic CLB" and "Cloud Load Balancer" (previously known as "Application Load Balancer"). +The Tencent Cloud CLB supports TCP, UDP and HTTP(S) listeners, an own listener must be configured for each protocol. +The number of listeners does not affect CLB pricing. CLB pricing is time (day) based and not tied to the number of listeners.+https://intl.cloud.tencent.com/document/product/214/8848++A user can ask for an internal Load Balancer via a K8s Service definition that  has the annotation `service.kubernetes.io/qcloud-loadbalancer-internal-subnetid: subnet-xxxxxxxx`. Internal CLBs are free.++Summary: Tencent Cloud CPI and LBs seem to support mixed protocol Services, and the pricing is not based on the number of protocols per Service.+++### Risks and Mitigations++#### Billing perspective++The goal of the current restriction on the K8s API was to prevent an unexpected extra charging for Load Balancers that were created based on Services with mixed protocol definitions.+If the current limitation is removed without any option control we introduce the same risk. Let's see which clouds are exposed:+- Alibaba: the pricing here is not protocol or forwarding rule or listener based. No risk.+- AWS: there is no immediate impact on the pricing side as the AWS CPI limits the scope of protocols to TCP only.+- Azure: Azure pricing is indeed based on load-balancing rules, but this cloud already supports mixed protocols via annotations. There is another risk for Azure, though: if the current restriction is removed from the K8s API, the Azure CPI must be prepared to handle Services with mixed protocols.+- GCE: here the risk is valid once the user exceeds the threshold of 5 forwarding rules.+- IBM Cloud: no risk+- OpenStack: here the risk is that there is almost no chance to analyze all the public OpenStack cloud providers with regard to their pricing policies+- Oracle: no risk+- Tencent Cloud: no risk++#### API change and upgrade/downgrade situations++We relax here a validation rule, which is considered as an API-breaking act by the [K8s API change guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md) Even if the change is implemented behind a feature flag the following actions can cause problems:+- the user creates a Service with mixed protocols and then+  - turns off the feature flag; or+  - executes K8s version rollback to the N-1 version where this feature is not available at all++When investigating the possible issues with such a change we must consider [the supported version skew among components](https://kubernetes.io/docs/setup/release/version-skew-policy/), which are the K8s API server and the cloud controller managers (CPI implementations) in our case.++First of all, feature gate based (a.k.a conditional) field validation must be implemented as defined in the [API change guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#alpha-field-in-existing-api-version) One can see a good example for a case when an existing field got a new optional value [here](https://github.com/kubernetes/kubernetes/issues/72651). This practice ensures that upgrade and rollback between _future releases_ is safe. Also it enables the further management of existing API objects that were created before turning off the feature flag. Though it does not save us when a K8s API server version rollback is executed to a release in which this feature is not implemented at all.++Our feature does not introduce new values or new fields. It enables the usage of an existing value in existing fields, but with a different logic. I.e. if someone creates a Service with mixed protocol setup and then rollbacks the API server to a version that does not implement this feature the clients will still get the Service with mixed protocols when they read that via the rollback'd API. If the client (CPI implementation) has been rollbacked, too, then the client may receive such a Service setup that it does not support.++- Alibaba: no risk. The current CPI and LB already supports the mixed protocols in the same Service definition. If this feature is enabled in an API server and then the API server rollback is executed the CPI can still handle the Services with mixed protocol sets.+- AWS: no risk. The current CPI accepts Services with TCP protocol only, i.e. after a K8s upgrade a user still cannot use this feature. Consequently, a rollback in the K8s version does not introduce any issues.+- Azure: no risk. The current CPI and LB already supports the mixed protocols in the same Service definition. The situation is the same as with the Alibaba CPI.+- GCE: currently the GCE CPI assumes that a Service definition contains a single protocol value, as it assumes that the Service Controller already rejected Services with mixed protocols. While the Service Controller really did so a while ago, it does not do this anymore. It means a risk.+- DigitalOcean: no risk. The current CPI accepts Services with TCP protocol only, i.e. after a K8s upgrade a user still cannot use this feature. Consequently, a rollback in the K8s version does not introduce any issues.+- IBM Cloud VPC: no risk. The same situation like in the case of AWS.+- IBM Cloud Classic: no risk. The CPI and NLB already supports TCP and UDP in the same Service definition. The same situation like in the case of Alibaba.+- OpenStack: no risk. The CPI and NLB already supports TCP and UDP in the same Service definition. The same situation like in the case of Alibaba.+- Oracle: no risk. The CPI and LB already supports mixed protocols. The same situation like in the case of Alibaba.+- Tencent Cloud: no risk. The CPI and LB already supports mixed protocols. The same situation like in the case of Alibaba.+++The other risk is in the introduction of this feature without an option control mechanism, i.e. as a general change in Service handling. In that case there is the question whether this feature should be a part of the conformance test set, because it can affect the conformance of cloud providers.++A possible mitigation is to put the feature behind option control. ++## Design Details++The implementation of the basic logic is ready in this PR:+https://github.com/kubernetes/kubernetes/pull/75831++Currently a feature gate is used to control its activation status. Though if we want to keep this feature behind option control even after it reaches its GA state we should come up with a different solution, as feature gates are used to control the status of a feature as that graduates from alpha to GA, and they are not meant for option control for features with GA status.++### Option Control Alternatives++#### Annotation in the Service definition++In this alternative we would have a new annotation in the `kubernetes.io` annotation namespace, as it was planned in the original PR. Example: `kubernetes.io/mixedprotocol`. If this annotation is applied on a Service definition the Service would be accepted by the K8s API.++Pro: +- a kind of "implied conduct" from the user's side. The user explicitly defines with the usage of the annotation that the usage of multiple protocols on the same LoadBalancer is accepted+- Immediate feedback from the K8s API if the user configures a Service with mixed protocol set but without this annotation+Con:+- Additional configuration task for the user+- Must be executed for each and every Service definitions that are to define different protocols for the same LB++#### Merging Services in CPI++This one is not really a classic option control mechanism. The idea comes from the current practice implemented in MetalLB: https://metallb.universe.tf/usage/#ip-address-sharing+The same works in GCE as a workaround.++I.e. if a cloud provider wants to support this feature the CPI must have a logic to apply the Service definitions with a common key value (for example loadBalancerIP) on the same LoadBalancer instance. If the CPI does not implement this support it will work as it does currently.+Pro:+- the cloud provider can decide when to support this feature, and until that it works as currently for this kind of config+- the restriction on mixed protocols can remain in the K8s API - i.e. there is no K8s API change, and the corresponding risks are mitigated+Con:+- the users must maintain two Service instances+- Atomic update can be a problem - the key must be such a Service attribute that cannot be patched++#### Proposed solution++In the first release: + - a feature flag shall control whether new loadbalancer Services with mixed protcol configuration can be created or not+ - we must add a note to the documentation that if such Service is created then it may break things after a rollback - it depends on the cloud provider implementation+ - the update of such Services shall be possible even if the feature flag is OFF. This is to prepare for the next release when the feature flag is removed from the create path, too, and after a rollback to the first release the update of existing Service objects must be possible+ - the CPI implementations shall be prepared to deal with Services with mixed protocol configurations. It does not mean that the CPI implementations have to accept such Services. It means that the CPI implementations shall be able to process such Services. The CPI implementation can still validate the Service objects to check whether any of those has mixed protocol configuration and if the CPI cannot support such Service for any reasons the CPI shall indicate that clearly to the user, and the CPI should not create Cloud LB resources for the "accepted part" of the Service. The point is, that the system (Service - CPI implementation - cloud LB) must be in a consitent state in the end, and the user must have meaningful information about that state. For example, if the Service is not accepted by the CPI implementation this fact shall be available for the user, either as a new field in `Service.status.loadBalancer` or as a new Event (or both).

I'm exploring how much work is involved to have the CPI setup a validating webhook for Services Type=LB, so we can have provider-specific validation of port protocols.

Hopefully it's available around the same time we lift this validation but don't think we need to block on it.

janosi

comment created time in a day

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha cebffa5cfe5a5ef0e399d8353601eb6598a1fb30

tls: add periodic certificate loader This is primarily useful for key rotation or if the certificates should be switched out dynamically. Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 764eedc60e39df71a147d28f6a95e50a38c0bf97

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha db580576a4c118a991d4970df8b1cd528507df1d

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 34532920329003ec223cfe3743ea4e7fe0161bc5

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in a day

pull request commentkubernetes/kubernetes

cloud/node-controller use InstanceMetadataByProviderID

So yes, let's worry about that in a follow-up PR

gongguan

comment created time in a day

pull request commentkubernetes/kubernetes

cloud/node-controller use InstanceMetadataByProviderID

Oh I see, but that's specific to the provider implementation. We should defer to the providers to update that and the original PR should not have implemented it for all the providers.

gongguan

comment created time in a day

pull request commentkubernetes/kubernetes

Set CSIMigrationvSphere feature gates to beta

Yes happy to! I think we should first discuss this in more detail in both the VMware User Group and the vSphere provider subproject, I'll add these items to the next agenda cc @frapposelli @dvonthenen @cantbewong

Btw, don't think we need to block this PR on this, but we should definitely have something ready before releasing v1.19

divyenpatel

comment created time in a day

pull request commentkubernetes/kubernetes

cloud/node-controller use InstanceMetadataByProviderID

And one concern: InstanceMetadataByProviderID checked instance existence, do we need to remove these logic?

Looking at the code again I don't see where InstanceMetadataByProviderID checks for existence, can you show me where this is happening?

gongguan

comment created time in 2 days

pull request commentkubernetes/kubernetes

Set CSIMigrationvSphere feature gates to beta

/priority important-soon /sig cloud-provider /sig storage

divyenpatel

comment created time in 2 days

pull request commentkubernetes/kubernetes

Set CSIMigrationvSphere feature gates to beta

@andrewsykim Can you help with deprecation notice for vsphere versions < v7.0u1 .

Yes happy to! I think we should first discuss this in more detail in both the VMware User Group and the vSphere provider subproject, I'll add these items to the next agenda cc @frapposelli @dvonthenen @cantbewong

divyenpatel

comment created time in 2 days

Pull request review commentvmware/hamlet

Add Helper Functions for Reloading TLS Certificates

+// Copyright 2020 VMware, Inc. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0++package tls++import (+	"crypto/tls"+	"io/ioutil"+	"os"+	"testing"+	"time"+)++func Test_PeriodicReload(t *testing.T) {+	rootCAFile, err := ioutil.TempFile("", "rootCA.crt")+	if err != nil {+		t.Fatalf("error creating tmp root CA: %v", err)+	}+	defer os.Remove(rootCAFile.Name())+	if _, err := rootCAFile.Write([]byte(testRootCA)); err != nil {+		t.Fatalf("error writing to tmp root CA: %v", err)+	}++	serverCertFile, err := ioutil.TempFile("", "server.crt")+	if err != nil {+		t.Fatalf("error creating tmp server cert: %v", err)+	}+	defer os.Remove(serverCertFile.Name())+	if _, err := serverCertFile.Write([]byte(testServerCert)); err != nil {+		t.Fatalf("error writing to tmp server cert: %v", err)+	}++	serverKeyFile, err := ioutil.TempFile("", "server.key")+	if err != nil {+		t.Fatalf("error creating tmp server key: %v", err)+	}+	defer os.Remove(serverKeyFile.Name())+	if _, err := serverKeyFile.Write([]byte(testServerKey)); err != nil {+		t.Fatalf("error writing to tmp server key: %v", err)+	}++	certLoader, err := NewPeriodicCertLoader(serverCertFile.Name(), serverKeyFile.Name(), 1*time.Second)+	if err != nil {+		t.Fatalf("error starting periodic cert reloader: %v", err)+	}++	serverConfig := &tls.Config{}+	serverConfig.GetCertificate = func(*tls.ClientHelloInfo) (*tls.Certificate, error) {+		return certLoader.Current(), nil+	}+	clientConfig := &tls.Config{}+	clientConfig.RootCAs = prepareCertPool(rootCAFile.Name())++	// first check client -> server w/ TLS to validate initial reload worked+	testClientServerHello(t, clientConfig, serverConfig)++	// start certificate loader and nil out the current certificate, should expect an error+	go certLoader.Start()+	certLoader.current = nil+	testClientServerHelloError(t, clientConfig, serverConfig)

Updated, hope it reads better now :)

andrewsykim

comment created time in 2 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha 080f002407244c536e49da5b50f7d6357831c6f5

tls: add periodic certificate loader This is primarily useful for key rotation or if the certificates should be switched out dynamically. Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 45669547034a3a9ac6aa46eca0b321882b3b208f

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 39160d3cc8c89c9c5eb3b582e2b8d3b19c57233b

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 45591354247f81e725eb934f54b436d246cf40ff

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 2 days

Pull request review commentvmware/hamlet

Add Helper Functions for Reloading TLS Certificates

+// Copyright 2020 VMware, Inc. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0++package tls++import (+	"crypto/tls"+	"io/ioutil"+	"os"+	"testing"+	"time"+)++func Test_PeriodicReload(t *testing.T) {+	rootCAFile, err := ioutil.TempFile("", "rootCA.crt")+	if err != nil {+		t.Fatalf("error creating tmp root CA: %v", err)+	}+	defer os.Remove(rootCAFile.Name())+	if _, err := rootCAFile.Write([]byte(testRootCA)); err != nil {+		t.Fatalf("error writing to tmp root CA: %v", err)+	}++	serverCertFile, err := ioutil.TempFile("", "server.crt")+	if err != nil {+		t.Fatalf("error creating tmp server cert: %v", err)+	}+	defer os.Remove(serverCertFile.Name())+	if _, err := serverCertFile.Write([]byte(testServerCert)); err != nil {+		t.Fatalf("error writing to tmp server cert: %v", err)+	}++	serverKeyFile, err := ioutil.TempFile("", "server.key")+	if err != nil {+		t.Fatalf("error creating tmp server key: %v", err)+	}+	defer os.Remove(serverKeyFile.Name())+	if _, err := serverKeyFile.Write([]byte(testServerKey)); err != nil {+		t.Fatalf("error writing to tmp server key: %v", err)+	}++	certLoader, err := NewPeriodicCertLoader(serverCertFile.Name(), serverKeyFile.Name(), 1*time.Second)+	if err != nil {+		t.Fatalf("error starting periodic cert reloader: %v", err)+	}++	serverConfig := &tls.Config{}+	serverConfig.GetCertificate = func(*tls.ClientHelloInfo) (*tls.Certificate, error) {+		return certLoader.Current(), nil+	}+	clientConfig := &tls.Config{}+	clientConfig.RootCAs = prepareCertPool(rootCAFile.Name())++	// first check client -> server w/ TLS to validate initial reload worked+	testClientServerHello(t, clientConfig, serverConfig)++	// start certificate loader and nil out the current certificate, should expect an error+	go certLoader.Start()+	certLoader.current = nil+	testClientServerHelloError(t, clientConfig, serverConfig)

Ah I see. There's an initial reload as part of NewPeriodicCertLoader, which is how testClientServerHello on line 55 is passing. I agree that it would be more intuitive to start the periodic sync after we check the error case. I'll update to do this

andrewsykim

comment created time in 2 days

pull request commentkubernetes/kubernetes

cloud/node-controller use InstanceMetadataByProviderID

Yes I would like to get this into v1.19, pending unit tests

gongguan

comment created time in 2 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha 256a4a49af1ed36bb55729e0685309d5cc671207

tls: add periodic certificate loader This is primarily useful for key rotation or if the certificates should be switched out dynamically. Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 87fa0320d81ec754b541b97f691552085b7a834d

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha d4a26fa0f99d7863bf424868a3a11ad440ee6779

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha c811d2b64c4bc1d14041e17db425e81ece375a5c

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 2 days

Pull request review commentvmware/hamlet

Add Helper Functions for Reloading TLS Certificates

+// Copyright 2020 VMware, Inc. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0++package tls++import (+	"crypto/tls"+	"io/ioutil"+	"os"+	"testing"+	"time"+)++func Test_PeriodicReload(t *testing.T) {+	rootCAFile, err := ioutil.TempFile("", "rootCA.crt")+	if err != nil {+		t.Fatalf("error creating tmp root CA: %v", err)+	}+	defer os.Remove(rootCAFile.Name())+	if _, err := rootCAFile.Write([]byte(testRootCA)); err != nil {+		t.Fatalf("error writing to tmp root CA: %v", err)+	}++	serverCertFile, err := ioutil.TempFile("", "server.crt")+	if err != nil {+		t.Fatalf("error creating tmp server cert: %v", err)+	}+	defer os.Remove(serverCertFile.Name())+	if _, err := serverCertFile.Write([]byte(testServerCert)); err != nil {+		t.Fatalf("error writing to tmp server cert: %v", err)+	}++	serverKeyFile, err := ioutil.TempFile("", "server.key")+	if err != nil {+		t.Fatalf("error creating tmp server key: %v", err)+	}+	defer os.Remove(serverKeyFile.Name())+	if _, err := serverKeyFile.Write([]byte(testServerKey)); err != nil {+		t.Fatalf("error writing to tmp server key: %v", err)+	}++	certLoader, err := NewPeriodicCertLoader(serverCertFile.Name(), serverKeyFile.Name(), 1*time.Second)+	if err != nil {+		t.Fatalf("error starting periodic cert reloader: %v", err)+	}++	serverConfig := &tls.Config{}+	serverConfig.GetCertificate = func(*tls.ClientHelloInfo) (*tls.Certificate, error) {+		return certLoader.Current(), nil+	}+	clientConfig := &tls.Config{}+	clientConfig.RootCAs = prepareCertPool(rootCAFile.Name())++	// first check client -> server w/ TLS to validate initial reload worked+	testClientServerHello(t, clientConfig, serverConfig)++	// start certificate loader and nil out the current certificate, should expect an error+	go certLoader.Start()+	certLoader.current = nil+	testClientServerHelloError(t, clientConfig, serverConfig)

Do we need a delay before running this test?

Yes this is to test that the certificate is reloaded after the sync period.

Also, shouldn't we be setting certLoader.current to nil prior to starting the cert loader?

I don't think so because we're testing whether the certificate is reloaded from disk after the sync period.

andrewsykim

comment created time in 2 days

Pull request review commentvmware/hamlet

Add Helper Functions for Reloading TLS Certificates

+// Copyright 2020 VMware, Inc. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0++package tls++import (+	"crypto/tls"+	"sync"+	"time"++	log "github.com/sirupsen/logrus"+)++const (+	defaultSyncPeriod = 10 * time.Minute

Will do

andrewsykim

comment created time in 2 days

Pull request review commentkubernetes/kubernetes

Move pkg/cloudprovider/provider to k8s.io/legacy-cloud-providers

 See the License for the specific language governing permissions and limitations under the License. */ -package cloudprovider+package install

I agree and had the same comment here https://github.com/kubernetes/kubernetes/pull/91218#discussion_r427516521

For the cloud-controller-manager case where we can't import k8s.io/kubernetes we can import the legacy providers individually. The other components can continue importing k8s.io/kubernetes/pkg/cloudprovider/providers.

cici37

comment created time in 3 days

pull request commentkubernetes/kubernetes

Automated cherry pick of #92584: kube-proxy ipvs masquerade hairpin traffic

/assign @spiffxp

Can you sign off on the /cluster changes here?

aojea

comment created time in 3 days

pull request commentkubernetes/kubernetes

Automated cherry pick of #92584: kube-proxy ipvs masquerade hairpin traffic

/kind bug /priority important-soon

aojea

comment created time in 3 days

pull request commentvmware/hamlet

Add Helper Functions for Reloading TLS Certificates

@sergiopozoh updated to remove one-way TLS helper functions, PTAL :)

andrewsykim

comment created time in 3 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha 0ad00f9278e58f85f8d0e750dfbca0cc3d8cfb43

tls: add periodic certificate loader This is primarily useful for key rotation or if the certificates should be switched out dynamically. Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 0ca149be497775d0e56081b15f5c41206a9ad0dd

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha f7fe50eb5f87c4a6b9234fa561ccae85cedebaba

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 4ae26796fba6322218e20c18b85bb4d3ca5d7465

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 3 days

Pull request review commentkubernetes/kubernetes

Remove dependency pkg/features from CCM

 const ( 	// 	// Allows sending warning headers in API responses. 	WarningHeaders featuregate.Feature = "WarningHeaders"++	// owner @smarterclayton+	// alpha: v1.16+	// beta:  v1.19+	//+	// Enable legacy behavior to vary cluster functionality on the node-role.kubernetes.io labels. On by default (legacy), will be turned off in 1.18.+	LegacyNodeRoleBehavior featuregate.Feature = "LegacyNodeRoleBehavior"

Wouldn't be opposed to a features package like this one in k8s.io/cloud-provider, not sure if there's been discussions around a centralized place for all feature gates.

cici37

comment created time in 3 days

Pull request review commentkubernetes/kubernetes

Move pkg/cloudprovider/provider to k8s.io/legacy-cloud-providers

-### Deprecation Notice

This README may still be useful in the legacy-cloud-provider repo. May want to re-add it in a follow-up PR

cici37

comment created time in 3 days

Pull request review commentkubernetes/kubernetes

Remove dependency pkg/features from CCM

 const ( 	// 	// Allows sending warning headers in API responses. 	WarningHeaders featuregate.Feature = "WarningHeaders"++	// owner @smarterclayton+	// alpha: v1.16+	// beta:  v1.19+	//+	// Enable legacy behavior to vary cluster functionality on the node-role.kubernetes.io labels. On by default (legacy), will be turned off in 1.18.+	LegacyNodeRoleBehavior featuregate.Feature = "LegacyNodeRoleBehavior"

Ideally there's a better hope for these but I think this will be okay for v1.19

cici37

comment created time in 3 days

pull request commentvmware/hamlet

Add Helper Functions for Reloading TLS Certificates

Fair enough. Will update the PR to remove the regular TLS config

andrewsykim

comment created time in 3 days

pull request commentvmware/hamlet

Add Helper Functions for Reloading TLS Certificates

What is more important, I do not think there is a way to enforce the optionality of the condition. It may be the case that implementers decide to do it in this way (unauthenticated clients are ok) and a server starts leaking info to a client that has connected randomly. You cannot know in advance what clients are trusted or not, but only authenticating them.

Yes, I'm thinking this would be up to the implementer based on what is expected/required of their platform.

What issue do you find requiring client-side auth?

Having to manage client-side certificates in an environment where all Hamlets are within the same trust boundary. It probably doesn't hurt to use mTLS here but it doesn't seem strictly required in this scenario. The implementer should use at their own risk, but it should be allowed IMO.

andrewsykim

comment created time in 3 days

pull request commentvmware/hamlet

Add Helper Functions for Reloading TLS Certificates

It's part of the current spec. The reason is that by definition in this spec, you are sharing data across untrusted entities (the federated platforms) and thus both must be authenticated.

Could we allow regular TLS under the condition that data is shared across two trusted entites? For example, two clusters managed by different teams but in the same organization?

andrewsykim

comment created time in 3 days

pull request commentvmware/hamlet

Add Helper Functions for Reloading TLS Certificates

They are really appreciated. In order to not break existing v1alpha1 servers and clients, could you please make these contributions part of the v1alpha2 version?

Can you be more specific here? Is there a v1alpha2 branch I should use or should I create a v1alpha2 package in pkg/tls?

Also, please remember to update the client and server bootstrapping examples.

I believe I updated the examples in last commit, please let me know if I missed anything.

andrewsykim

comment created time in 3 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha b3a2af6bf744c97df99dcfdbadeb8b00cbb6360a

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 5be2738d8feffa4e45b3dd3e8d08853b843087c9

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha f88cb2c35e880fa651f6ed9e5a1412a188b1a936

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 3 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha f35ed4913618ec757733ed284adf5a59a574576f

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 6d3a7c8315fd9e091e710ebd6f308e6a434ee3a4

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 390bfce2b604938fc8a2e10827ff90324c33047f

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 3 days

pull request commentvmware/hamlet

Add Helper Functions for Reloading TLS Certificates

@sergiopozoh what's the context/reason behind only supporting mTLS?

andrewsykim

comment created time in 3 days

pull request commentkubernetes/kubernetes

[WIP] Implement ServiceSpec.AllocateLoadBalancerNodePorts

TestCompatibility/core.v1.Service/HEAD: compatibility.go:329: if the diff is expected because of a new type or a new field, re-run with UPDATE_COMPATIBILITY_FIXTURE_DATA=true to update the compatibility data

^ saw this in the logs and there's a hack script to update this hack/update-generated-api-compatibility-data.sh.

uablrek

comment created time in 3 days

Pull request review commentvmware/hamlet

Add Helper Functions for Reloading TLS Certificates

 func prepareCommonConfig(peerCert string, peerKey string) *tls.Config { 	return config } -// PrepareServerConfig prepares a TLS config instance for a server that wants-// mTLS enabled.+func prepareServerConfigWithPeriodicReload(peerCert string, peerKey string, syncPeriod time.Duration) *tls.Config {+	certLoader, err := NewPeriodicCertLoader(peerCert, peerKey, syncPeriod)+	if err != nil {+		log.WithFields(log.Fields{+			"err": err,+		}).Fatalln("Error starting TLS certificate reloader")+	}+	go certLoader.Start()++	config := &tls.Config{}+	config.GetCertificate = func(*tls.ClientHelloInfo) (*tls.Certificate, error) {+		return certLoader.Current(), nil+	}+	return config+}++func prepareClientConfigWithPeriodicReload(peerCert string, peerKey string, syncPeriod time.Duration) *tls.Config {+	certLoader, err := NewPeriodicCertLoader(peerCert, peerKey, syncPeriod)+	if err != nil {+		log.WithFields(log.Fields{+			"err": err,+		}).Fatalln("Error starting TLS certificate reloader")+	}+	go certLoader.Start()++	config := &tls.Config{}+	config.GetClientCertificate = func(*tls.CertificateRequestInfo) (*tls.Certificate, error) {+		return certLoader.Current(), nil+	}+	return config+}++// PrepareServerConfig prepares a TLS config instance for a server that wants mTLS enabled.

Fine to remove it -- figured doing this is more friendly to consumers of this package that depend on this function signature.

andrewsykim

comment created time in 3 days

pull request commentkubernetes/kubernetes

cloud/node-controller use InstanceMetadataByProviderID

@gongguan let's remove the exists check in updateNodeAddress, please keep it in a separate commit so it's easier to revert.

gongguan

comment created time in 3 days

pull request commentkubernetes/kubernetes

Automated cherry pick of #92584: kube-proxy ipvs masquerade hairpin traffic

@aojea I think we need to cherry-pick these two commits to get the e2es working on this branch https://github.com/kubernetes/kubernetes/pull/89327 & https://github.com/kubernetes/kubernetes/pull/89854.

aojea

comment created time in 3 days

pull request commentkubernetes/kubernetes

Automated cherry pick of #92584: kube-proxy ipvs masquerade hairpin traffic

@aojea I think we need to cherry-pick these two commits to get the e2es working on this branch https://github.com/kubernetes/kubernetes/pull/89327 & https://github.com/kubernetes/kubernetes/pull/89854.

aojea

comment created time in 3 days

pull request commentkubernetes/kubernetes

Automated cherry pick of #92584: kube-proxy ipvs masquerade hairpin traffic

@aojea I think we need to cherry-pick these two commits to get the e2es working on this branch https://github.com/kubernetes/kubernetes/pull/89327 & https://github.com/kubernetes/kubernetes/pull/89854.

aojea

comment created time in 3 days

pull request commentkubernetes/kubernetes

Automated cherry pick of #92584: kube-proxy ipvs masquerade hairpin traffic

hehe, the flexvolume test failure happened in 1.19

Hmm.. good to know

aojea

comment created time in 3 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha 706840b7840dab66d071de8309869980b9d7bb17

tls: add periodic certificate loader This is primarily useful for key rotation or if the certificates should be switched out dynamically. Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 0eb6b6ac00bbba317e5c76817efecf9e449df183

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 6f380105ab235a74d3ce96fd5dc9c9fe58090ba2

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 62a498497193f455eb013942b3b4a8182e741765

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 4 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha 1e70c894a2d06ca54bf7dd696960471eea20251c

tls: add periodic certificate loader This is primarily useful for key rotation or if the certificates should be switched out dynamically. Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha d6cd1df03ae022eb81e97e26b482ef1bdbb26e1c

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha add63e179e7ea2be126af5dc8b8a98f6d3c328cf

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 46ac15c12fee6161dac56007d6e0f23a151d63e5

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 4 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha d5d87fb93f43969279df70ba064bdb400f73ec00

tls: add periodic certificate loader This is primarily useful for key rotation or if the certificates should be switched out dynamically. Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha d9289b23179882fb151c232685857a5b34981723

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha dcb63b9a323c91318ecefa88c3519d3e53b7ee85

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 5b1e3bdb62ddecd6eca17528651f653060945551

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 6 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha 6d7f3567861f08fc3763b6dc423681833596a98b

tls: add periodic certificate loader This is primarily useful for key rotation or if the certificates should be switched out dynamically. Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 9e4f7e47995dc5a79a9207d7df4287347f347618

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 0f33ef7e69422044f0ef5716688d2659a55393b2

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha c4ed59173f7d1799a2f993048a896253c6076f2e

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 6 days

PR opened vmware/hamlet

Add Helper Functions for Reloading TLS Certificates

Fixes https://github.com/vmware/hamlet/issues/12

Adds helper functions to pkg/tls that returns a tls.Config based on a periodic watch of the TLS cert/key file. This is useful for TLS key rotation or if keys need to be updated without restarting the hamlet client/server. In a future PR we may want to watch the TLS cert/key file by watching file system notifcations. For now I've kept it simple by just periodically reloading the certificates.

This PR also:

  • adds helper functions for regular TLS client/server config
  • deprecate the existing helper functions PrepareServerConfig and PrepareClientConfig in favor of PrepareMutualTLSServerConfig and PrepareMutualTLSClientConfig so it is clear that mTLS is enabled.
  • adds unit tests for TLS helper functions
  • update examples to use new TLS helper functions.

cc @sergiopozoh

+497 -8

0 comment

6 changed files

pr created time in 6 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha cf8285be9a0408300bf736dad57432665ee2aa92

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 59b3475a2b465cc37ad6588c5272615e307478cc

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 703c8017b4e3498dd56e15e9abe947a23650a72a

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 6 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha 61b8c2983b300b6a8e058eda9f166bc13bfb0572

tls: add periodic certificate loader This is primarily useful for key rotation or if the certificates should be switched out dynamically. Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha c9cbd2f23bf960457ed65f372190a190f7518685

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 289c6171a25bc714e3599c9e48fb0177514c1500

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha ba7837bc8e462ce98becec95b591023b63f1b434

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 6 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha e82be5e49c42e44c680ee2d00443479f0d342704

tls: add periodic certificate loader This is primarily useful for key rotation or if the certificates should be switched out dynamically. Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 7ee0f6c612956a5e73591aa0733f9d0407ee8222

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 9b6e204018a775ad478e3e88f1cd0907254e5654

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha e2442fd824e5697550d350fe8eb893520fe5ca1f

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 6 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha 53a246236b7c182f04f0b1e15eaaccd57d204f79

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 3eef62872a435a3dfd45eab6793dec5889ed1389

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 6 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha 4298aea696affc917e1c7c1c436a525a66b51887

examples: update to use new helper functions for mTLS config Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 6 days

push eventandrewsykim/hamlet

Andrew Sy Kim

commit sha d3f3cb92029c6b868ea396fd8f5633453803a84d

tls: add periodic certificate loader This is primarily useful for key rotation or if the certificates should be switched out dynamically. Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha 2695046290fc6def1867f82fd264823122b09271

tls: add helper functions for tls.Config using periodic certificate loader and regular TLS Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

Andrew Sy Kim

commit sha fe144fdd91a1ccbf59a260dec8886638429a8835

tls: add unit tests for tls.Config helper functions Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 6 days

pull request commentkubernetes/kubernetes

proxier/ipvs: check already binded addresses in the IPVS dummy interface

IPVS tests back to "normal" (FlexVolume tests failing)

andrewsykim

comment created time in 7 days

create barnchandrewsykim/hamlet

branch : tls-reload

created branch time in 7 days

fork andrewsykim/hamlet

Multi-Vendor Service Mesh Interoperation

fork in 7 days

pull request commentkubernetes/kubernetes

proxier/ipvs: check already binded addresses in the IPVS dummy interface

Fixed, thanks for catching that

andrewsykim

comment created time in 7 days

push eventandrewsykim/kubernetes

Andrew Sy Kim

commit sha de2ecd7e2fbfa3be2d0fa5bcc7448dde8aaaa456

proxier/ipvs: check already binded addresses in the IPVS dummy interface Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com> Co-authored-by: Laurent Bernaille <laurent.bernaille@gmail.com>

view details

push time in 7 days

push eventandrewsykim/kubernetes

Andrew Sy Kim

commit sha 6b74f5d0a13f652c67f29d3a39743cc6a39ab578

proxier/ipvs: check already binded addresses in the IPVS dummy interface Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com> Co-authored-by: Laurent Bernaille <laurent.bernaille@gmail.com>

view details

push time in 7 days

Pull request review commentkubernetes/kubernetes

proxier/ipvs: check already binded addresses in the IPVS dummy interface

 func (r *realIPGetter) NodeIPs() (ips []net.IP, err error) { 	return ips, nil } +// BindedIPs returns all addresses there were binded to the IPVS dummy interface kube-ipvs0

fixed

andrewsykim

comment created time in 7 days

pull request commentkubernetes/kubernetes

proxier/ipvs: check already binded addresses in the IPVS dummy interface

@lbernail had to fix a conflict, can you lgtm this again?

andrewsykim

comment created time in 7 days

pull request commentkubernetes/kubernetes

Automated cherry pick of #92584: kube-proxy ipvs masquerade hairpin traffic

@aojea in https://github.com/kubernetes/kubernetes/pull/90678 I had to back port some commits to get IPVS e2e working since we got these tests working earlier in the 1.19 cycle. That's probably why it's failing

aojea

comment created time in 7 days

push eventandrewsykim/kubernetes

Patrick Ohly

commit sha c20721aa9f0a69f267f899ca387fdc489cfe6f24

storage: enhance test for ValidateCSIDriverUpdate This revised test changes one field at a time in various ways. This ensures that one failure reason doesn't mask the other. An incorrect comment also gets fixed. Suggested in https://github.com/kubernetes/kubernetes/pull/80568#pullrequestreview-272836101.

view details

Claudiu Belu

commit sha 31ea600b284515dd3470b244f9c6e1548408ecfe

images: Adds GOARM to images' Makefiles In order to build the image binaries, the GOARM variable is required, but not all Makefiles have it defined, causing the make to fail on those images. This adds the require GOARM variable to the Makefiles in question.

view details

Alena Prokharchyk

commit sha d634ed3850ff1980f21590eee051d4428acf4601

Removed unnecessary not nil check in node registration process

view details

Manuel Rüger

commit sha eb6c7169276a1978b851deafb25b507caf696ac4

PodTolerationRestriction: Mention Whitelist Scope in Error Currently it's not clear if the issue came from the namespace whitelist of if the namespace whitelist was not applied at all (i.e. via a misspelled annotation). This makes the error more explicit if the pod tolerations caused a conflict with cluster-level or namespace-level whitelist. Signed-off-by: Manuel Rüger <manuel@rueg.eu>

view details

Seth Pollack

commit sha 75af2fca6125516dff42e9825ceea89367986f78

add labels to diff command

view details

Paulo Gomes

commit sha e7ced21235820139afc8dbb2e99314b9b69ec7fa

Invert error validation

view details

junxu

commit sha c58959c8ba7819345d5cc2b17ce4e95ccbc92d5b

Fix code style

view details

liuxu

commit sha 2367569f138ddb35385aed5e7e485eed425c73a9

fix if don't set ephemeral-storage limit emptyDir's sizeLimit doesn't work

view details

SataQiu

commit sha 17f3cd48a54483b4c6b7dc1d742194a1f41daf0a

add '--logging-format' flag to kube-controller-manager Signed-off-by: SataQiu <1527062125@qq.com>

view details

Marcin Maciaszczyk

commit sha e5af792ad29fdbd8f401b5d0d78e8fb0d64c1fe6

Bump Dashboard to v2.0.1

view details

lo24

commit sha cda593e822d2e03f621167f007c183faf5b1d910

fix TestValidateNodeIPParam: break was being used instead of continue

view details

Dan Winship

commit sha ddebbfd806b5813cc0f6d67ae9608be393729922

update for APIs being moved to utilnet Several of the functions in pkg/registry/core/service/ipallocator were moved to k8s.io/utils/net, but then the original code was never updated to used to the vendored versions. (utilnet's version of RangeSize does not have the IPv6 special case that the original code did, so we need to move that to NewAllocatorCIDRRange now.)

view details

Dan Winship

commit sha 4a7c86c105f49972a5d7b8150cdba59eafb8a0fd

make test a bit more generic

view details

Dan Winship

commit sha f6dcc1c07e0a2d3c583cb90e1cdb7ec4718625ce

Minor tweak to IPv6 service IP allocation The service allocator skips the "broadcast address" in the service CIDR, but that concept only applies to IPv4 addressing.

view details

Lubomir I. Ivanov

commit sha 7ddd966ed2038133dce3f93a062a9da1cb809088

kubeadm: mark --experimental-kustomize as deprecated

view details

SataQiu

commit sha 800dd19fc23f14f9e83f09ff40155395af9f7cb0

increase robustness for kubeadm etcd operations Signed-off-by: SataQiu <1527062125@qq.com>

view details

David Eads

commit sha 7f172286342c4238d4b636c113d35835ff9299ee

prevent panic in azure cloud provider from killing the process

view details

Antonio Ojea

commit sha 069707f75a14b9c98538755b089a123192bb0368

refactor and instrument range_allocator cidr_sets refactor and add the following metrics to the cidr_sets used by the range allocator:, under the subsystem: node_ipam_controller cidrset_cidrs_allocations_total cidrset_cidrs_releases_total cidrset_usage_cidrs cidrset_allocation_tries_per_request

view details

Yan Yao

commit sha cef787695235fb0fee1783f9e68edbc345147801

Fix golint failures in pkg/kubelet/lifecycle Co-authored-by: Sergey Kanzhelev <S.Kanzhelev@live.com>

view details

Luigi Bitonti

commit sha 51f788c6dc962d3773c92094d0b67cc1da081d49

Add ebtables rule delete function + broute table + brouting chain

view details

push time in 7 days

push eventandrewsykim/cloud-provider-vsphere

Andrew Sy Kim

commit sha 5b971a690341fb44cfa9b3f7fbe4ec18136a9f1d

skip staticcheck for SA1019 Signed-off-by: Andrew Sy Kim <kim.andrewsy@gmail.com>

view details

push time in 8 days

PR opened kubernetes/cloud-provider-vsphere

nsx-t: ignore linter for SA1019

Signed-off-by: Andrew Sy Kim kim.andrewsy@gmail.com

<!-- Thanks for sending a pull request! -->

What this PR does / why we need it: We can't remove tlsConfig.BuildNameToCertificate() until we're on Go 1.14 so we should ignore the linter for now.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

Release note: <!-- Steps to write your release note:

  1. Use the release-note-* labels to set the release note state (if you have access)
  2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write NONE. -->
NONE
+1 -0

0 comment

1 changed file

pr created time in 8 days

create barnchandrewsykim/cloud-provider-vsphere

branch : fix-nsxt-linter

created branch time in 8 days

issue commentkubernetes/cloud-provider-aws

Status of project and documentation

I still do not think "starting from scratch" is a great idea....

Starting a v2 provider from scratch wouldn't mean we abandon the existing one. There are some feature requests for the legacy provider, like the node and ELB name change, that are just too difficult to implement without breaking existing clusters. We can maintain both providers for the forseeable future.

darwin67

comment created time in 8 days

Pull request review commentkubernetes/kubernetes

ipvs: ensure ip_vs_sed kernel module is loaded

 func GetRequiredIPVSModules(kernelVersion *version.Version) []string { 	// "nf_conntrack_ipv4" has been removed since v4.19 	// see https://github.com/torvalds/linux/commit/a0ae2562c6c4b2721d9fddba63b7286c13517d9f 	if kernelVersion.LessThan(version.MustParseGeneric("4.19")) {-		return []string{KernelModuleIPVS, KernelModuleIPVSRR, KernelModuleIPVSWRR, KernelModuleIPVSSH, KernelModuleNfConntrackIPV4}+		return []string{KernelModuleIPVS, KernelModuleIPVSRR, KernelModuleIPVSWRR, KernelModuleIPVSSH, KernelModuleNfConntrackIPV4, KernelModuleIPVSSED} 	}-	return []string{KernelModuleIPVS, KernelModuleIPVSRR, KernelModuleIPVSWRR, KernelModuleIPVSSH, KernelModuleNfConntrack}+	return []string{KernelModuleIPVS, KernelModuleIPVSRR, KernelModuleIPVSWRR, KernelModuleIPVSSH, KernelModuleNfConntrack, KernelModuleIPVSSED}

Alternatively we can check the existence of the module based on the passed in scheduling algorithm as opposed to checking a set of scheduling algorithms every time.

cmluciano

comment created time in 8 days

Pull request review commentkubernetes/kubernetes

ipvs: ensure ip_vs_sed kernel module is loaded

 func GetRequiredIPVSModules(kernelVersion *version.Version) []string { 	// "nf_conntrack_ipv4" has been removed since v4.19 	// see https://github.com/torvalds/linux/commit/a0ae2562c6c4b2721d9fddba63b7286c13517d9f 	if kernelVersion.LessThan(version.MustParseGeneric("4.19")) {-		return []string{KernelModuleIPVS, KernelModuleIPVSRR, KernelModuleIPVSWRR, KernelModuleIPVSSH, KernelModuleNfConntrackIPV4}+		return []string{KernelModuleIPVS, KernelModuleIPVSRR, KernelModuleIPVSWRR, KernelModuleIPVSSH, KernelModuleNfConntrackIPV4, KernelModuleIPVSSED} 	}-	return []string{KernelModuleIPVS, KernelModuleIPVSRR, KernelModuleIPVSWRR, KernelModuleIPVSSH, KernelModuleNfConntrack}+	return []string{KernelModuleIPVS, KernelModuleIPVSRR, KernelModuleIPVSWRR, KernelModuleIPVSSH, KernelModuleNfConntrack, KernelModuleIPVSSED}

In the past we haven't been checking for all the IPVS scheduling modules aside from the really common ones and the one we use by default (ip_vs_rr). I don't think it's worthwhile to stop kube-proxy if this module is missing but we should definitely document that it should be loaded if it's the desired scheduling algorithm.

cmluciano

comment created time in 8 days

issue commentvmware/hamlet

Support dynamic reload of TLS certificates

Nvm, forgot that there's a TLS pkg here https://github.com/vmware/hamlet/tree/master/pkg/tls -- so we could potentially add the certwatch logic in there if consumers want to use it.

andrewsykim

comment created time in 8 days

issue commentvmware/hamlet

Support dynamic reload of TLS certificates

Realizing now that hamlet receives a tls.Config for the server constructor so maybe it was intentionally up to the consumer of Hamlet to implement this. Either way it would be nice if Hamlet can natively support this given a certificates directory or the path to the TLS certificate/key files.

andrewsykim

comment created time in 8 days

issue openedvmware/hamlet

Support dynamic reload of TLS certificates

It would be useful if Hamlet can support watching the paths to the TLS certificates and dynamically update the serving certificates when they change. This would allow for key rotation by simply updating the certificates on disk.

This should be possible using inotify watchers and the GetCertificates field in tls.Config.

created time in 8 days

issue openedkubernetes-sigs/cluster-api-provider-vsphere

Set InternalIP machine status.addresses

/kind feature

Describe the solution you'd like Almost always a VM will have an address internal to the datacenter. In the case where we don't know if an address is external or internal to the datacenter, we should assume it is internal and set the type as InternalIP. Today we always set it as an ExternalIP

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

  • Cluster-api-provider-vsphere version:
  • Kubernetes version: (use kubectl version):
  • OS (e.g. from /etc/os-release):

created time in 8 days

Pull request review commentkubernetes/kubernetes

cloud/node-controller use InstanceMetadataByProviderID

 func (cnc *CloudNodeController) reconcileNodeLabels(nodeName string) error { }  // UpdateNodeAddress updates the nodeAddress of a single node-func (cnc *CloudNodeController) updateNodeAddress(ctx context.Context, node *v1.Node, instances cloudprovider.Instances) {+func (cnc *CloudNodeController) updateNodeAddress(ctx context.Context, node *v1.Node) { 	// Do not process nodes that are still tainted 	cloudTaint := getCloudTaint(node.Spec.Taints) 	if cloudTaint != nil { 		klog.V(5).Infof("This node %s is still tainted. Will not process.", node.Name) 		return 	}-	// Node that isn't present according to the cloud provider shouldn't have its address updated-	exists, err := ensureNodeExistsByProviderID(ctx, instances, node)-	if err != nil {-		// Continue to update node address when not sure the node is not exists-		klog.Errorf("%v", err)-	} else if !exists {-		klog.V(4).Infof("The node %s is no longer present according to the cloud provider, do not process.", node.Name)-		return++	instanceMetadataGetter := func(providerID string, node *v1.Node) (*cloudprovider.InstanceMetadata, error) {

nice :)

gongguan

comment created time in 9 days

pull request commentkubernetes/kubernetes

IPVS: kubelet, kube-proxy: unmark packets before masquerading …

Maybe we just need to update to use the staging CI image for quay? @BenTheElder might know?

aojea

comment created time in 9 days

pull request commentkubernetes/kubernetes

IPVS: kubelet, kube-proxy: unmark packets before masquerading …

I thought the flex volume tests were failing because quay.io was down but that seemed to be fixed now https://github.com/kubernetes/kubernetes/issues/91242 ?

aojea

comment created time in 9 days

pull request commentkubernetes/enhancements

add node shutdown KEP

@thockin I think some of this work overlaps with the node shutdown work @bobbypage and @mrunalp are working on, but I think the scope of this KEP is slightly bigger since it tries to cover node failures, not just nodes shutting down.

yastij

comment created time in 9 days

issue commentkubernetes/kubernetes

Add unit tests for iptables rules in the IPVS proxy

/remove-triage unresolved

andrewsykim

comment created time in 9 days

Pull request review commentkubernetes/kubernetes

Move cmd/controller-manager to k8s.io/controller-manager

 package options  import ( 	"github.com/spf13/pflag"--	kubectrlmgrconfig "k8s.io/kubernetes/pkg/controller/apis/config"+	config2 "k8s.io/controller-manager/config"

cmconfig or controllermanagerconfig

cici37

comment created time in 9 days

Pull request review commentkubernetes/kubernetes

Move cmd/controller-manager to k8s.io/controller-manager

 import ( 	"k8s.io/apimachinery/pkg/util/sets" 	cliflag "k8s.io/component-base/cli/flag" 	"k8s.io/component-base/config/options"-	kubectrlmgrconfig "k8s.io/kubernetes/pkg/controller/apis/config"+	config2 "k8s.io/controller-manager/config"

cmconfig or controllermanagerconfig

cici37

comment created time in 9 days

more