profile
viewpoint
Nic Cope negz @upbound Georgetown Liquor Company https://www.linkedin.com/in/ncope/ @crossplane engineering lead.

negz/grabby 9

A Usenet binary grabber.

crossplane/agent 4

Connector Agent to connect to Crossplane from any cluster

negz/crossdocs 3

API Reference Docs generator for Crossplane

negz/draino 1

Automatically cordon and drain Kubernetes nodes based on node conditions

negz/example-cnp 1

Platform API as Configuration

negz/haproxy-docker-wrapper 1

Docker-friendly wrapper for haproxy

negz/addon-oam-kubernetes-local 0

Run OAM workloads on a Kubernetes cluster.

negz/app-wordpress 0

A sample Crossplane Wordpress app

negz/autoscaler 0

Autoscaling components for Kubernetes

Pull request review commentnegz/example-cnp

Make the kind of a published composite resource configurable.

 spec:     group: cnp.example.org     version: v1alpha1     names:-      kind: Cluster-      listKind: ClusterList-      plural: clusters-      singular: cluster+      kind: AllocatedCluster+      listKind: AllocatedClusterList+      plural: allocatedclusters+      singular: allocatedcluster

The platform builder gets to pick the name of both the composite resource and the published resource, so they can pick which one has a differentiating suffix or prefix. This would allow the builder to choose between:

  • ClusterClaim for the published type and Cluster for the composite type.
  • ClusterRequirement for the published type and Cluster for the composite type.
  • Cluster for the published type and AllocatedCluster for the composite type.
  • Cluster for the published type and CompositeCluster for the composite type.

The only limitation is that both types can't share the same name - they can't both be Cluster.

negz

comment created time in 4 days

push eventnegz/example-cnp

Nic Cope

commit sha 22c3415d1a5d0b5230bf7bf3ddeeadd7c66322e1

Make the kind of a published composite resource configurable. Signed-off-by: Nic Cope <negz@rk0n.org>

view details

push time in 4 days

create barnchnegz/example-cnp

branch : publishedname

created branch time in 4 days

push eventnegz/example-cnp

Nic Cope

commit sha 7c804b5633efa325b7d0fdd4e7373f7d0610e468

Rebrand from AAC to CNP

view details

push time in 4 days

push eventnegz/aac

Nic Cope

commit sha 60def52595d38955ac6fa773a408353102f0b854

An example sketch of a platform configuration Signed-off-by: Nic Cope <negz@rk0n.org>

view details

push time in 4 days

push eventnegz/aac

Nic Cope

commit sha e8f2e92d0e579e81812f58dc2bbadb774f02a23e

An example sketch of a platform configuration Signed-off-by: Nic Cope <negz@rk0n.org>

view details

push time in 4 days

push eventnegz/aac

Nic Cope

commit sha c1d02157eb5afa180f6d70f4df0889e53f3e330c

An example sketch of a platform configuration Signed-off-by: Nic Cope <negz@rk0n.org>

view details

push time in 4 days

Pull request review commentcrossplane/crossplane

One-pager for Crossplane Helm Provider

+# Crossplane Helm Provider++* Owner: Hasan Turken (@turkenh)+* Reviewers: Crossplane Maintainers+* Status: Accepted++## Background++As a platform builder, as part of my Composite Resource creation, I would like to make some initial provisioning on my +infrastructure resources after creating them. This provisioning could have different meanings depending on the type of +the infrastructure. For example, for a database instance (e.g. `CloudSQLInstance`, `RDSInstance` ...), this could mean+creating additional [databases, users and roles](https://github.com/crossplane/crossplane/issues/29) which would require+a controller using clients of the database. This could be achieved via another crossplane provider similar to +[Terraform’s MySQL Provider](https://www.terraform.io/docs/providers/mysql/index.html). ++When the infrastructure resource is a Kubernetes cluster (e.g. `GKECluster`, `EKSCluster` ...), by provisioning we usually+mean creating Kubernetes resources on the cluster which could be in the form of raw yaml manifests or in the form of+application packages. Helm is currently the most popular packaging solution for Kubernetes applications and a Crossplane+Helm Provider could enable easy and quick demonstration of Crossplane capabilities. ++This provider will enable deployment of helm charts to (remote) Kubernetes Clusters typically provisioned via+Crossplane. Considering the issues with helm 2 (e.g. security problems regarding tiller and lack of proper go +clients/libraries for helm), **we will focus and only support Helm 3**.++## Design++We will implement a Kubernetes controller watching `Release` custom resources and deploying helm charts with the desired+configuration. Since this controller needs to interact with Kubernetes API server, it is a good fit for [Kubernetes+native providers](https://github.com/crossplane/crossplane/blob/master/design/one-pager-k8s-native-providers.md#kubernetes-native-providers)+concept in Crossplane. By using existing [Kubernetes Provider](https://github.com/crossplane/crossplane/blob/master/design/one-pager-k8s-native-providers.md#proposal-kubernetes-provider-kind)+Kind, we will be able to manage helm releases in **Crossplane provisioned external clusters**, **existing external+clusters** and also **Crossplane control cluster** (a.k.a. local cluster).++Helm 3 introduced a new feature called [`post rendering`](https://helm.sh/docs/topics/advanced/#post-rendering) which+enables manipulation of generated manifests before deploying into the cluster. This increases usability of existing helm+charts for advanced use cases by allowing to apply last mile configurations. With Crossplane Helm Provider, we would+like to leverage this feature to enable post rendering charts via simple patches.++### `Release` Custom Resource++```+apiVersion: helm.crossplane.io/v1alpha1+kind: Release+metadata:+  name: wordpress-example+spec:+  forProvider:+    chart:+      name: wordpress+      repository: https://charts.bitnami.com/bitnami+      version: 9.3.19+    namespace: wordpress+    values:+      mariadb:+        enabled: false+      externaldb:+        enabled: true+    valuesFrom:+    - configMapKeyRef:+        name: wordpress-defaults+        namespace: prod+        key: values.yaml+        optional: false+    set:+    - name: wordpressBlogName+      value: "Hello Crossplane"+    - name: externalDatabase.host+      valueFrom:+        secretKeyRef:+          name: dbconn+          key: host+    - name: externalDatabase.user+      valueFrom:+        secretKeyRef:+          name: dbconn+          key: username+    - name: externalDatabase.password+      valueFrom:+        secretKeyRef:+          name: dbconn+          key: password+    patchesFrom:+    - configMapKeyRef:+        name: labels+        namespace: prod+        key: patches.yaml+        optional: false+    - configMapKeyRef:+        name: wordpress-nodeselector+        namespace: prod+        key: patches.yaml+        optional: false+    - secretKeyRef:+        name: image-pull-secret-patch+        namespace: prod+        key: patches.yaml+        optional: false+  providerRef: +    name: cluster-1-provider+  reclaimPolicy: Delete+```++### Value Overrides++There are multiple ways to provide value overrides and final values will be composed with the following precedence:++1. `spec.forProvider.valuesFrom` array, items with increasing precedence+1. `spec.forProvider.values`+1. `spec.forProvider.set` array, items with increasing precedence++### Post Rendering Patches++It will be possible to provide post rendering patches which will make last mile configurations using+[`post rendering`](https://helm.sh/docs/topics/advanced/#post-rendering) option of Helm. `spec.forProvider.patchesFrom`+array will be used to specify patch definitions satisfying [kustomizes patchTransformer interface](https://kubernetes-sigs.github.io/kustomize/api-reference/kustomization/patches/).++Example:++```+patches:+- patch: |-+    - op: replace+      path: /some/existing/path+      value: new value+  target:+    kind: MyKind+    labelSelector: "env=dev"+- patch: |-+    - op: add+      path: /spec/template/spec/nodeSelector+      value:+        node.size: really-big+        aws.az: us-west-2a+  target:+    kind: Deployment+    labelSelector: "env=dev"+```++### Triggering a Helm Upgrade++Running `helm upgrade` for a helm release creates a new [`Revision`](https://helm.sh/docs/helm/helm_history/) regardless+of there is a change in generated manifests or not. With a controller running inside the cluster with active+reconciliation, we need a consistent mechanism to decide whether we need an `helm upgrade` or not to prevent redundant+revisions. [Helm Go SDK](https://helm.sh/docs/topics/advanced/#go-sdk) represents an existing helm release with+[`Release`](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/release/release.go#L22)+object which keeps values like [Chart information](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/chart/chart.go#L28),+[user configuration](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/release/release.go#L31)+(e.g. value overrides), [Release information](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/release/info.go#L21)+and [string representation of rendered templates](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/release/release.go#L33).++Everything in the `Release` custom resource spec is observable via Helm's `Release` struct except post rendering patches+(e.g. `PatchesFrom`) we applied. We will store the information about used patches in the last deployed Helm `Release` +as an [annotation](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/chart/metadata.go#L58),+so that whole actual state will be kept on Helm Storage later to be observed. We will store [`resourceVersion`](https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions)+of used patches as follows:++```+"release.helm.crossplane.io/patch-1-e286dbc1-707d-4e39-b0e8-1012c047e662" = "314853"+```++Here, `e286dbc1-707d-4e39-b0e8-1012c047e662` is the UID of the ConfigMap referenced in `PatchesFrom` field and `314853`+is the [`resourceVersion`](https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions). This way,+we will be able to decide whether there is a change related to patches which requires a `helm upgrade`.++Flow to decide an Helm Upgrade:++1. Get last [release](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/release/release.go#L22)+information using Helm Go Client.+1. Compare desired chart spec with the observe [Chart information](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/chart/chart.go#L28).+1. Compose final value overrides and compare with [Release.Config](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/release/release.go#L31).+1. Compose final patch annotations and compare with [Chart.Metadata.Annotations](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/chart/metadata.go#L58)++Also see [Triggering a Helm Upgrade Based on Rendered Templates](#triggering-a-helm-upgrade-based-on-rendered-templates)+as an alternative considered.++### `Release` Controller++We will implement the controller using the [managed reconciler of crossplane runtime](https://godoc.org/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed).++Following logic will be implemented for corresponding functions:++[Connect](https://godoc.org/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed#ExternalConnectorFn.Connect):++- Using provided Kubernetes Provider, create a kubernetes client and helm client as `ExternalClient`. ++[Observe](https://godoc.org/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed#ExternalClientFns.Observe):++1. Using [history action](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/action/history.go#L43)+of helm client, get last [release](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/release/release.go#L22)+information.+    1. If there is no last release, return [`ExternalObservation`](https://godoc.org/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed#ExternalObservation)+    as `ResourceExists = false` which will result in `Create` to be called.+    1. If there is last release, [decide whether desired state matches with observed or not](#triggering-a-helm-upgrade).+        1. If desired state matches with observed, return `ResourceUpToDate = true`+        1. If desired state differs from observed, return `ResourceUpToDate = false`++[Create](https://godoc.org/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed#ExternalClientFns.Create):++1. Pull and load the helm chart using [pull action](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/action/pull.go#L56).+1. Compose [value overrides](#value-overrides) as desired config.+1. Create the `spec.forProvider.namespace`, if not exists.+1. Using [install action](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/action/install.go#L150)+of helm client, `helm install` with the desired config.++[Update](https://godoc.org/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed#ExternalClientFns.Update):++1. Pull and load the helm chart using [pull action](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/action/pull.go#L56).+1. Prepare desired config by combining `spec.forProvider.values` and `spec.forProvider.set`s+1. Using [upgrade action](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/action/upgrade.go#L71) of+helm client, `helm upgrade` with the desired config.++[Delete](https://godoc.org/github.com/crossplane/crossplane-runtime/pkg/reconciler/managed#ExternalClientFns.Delete):++1. Using [uninstall action](https://github.com/helm/helm/blob/3d64c6bb5495d4e4426c27b181300fff45f95ff0/pkg/action/uninstall.go#L49) of+helm client, `helm uninstall` the release.+1. Once uninstall is successful, finalizer is removed (by crossplane-runtime).++#### Namespaced vs Cluster Scoped++Helm releases are namespaced and with Helm 3, Helm itself also keeping release information in the same namespace where+deployment is made. This was an improvement and actually a better fit to Kubernetes deployment model compared to keeping+all release information in a single namespace (`kube-system` by default) in Helm 2. If we were designing this helm+controller for in cluster deployment only (similar to existing alternatives), it would make super sense to make it+simply `Namespaced` and deploy helm release into that namespace. However, in our case, we want to primarily support+deploying to remote clusters. So we cannot use namespace of the custom resource to decide Helm deployment namespace. ++We have two options here:++1. `Cluster` scoped resource and `spec.namespace` for helm deployment resource.+1. `Namespaced` resource and an optional `spec.remoteNamespace`++Even option 2 makes more sense for local deployment case, we need to design for remote clusters first (e.g. crossplane +provisioned) and option 1 better fits current patterns in Crossplane for managed resources (i.e. all existing managed+resources are Cluster Scoped).++## Using in Composition++### Use Case Example 1: MonitoredCluster Composite Resource++As a platform builder, I would like to define MonitoredCluster as a Crossplane Composite Resource which is basically +a managed Kubernetes Cluster of my preferred Cloud + our Monitoring Stack configured properly.++### Use Case Example 2: WordpressCluster Composite Resource++As a platform builder, I would like to define WordpressCluster as a Crossplane Composite Resource which creates an+external database, a Kubernetes cluster, deploys Wordpress application.++We can use `Release` resource in a composition to provision a cluster created by Crossplane. Following example shows+how to define a composition for a Wordpress application deployed on a Kubernetes cluster and consuming a database+both created with Crossplane. ++```+apiVersion: apiextensions.crossplane.io/v1alpha1+kind: Composition+metadata:+  name: wordpressclusters.apps.example.org+spec:+  writeConnectionSecretsToNamespace: crossplane-system+  reclaimPolicy: Delete+  from:+    apiVersion: apps.example.org/v1alpha1+    kind: WordpressCluster+  to:+    - base:+        apiVersion: container.gcp.crossplane.io/v1beta1+        kind: GKECluster+        spec:+          providerRef:+            name: gcp-provider+          forProvider:+            addonsConfig:+              kubernetesDashboard:+                disabled: true+              networkPolicyConfig:+                disabled: true+            databaseEncryption:+              state: DECRYPTED+            defaultMaxPodsConstraint:+              maxPodsPerNode: 110+            description: Wordpress Cluster+            ipAllocationPolicy:+              createSubnetwork: true+              useIpAliases: true+            networkPolicy:+              enabled: false+            legacyAbac:+              enabled: false+            podSecurityPolicyConfig:+              enabled: false+            verticalPodAutoscaling:+              enabled: false+            masterAuth:+              username: admin+            loggingService: logging.googleapis.com/kubernetes+            monitoringService: monitoring.googleapis.com/kubernetes+            networkRef:+              name: gcp-wpclusters-network+            location: us-central1+            locations:+              - us-central1-a+          writeConnectionSecretToRef:+            namespace: crossplane-system+          reclaimPolicy: Delete+      patches:+        - fromFieldPath: "metadata.name"+          toFieldPath: "metadata.name"+        - fromFieldPath: "metadata.name"+          toFieldPath: "spec.writeConnectionSecretToRef.name"+      connectionDetails:+        - fromConnectionSecretKey: kubeconfig+    - base:+        apiVersion: database.gcp.crossplane.io/v1beta1+        kind: CloudSQLInstance+        spec:+          forProvider:+            databaseVersion: MYSQL_5_6+            region: us-central1+            settings:+              tier: db-custom-1-3840+              dataDiskType: PD_SSD+              ipConfiguration:+                ipv4Enabled: true+                authorizedNetworks:+                  - value: "0.0.0.0/0"+          writeConnectionSecretToRef:+            namespace: crossplane-system+          providerRef:+            name: gcp-provider+          reclaimPolicy: Delete+      patches:+        - fromFieldPath: "metadata.uid"+          toFieldPath: "spec.writeConnectionSecretToRef.name"+          transforms:+            - type: string+              string:+                fmt: "%s-mysql"+        - fromFieldPath: "spec.parameters.storageGB"+          toFieldPath: "spec.forProvider.settings.dataDiskSizeGb"+      connectionDetails:+        - fromConnectionSecretKey: username+        - fromConnectionSecretKey: password+        - fromConnectionSecretKey: endpoint+        - name: port+          value: "5432"+    - base:+        apiVersion: kubernetes.crossplane.io/v1alpha1+        kind: Provider+      patches:+        - fromFieldPath: "metadata.name"+          toFieldPath: "spec.credentialsSecretRef.name"       +    - base:+        apiVersion: helm.crossplane.io/v1alpha1+        kind: Release+        spec:+          forProvider:+            chart:+              name: wordpress+              repository: https://charts.bitnami.com/bitnami+            namespace: wordpress+            values: |+              mariadb.enabled: false+              externaldb.enabled: true+            set:+            - name: externalDatabase.host+              valueFrom:+                secretKeyRef:+                  key: host+            - name: externalDatabase.user+              valueFrom:+                secretKeyRef:+                  key: username+            - name: externalDatabase.password+              valueFrom:+                secretKeyRef:+                  key: password+            - name: blogName+          reclaimPolicy: Delete+          providerSelector:+            matchControllerRef: true

I think this is trying to select the Kubernetes provider created in the same composition, right? This functionality (selecting rather than referencing a provider) does not exist today. I'd support introducing it, but I'd prefer to do so for all providers rather than making a special case for the Helm provider.

turkenh

comment created time in 4 days

pull request commentcrossplane/crossplane

One-pager for Crossplane Helm Provider

should that helm+kustomize controller be still called as "helm" provider?

This is a large part of the reason I've been advocating for a provider that targets a layer above specific deployment tools like Helm or Kustomize. If we wrote a provider for a tool such as Rancher Fleet and ArgoCD this problem would be solved for us.

turkenh

comment created time in 4 days

push eventcrossplane/crossplane

Muvaffak Onus

commit sha 51736ddab1fb8f45eaac87d3f4fceff17868b952

requirement reconciler: propagate labels and annotations to composite resource Signed-off-by: Muvaffak Onus <onus.muvaffak@gmail.com>

view details

Nic Cope

commit sha 01a480dd173c189dfb0efae2c13c49e2507efa2c

Merge pull request #1664 from muvaf/prop-labels Propagate labels and annotations to composite resource

view details

push time in 5 days

PR merged crossplane/crossplane

Reviewers
Propagate labels and annotations to composite resource

<!-- Thank you for helping to improve Crossplane!

Please read through https://git.io/fj2m9 if this is your first time opening a Crossplane pull request. Find us in https://slack.crossplane.io/messages/dev if you need any help contributing. -->

Description of your changes

Propagates annotations and labels from *Requirement instance to corresponding composite resource.

Fixes https://github.com/crossplane/crossplane/issues/1654

<!-- Briefly describe what this pull request does. Be sure to direct your reviewers' attention to anything that needs special consideration.

We love pull requests that resolve an open Crossplane issue. If yours does, you can uncomment the below line to indicate which issue your PR fixes, for example "Fixes #500":

--> Fixes #

I have:

  • [x] Read and followed Crossplane's contribution process.
  • [x] Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

Manually.

<!-- Before reviewers can be confident in the correctness of this pull request, it needs to tested and shown to be correct. Briefly describe the testing that has already been done or which is planned for this change. -->

+10 -9

0 comment

1 changed file

muvaf

pr closed time in 5 days

issue closedcrossplane/crossplane

Metadata annotations fields and label should be propagated from requirement to managed resource object

What problem are you facing?

From requirement object : MySQLInstanceRequirement, i would like to pass the metadata annotations and labels information to the managed resource object : MySQLInstance

So that when the composition object could extract the metadata annotations or labels (other than external-name) to be map in the specification of the DB Object, this is useful when the annotation information is required as a Tag informations of the DBInstance that is being provisioned


apiVersion: apiextensions.crossplane.io/v1alpha1
kind: Composition
.
.
spec:
          forProvider:
            skipFinalSnapshotBeforeDeletion: true
            publiclyAccessible: false
            backupRetentionPeriod: 1
            copyTagsToSnapshot: true
            applyModificationsImmediately: true
            tags:
              - key: "Key0"
                value: ""
              - key: "Key1"
                value: ""
              - key: "Key2"
                value: ""
.
.
.
- fromFieldPath: "metadata.annotations.key0"
          toFieldPath: "spec.forProvider.tags[0].value"
        - fromFieldPath: "metadata.annotations.key1"
          toFieldPath: "spec.forProvider.tags[1].value"
        - fromFieldPath: "metadata.annotations.key2"
          toFieldPath: "spec.forProvider.tags[2].value"
.
.

How could Crossplane help solve your problem?

By allowing any metadata annotations and label objects to be mapped.

closed time in 5 days

kelvinwijaya

create barnchnegz/aac

branch : master

created branch time in 5 days

created repositorynegz/aac

Platform API as Configuration

created time in 5 days

pull request commentcrossplane/crossplane

[WIP] Package Manager refactor design doc

I'm going to be offline for two weeks, so I wanted to mention that I feel good about the direction this design is going.

I'm not going to formally approve just yet since I think the design might undergo a little more polish and scoping, but I'm definitely a fan of the Package and PackageRevision changes proposed in phase one. I like that PackageRevision is now completely responsible for running the packaged entity, and that there's an elegant handoff process between revisions. I also like the proposed phases; we have an idea of the direction we're going but that we can add value without tackling it all at once.

hasheddan

comment created time in 25 days

issue commentcrossplane/crossplane

Rename, remove, or rethink KubernetesApplication

@negz do you want to keep this issue open to track actual deprecation & removal?

Yes please. We should make sure to at least add comments to the CRDs letting folks know they'll be removed in a few releases.

negz

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a platform+team in an organization, you might want to publish a set of APIs that are+audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you would+also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's available as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster+    will be needed.++Note that there will be no controller reconciling `InfrastructurePublication`+and `InfrastructureDefinition` types to generate their corresponding CRDs; the+agent will blindly pull the resulting CRDs from the central cluster so that in+case of a version mismatch between the agent(s) and the Crossplane in the+central cluster, there won't be any schema difference. The source of truth for+all these listed resources except the `*Requirement`s is the central cluster.++Here is an illustration of how synchronization will work:++![Synchronization Flows][sync-diagram]++### RBAC++As we have two different Kubernetes API servers, there will be two separate+security domains and because of that, the `ServiceAccount`s in the remote+cluster will not be able to do any operation in the central cluster. Since the+entity that will execute the operations in the central cluster is the agent, we+need to define how we can deliver the necessary credentials to the agent so that+it can connect to the central cluster. Additionally, it will need some+permissions to execute operations in the remote cluster like `Secret` and+`CustomResourceDefinition` creation. We will look at how the agent will be+granted permissions to do its job in two separate domains with different+mechanisms.++#### Authenticating to The Central Cluster++In order to execute any operation, a `ServiceAccount` needs to exist in the+central cluster with appropriate permissions to read during pull and write+during push operations while synchronizing with central cluster. Since the agent+is running in the remote cluster, the credentials of this `ServiceAccount` will+be stored in a `Secret` in the remote cluster. Alongside the credentials, the+agent needs to know the namespace that it should sync to in the central cluster.++The easiest configuration would be the one where we specify the `Secret` and a+target namespace via installation commands of the agent. Both of these inputs+will act as default and they can be overriden for each `Namespace`+independently. For example, multiple namespaces can have different requirements+with the same name which could cause conficts in the central cluster because all+namespaces are rolled up into one namespace. In order to prevent conflicts, the+agent will annotate the requirements in the central cluster with the UID of the+namespace in the remote cluster and do the necessary checks to avoid conflicts.++An illustrative installation command:+```bash+helm install crossplane/agent \+  --set default-credentials-secret-name=my-sa-creds \+  --set default-target-namespace=ns-in-central+```++While this installation time configuration provides a simple setup, it comes+with the restriction that you cannot have the requirements in different+namespaces with the same name. You can either try using different names or you+can specify which namespace in the remote cluster should be synced to which+namespace in the central cluster. In the remote cluster, `Namespace` can be+annotated as such:++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar+```++The agent then will try to sync the requirements in `foo` namespace of the+remote cluster into `bar` namespace of the central cluster instead of the+default target namespace which is `ns-in-central` in this example. But it will+keep using the default credentials `Secret`. In case you don't want different+namespaces to share the same credentials `Secret`, then you can override this+setting, too:++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar+    "agent.crossplane.io/credentials-secret-name": my-other-sa-creds+---+apiVersion: v1+kind: Secret+metadata:+  name: my-sa-creds+  namespace: foo+type: Opaque+data:+  kubeconfig: MWYyZDFlMmU2N2Rm...+```++Now, all the requirements in the `foo` namespace of the remote cluster will be+synced to `bar` namespace of the central cluster and all of the operations will+be done using the credentials in `my-other-sa-creds`.++There is also the option to automate the namespace pairings in a way that if+requirement `A` is deployed in namespace `foo` of the remote cluster, then it+gets synced to namespace `foo` of the central cluster. You can enable this+automation using a flag. Helm command would look like:+```bash+helm install crossplane/agent \+  --set default-credentials-secret-name=my-sa-creds \+  --set match-namespaces=true+```++As with all cases, you can override specific pairings via annotations on the+namespaces of the remote cluster.++#### Conflicts++The agent does not allow any conflicts to happen, meaning if two requirements+created with the same name by different agents in some namespace in the pool of+clusters, then it will reject syncing it to the central cluster. In order to do+that, there needs to be unique identifier of the source of the requirement.++Let's go over the different setups and consider conflict cases:+* Each namespace is annotated with the target namespace.+  * No conflict for single remote cluster.+  * Could conflict if more than one remote cluster is connected and its namespaces have+    common annotations with other namespaces of other remote clusters.+* Each namespace targets a namespace with the same name in the central cluster.+  * No conflict for single remote cluster.+  * Could conflict if more than one cluster is connected and its namespaces have+    the same names.+* All namespaces roll up to a default namespace.+  * Could conflict for single remote cluster if different namespaces have+    requirements with same name.+  * Could conflict if more than one cluster is connected to the same namespace+    but it's less likely to be in that situation since the default namespace+    selection is done explicitly during agent setup, probably by an admin.++In order to prevent conflicts, the agent will add two annotations to the+requirements it syncs:+* `agent.crossplane.io/source-namespace`: The name of the namespace in the+  remote cluster.+* `agent.crossplane.io/source-uid`: The unique identifier of the remote

source-cluster-uid? source-uid sounds like it could mean the UID of the remote requirement.

muvaf

comment created time in a month

pull request commentcrossplane/crossplane

Crossplane Agent design doc

There could be 2 possibilities there; you have 1 cluster with everything in it and then remote clusters start to connect with an agent or you decide that the initial cluster should be a remote cluster. The former is possible as is but the latter has some caveats although I can see that one being more popular since the initial cluster would tend to be a beefy one while IMO central clusters will usually be rather small.

@muvaf So far we've only discussed the latter use case. I agree it's likely to be more popular and I think it's the only one we need to support, at least to begin with.

I would like to open a separate issue for migration topic and come back to it after the MVP.

I'd prefer there to be some mention of this in this design, even if it's just a 'future considerations' section. I'm okay deferring the full design for the migration path, but I want to make sure we're building the agent in a way that we expect to support really smooth migrations.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a+platform team in an organization, you might want to publish a set of APIs that+are audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you+would also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.

Got it. I guess RBAC could be used as well to ensure that no one has access to update them in the first place.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a+platform team in an organization, you might want to publish a set of APIs that+are audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you+would also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's available as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster+    will be needed.++Here is an illustration of how synchronization will work:++![Synchronization Flows][sync-diagram]++### RBAC++As we have two different Kubernetes API servers, there will be two separate security+domains and because of that, the `ServiceAccount`s in the remote cluster will+not be able to do any operation in the central cluster. Since the entity that+will execute the operations in the central cluster is the agent, we need to+define how we can deliver the necessary credentials to the agent so that it can+connect to the central cluster. Additionally, it will need some permissions to+execute operations in the remote cluster like `Secret` and+`CustomResourceDefinition` creation. We will look at how the agent will be+granted permissions to do its job in two separate domains with different+mechanisms.++#### Authenticating to The Central Cluster++In order to execute any operation, a `ServiceAccount` needs to exist in the+central cluster with appropriate permissions to read during pull and write+during push operations while synchronizing with central cluster. Since the agent+is running in the remote cluster, the credentials of this `ServiceAccount` will+be stored in a `Secret` in the remote cluster. Alongside the credentials, the+agent needs to know the namespace that it should sync to in the central cluster.++The easiest configuration would be the one where we specify the `Secret` and a+target namespace via installation commands of the agent. Both of these inputs+will act as default and they can be overriden for each `Namespace`+independently. For example, multiple namespaces can have different requirements+with the same name which could cause conficts in the central cluster because all+namespaces are rolled up into one namespace. In order to prevent conflicts, the+agent will annotate the requirements in the central cluster with the UID of the+namespace in the remote cluster and do the necessary checks to avoid conflicts.++An illustrative installation command:+```bash+helm install crossplane/agent \+  --set default-credentials-secret-name=my-sa-creds \+  --set default-target-namespace=ns-in-central+```++While this installation time configuration provides a simple setup, it comes+with the restriction that you cannot have the requirements in different+namespaces with the same name. You can either try using different names or you+can specify which namespace in the remote cluster should be synced to which+namespace in the central cluster. In the remote cluster, `Namespace` can be+annotated as such:++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar+```++The agent then will try to sync the requirements in `foo` namespace of the+remote cluster into `bar` namespace of the central cluster instead of the+default target namespace which is `ns-in-central` in this example. But it will+keep using the default credentials `Secret`. In case you don't want different+namespaces to share the same credentials `Secret`, then you can override this+setting, too:++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar+    "agent.crossplane.io/credentials-secret-name": my-other-sa-creds+---+apiVersion: v1+kind: Secret+metadata:+  name: my-sa-creds+  namespace: foo+type: Opaque+data:+  kubeconfig: MWYyZDFlMmU2N2Rm...+```++Now, all the requirements in the `foo` namespace of the remote cluster will be+synced to `bar` namespace of the central cluster and all of the operations will+be done using the credentials in `my-other-sa-creds`.++#### Authorization++##### Remote Cluster++Since there will be one agent for the whole cluster, its own mounted+`ServiceAccount` in that remote cluster needs to have read & write permissions+for all of the following kinds in the remote cluster listed below:++* `CustomResourceDefinition`+* `Composition`+* `InfrastructureDefinition`+* `InfrastructurePublication`+* `Secret`+* All `*Requirement` types++The last one is a bit tricky because the exact list of `*Requirement` types on+kind level is not known during installation and it's not static; new published+APIs should be available in the remote cluster dynamically. One option is to+allow agent to grant `Role` and `RoleBinding`s to itself as it creates the+necessary `CustomResourceDefinition`s in the remote cluster. However, an entity+that is able to grant permissions to itself could greatly increase the security+posture.++When you zoom out and think about how the `Role` will look like, in most of the+cases, it's something like the following:++```yaml+apiVersion: rbac.authorization.k8s.io/v1+kind: Role+metadata:+  name: crossplane-agent+  namespace: default+rules:+  # all kinds could be under one company/organization group+- apiGroups: ["acmeco.org"] +  resources: ["*"]+  verbs: ["*"]+  # or there could be logical groupings for different sets of requirements+- apiGroups: ["database.acmeco.org"]+  resources: ["*"]+  verbs: ["*"]+```++As you can see, it's either one group for all the new APIs or a logical group+for each set of APIs. In both cases, the frequency of the need to add a new+`apiGroup` is less than one would imagine thanks to the ability of allowing a+whole group; most frequently, the platform operators will be adding new kinds to+the existing groups.++In the light of this assumption, the initial approach will be that the `Role`+bound to the agent will be populated by a static list of the groups of the+requirement types during the installation like shown above and if a new group is+introduced, then an addition to this `Role` will be needed. A separate+controller to dynamically manage the `Role` is mentioned in the [Future+Considerations](#future-considerations) section.++##### Central Cluster++The `ServiceAccount` that will be created in the central cluster needs to have+the following permissions for agent to do its operations:++* Read+  * `CustomResourceDefinition`s+  * `InfrastructureDefinition`s+  * `InfrastructurePublication`s+  * `Composition`s+  * `Secret`s in the given namespace.+* Write+  * `*Requirement` types that you'd like to allow in given namespace.++### User Experience++In this section, a walkthrough from only a central cluster to a working+application will be shown step by step to show how the user experience will+shape.++Setup:+  * The Central Cluster: A Kubernetes cluster with Crossplane deployed &+    configured with providers and some published APIs.++Steps in the central cluster:+1. A new `Namespace` called `bar` is created.+1. A `ServiceAccount` called `agent1` in that namespace are created and+   necessary RBAC resources are created.++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: bar+---+# The ServiceAccount whose credentials will be copied over to remote cluster+# for agent to use to connect to the central cluster.+apiVersion: v1+kind: ServiceAccount+metadata:+  name: agent1+  namespace: bar+---+# To be able to create & delete requirements in the designated namespace of+# the central cluster.+apiVersion: rbac.authorization.k8s.io/v1+kind: Role+metadata:+  name: agent-requirement+  namespace: bar+rules:+- apiGroups: [""]+  resources: ["secrets"]+  verbs: ["*"]+- apiGroups: ["database.acmeco.org"]+  resources: ["*"]+  verbs: ["*"]+- apiGroups: ["network.acmeco.org"]+  resources: ["*"]+  verbs: ["*"]+---+apiVersion: rbac.authorization.k8s.io/v1+kind: RoleBinding+metadata:+  name: agent-requirement+  namespace: bar+subjects:+- kind: ServiceAccount+  name: agent1+  apiGroup: rbac.authorization.k8s.io+roleRef:+  kind: Role+  name: agent-requirement+  apiGroup: rbac.authorization.k8s.io+```++The YAML above includes what's necessary to sync a specific namespace. The YAML+below is for cluster-scoped resources that should be read by the agent and it's+generic to be used by all agents except the `subjects` list of+`ClusterRoleBinding`:++```yaml+# To be able to read the cluster-scoped resources.+apiVersion: rbac.authorization.k8s.io/v1+kind: ClusterRole+metadata:+  name: read-for-agents+rules:+- apiGroups: ["apiextensions.kubernetes.io"]+  resources: ["customresourcedefinitions"]+  verbs: ["get", "watch", "list"]+- apiGroups: ["apiextensions.crossplane.io"]+  resources:+  - infrastructuredefinitions+  - infrastructurepublications+  - compositions+  verbs: ["get", "watch", "list"]+---+apiVersion: rbac.authorization.k8s.io/v1+kind: ClusterRoleBinding+metadata:+  name: read-for-agents+subjects:+- kind: ServiceAccount+  name: agent1+  apiGroup: rbac.authorization.k8s.io+roleRef:+  kind: ClusterRole+  name: read-for-agents+  apiGroup: rbac.authorization.k8s.io+```++At this point, we have a `ServiceAccount` with all necessary permissions in our+central cluster. You can think of it like IAM user in the public cloud+providers; we have created it and allowed it to access a certain set of APIs.+Later on, its key will be used by the agent; just like provider-aws using the+key of an IAM User.++1. User provisions a network stack and a Kubernetes cluster in that private+   network through Crossplane (or via other methods, even kind cluster would+   work). This cluster will be used as remote cluster. We'll run the rest of the+   steps in that remote cluster.+1. The `Secret` that contains the kubeconfig of the `ServiceAccount` we created+   in the central cluster is replicated in the remote cluster with name+   `agent1-creds`.+1. The agent is installed into the remote cluster. The Helm package will have+   all the necessary RBAC resources but user will need to enter the API groups+   of the published types so that it can reconcile them in the remote cluster.+   ```bash+   helm install crossplane/agent \+   --set apiGroups=database.acmeco.org,network.acmeco.org \+   --set default-credentials-secret-name=agent1-creds \+   --set default-target-namespace=bar+   ```++At this point, the setup has been completed. Now, users can use the APIs in the+remote cluster just as if they are in the local mode. An example application to+deploy:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The `MySQLInstanceRequirement` will be synced to `bar` namespace in the central+cluster. In case there are other `MySQLInstanceRequirement` custom resources in+the remote cluster with same name but in different namespaces, then the agent+will reject syncing that in order to prevent the conflict.+++Note that these steps show the bare-bones setup. Most of the steps can be made+easier with simple commands in Crossplane CLI and YAML templates you can edit &+use.++Users can discover what resources available to them and how they+can consume them from their remote cluster.++List the published APIs:+```bash+kubectl get infrastructurepublications+```++See what keys are included in the connection `Secret` of a specific API so that+they know what keys to use in mounting process:+```+kubectl get infrastructuredefinition mysqlinstance.database.acmeco.org -o yaml+```++See what `Composition`s are available to select from:+```+kubectl get compositions+```++## Future Considerations++### Additional Crossplane CLI Commands++Crossplane CLI can have simple commands to do most of the setup. For example,+with one command it should be possible to create the `ServiceAccount` in the+central cluster together with all of its RBAC resources. Also, agent+installation could be a very smooth process if we use Crossplane CLI instead of+Helm.++### RBAC Controller++A controller with its own types to manage the `ServiceAccount`s in the central+cluster could be a boost to UX. You'd create a custom resource that will result+in all RBAC resources that are needed for the agent to work with all the APIs in+the central cluster and write the credentials to a secret. Then the user can+copy this secret into their remote cluster and refer to it.++### Dynamic Updates to Agent's Role++In the remote cluster, we provide the `Role` that has the static set of+`apiGroup`s we'd like agent to be able to manage in the remote cluster. There+could be a controller that is separately installed and it could add new+`apiGroup`s as they appear as `InfrastructurePublication`s in the remote+cluster.

Oh - I missed that this was granting access to the agent itself.

muvaf

comment created time in a month

push eventcrossplane/provider-aws

Sahil Lakhwani

commit sha f56e42d8930076b1e4c3f0b5310286665ff3823f

Add ELB resource (#262) * Add ELB resource Signed-off-by: sahil-lakhwani <sahilakhwani@gmail.com>

view details

push time in a month

PR merged crossplane/provider-aws

Add ELB resource

Signed-off-by: sahil-lakhwani sahilakhwani@gmail.com

Description of your changes

AWS offers 3 different types of load balancers.

This PR adds a one of them - Classic Load Balancer

Classic Load Balancer is commonly called as ELB hence the name of the custom resource is also ELB. This PR also adds a resource ELBAttachment which manages the attachment of EC2 instances to a Classic Load Balancer.

Example manifest of both the resources can be found under examples/elasticloadbalancing/

Checklist

<!-- Please run through the below readiness checklist. The first two items are relevant to every Crossplane pull request. --> I have:

  • [x] Run make reviewable to ensure this PR is ready for review.
  • [x] Ensured this PR contains a neat, self documenting set of commits.
  • [ ] Updated any relevant documentation, examples, or release notes.
  • [ ] Updated the dependencies in app.yaml to include any new role permissions.
+4507 -0

5 comments

21 changed files

sahil-lakhwani

pr closed time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a+platform team in an organization, you might want to publish a set of APIs that+are audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you+would also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster+    will be needed.++### RBAC++As we have two different Kubernetes clusters, there will be two separate security+domains and because of that, the `ServiceAccount`s in the remote cluster will+not be able to do any operation in the central cluster. Since the entity that+will execute the operations in the central cluster is the agent, we need to+define how we can deliver the necessary credentials to the agent so that it can+connect to the central cluster. Additionally, it will need some permissions to+execute operations in the remote cluster like `Secret` and+`CustomResourceDefinition` creation. We will look at how the agent will be+granted permissions to do its job in two separate domains with different+mechanisms.++#### Authenticating to The Central Cluster++In order to execute any operation, a `ServiceAccount` needs to exist in the+central cluster with appropriate permissions to read during pull and write+during push operations while synchronizing with central cluster. Since the agent+is running in the remote cluster, the credentials of this `ServiceAccount` will+be stored in a `Secret` in the remote cluster. Alongside the credentials, the+agent needs to know the namespace that it should sync to in the central cluster.++The easiest configuration would be the one where we specify the `Secret` and a+target namespace via installation commands of the agent. However, the namespace+pairing list could change dynamically after the installation. So, we need to be+able to supply the pairings after the installation. In order to specify which+namespace in the remote cluster should be synced to which namespace in the+central cluster, `Namespace` resource needs to be annotated.++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar

the uid of a namespace isn't durable

Agreed. I suspect we won't be able to use the UID of kube-system as this would prevent us from replacing a remote cluster that was completely destroyed and recreated. I suspect assigning each agent/remote cluster a UID manually would be the better path.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a+platform team in an organization, you might want to publish a set of APIs that+are audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you+would also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.

Could you add a little context somewhere in this doc that explains why we'd pull the require CRDs from the central cluster rather than mirroring the logic that generates a CRD from an InfraPub in the agent? I'm not entirely convinced that it wouldn't be more flexible for the agent to mirror the logic (e.g. if we wanted the CRD gen process to diverge slightly at some point in future).

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a+platform team in an organization, you might want to publish a set of APIs that+are audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you+would also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's available as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster+    will be needed.++Here is an illustration of how synchronization will work:++![Synchronization Flows][sync-diagram]++### RBAC++As we have two different Kubernetes API servers, there will be two separate security+domains and because of that, the `ServiceAccount`s in the remote cluster will+not be able to do any operation in the central cluster. Since the entity that+will execute the operations in the central cluster is the agent, we need to+define how we can deliver the necessary credentials to the agent so that it can+connect to the central cluster. Additionally, it will need some permissions to+execute operations in the remote cluster like `Secret` and+`CustomResourceDefinition` creation. We will look at how the agent will be+granted permissions to do its job in two separate domains with different+mechanisms.++#### Authenticating to The Central Cluster++In order to execute any operation, a `ServiceAccount` needs to exist in the+central cluster with appropriate permissions to read during pull and write+during push operations while synchronizing with central cluster. Since the agent+is running in the remote cluster, the credentials of this `ServiceAccount` will+be stored in a `Secret` in the remote cluster. Alongside the credentials, the+agent needs to know the namespace that it should sync to in the central cluster.++The easiest configuration would be the one where we specify the `Secret` and a+target namespace via installation commands of the agent. Both of these inputs+will act as default and they can be overriden for each `Namespace`+independently. For example, multiple namespaces can have different requirements+with the same name which could cause conficts in the central cluster because all+namespaces are rolled up into one namespace. In order to prevent conflicts, the+agent will annotate the requirements in the central cluster with the UID of the+namespace in the remote cluster and do the necessary checks to avoid conflicts.++An illustrative installation command:+```bash+helm install crossplane/agent \+  --set default-credentials-secret-name=my-sa-creds \+  --set default-target-namespace=ns-in-central+```++While this installation time configuration provides a simple setup, it comes+with the restriction that you cannot have the requirements in different+namespaces with the same name. You can either try using different names or you+can specify which namespace in the remote cluster should be synced to which+namespace in the central cluster. In the remote cluster, `Namespace` can be+annotated as such:++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar+```++The agent then will try to sync the requirements in `foo` namespace of the+remote cluster into `bar` namespace of the central cluster instead of the+default target namespace which is `ns-in-central` in this example. But it will+keep using the default credentials `Secret`. In case you don't want different+namespaces to share the same credentials `Secret`, then you can override this+setting, too:++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar+    "agent.crossplane.io/credentials-secret-name": my-other-sa-creds+---+apiVersion: v1+kind: Secret+metadata:+  name: my-sa-creds+  namespace: foo+type: Opaque+data:+  kubeconfig: MWYyZDFlMmU2N2Rm...+```++Now, all the requirements in the `foo` namespace of the remote cluster will be+synced to `bar` namespace of the central cluster and all of the operations will+be done using the credentials in `my-other-sa-creds`.++#### Authorization++##### Remote Cluster++Since there will be one agent for the whole cluster, its own mounted+`ServiceAccount` in that remote cluster needs to have read & write permissions+for all of the following kinds in the remote cluster listed below:++* `CustomResourceDefinition`+* `Composition`+* `InfrastructureDefinition`+* `InfrastructurePublication`+* `Secret`+* All `*Requirement` types++The last one is a bit tricky because the exact list of `*Requirement` types on+kind level is not known during installation and it's not static; new published+APIs should be available in the remote cluster dynamically. One option is to+allow agent to grant `Role` and `RoleBinding`s to itself as it creates the+necessary `CustomResourceDefinition`s in the remote cluster. However, an entity+that is able to grant permissions to itself could greatly increase the security+posture.++When you zoom out and think about how the `Role` will look like, in most of the+cases, it's something like the following:++```yaml+apiVersion: rbac.authorization.k8s.io/v1+kind: Role+metadata:+  name: crossplane-agent+  namespace: default+rules:+  # all kinds could be under one company/organization group+- apiGroups: ["acmeco.org"] +  resources: ["*"]+  verbs: ["*"]+  # or there could be logical groupings for different sets of requirements+- apiGroups: ["database.acmeco.org"]+  resources: ["*"]+  verbs: ["*"]+```++As you can see, it's either one group for all the new APIs or a logical group+for each set of APIs. In both cases, the frequency of the need to add a new+`apiGroup` is less than one would imagine thanks to the ability of allowing a+whole group; most frequently, the platform operators will be adding new kinds to+the existing groups.++In the light of this assumption, the initial approach will be that the `Role`+bound to the agent will be populated by a static list of the groups of the+requirement types during the installation like shown above and if a new group is+introduced, then an addition to this `Role` will be needed. A separate+controller to dynamically manage the `Role` is mentioned in the [Future+Considerations](#future-considerations) section.++##### Central Cluster++The `ServiceAccount` that will be created in the central cluster needs to have+the following permissions for agent to do its operations:++* Read+  * `CustomResourceDefinition`s+  * `InfrastructureDefinition`s+  * `InfrastructurePublication`s+  * `Composition`s+  * `Secret`s in the given namespace.+* Write+  * `*Requirement` types that you'd like to allow in given namespace.++### User Experience++In this section, a walkthrough from only a central cluster to a working+application will be shown step by step to show how the user experience will+shape.++Setup:+  * The Central Cluster: A Kubernetes cluster with Crossplane deployed &+    configured with providers and some published APIs.++Steps in the central cluster:+1. A new `Namespace` called `bar` is created.+1. A `ServiceAccount` called `agent1` in that namespace are created and+   necessary RBAC resources are created.++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: bar+---+# The ServiceAccount whose credentials will be copied over to remote cluster+# for agent to use to connect to the central cluster.+apiVersion: v1+kind: ServiceAccount+metadata:+  name: agent1+  namespace: bar+---+# To be able to create & delete requirements in the designated namespace of+# the central cluster.+apiVersion: rbac.authorization.k8s.io/v1+kind: Role+metadata:+  name: agent-requirement+  namespace: bar+rules:+- apiGroups: [""]+  resources: ["secrets"]+  verbs: ["*"]+- apiGroups: ["database.acmeco.org"]+  resources: ["*"]+  verbs: ["*"]+- apiGroups: ["network.acmeco.org"]+  resources: ["*"]+  verbs: ["*"]+---+apiVersion: rbac.authorization.k8s.io/v1+kind: RoleBinding+metadata:+  name: agent-requirement+  namespace: bar+subjects:+- kind: ServiceAccount+  name: agent1+  apiGroup: rbac.authorization.k8s.io+roleRef:+  kind: Role+  name: agent-requirement+  apiGroup: rbac.authorization.k8s.io+```++The YAML above includes what's necessary to sync a specific namespace. The YAML+below is for cluster-scoped resources that should be read by the agent and it's+generic to be used by all agents except the `subjects` list of+`ClusterRoleBinding`:++```yaml+# To be able to read the cluster-scoped resources.+apiVersion: rbac.authorization.k8s.io/v1+kind: ClusterRole+metadata:+  name: read-for-agents+rules:+- apiGroups: ["apiextensions.kubernetes.io"]+  resources: ["customresourcedefinitions"]+  verbs: ["get", "watch", "list"]+- apiGroups: ["apiextensions.crossplane.io"]+  resources:+  - infrastructuredefinitions+  - infrastructurepublications+  - compositions+  verbs: ["get", "watch", "list"]+---+apiVersion: rbac.authorization.k8s.io/v1+kind: ClusterRoleBinding+metadata:+  name: read-for-agents+subjects:+- kind: ServiceAccount+  name: agent1+  apiGroup: rbac.authorization.k8s.io+roleRef:+  kind: ClusterRole+  name: read-for-agents+  apiGroup: rbac.authorization.k8s.io+```++At this point, we have a `ServiceAccount` with all necessary permissions in our+central cluster. You can think of it like IAM user in the public cloud+providers; we have created it and allowed it to access a certain set of APIs.+Later on, its key will be used by the agent; just like provider-aws using the+key of an IAM User.++1. User provisions a network stack and a Kubernetes cluster in that private+   network through Crossplane (or via other methods, even kind cluster would+   work). This cluster will be used as remote cluster. We'll run the rest of the+   steps in that remote cluster.+1. The `Secret` that contains the kubeconfig of the `ServiceAccount` we created+   in the central cluster is replicated in the remote cluster with name+   `agent1-creds`.+1. The agent is installed into the remote cluster. The Helm package will have+   all the necessary RBAC resources but user will need to enter the API groups+   of the published types so that it can reconcile them in the remote cluster.+   ```bash+   helm install crossplane/agent \+   --set apiGroups=database.acmeco.org,network.acmeco.org \+   --set default-credentials-secret-name=agent1-creds \+   --set default-target-namespace=bar+   ```++At this point, the setup has been completed. Now, users can use the APIs in the+remote cluster just as if they are in the local mode. An example application to+deploy:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The `MySQLInstanceRequirement` will be synced to `bar` namespace in the central+cluster. In case there are other `MySQLInstanceRequirement` custom resources in+the remote cluster with same name but in different namespaces, then the agent+will reject syncing that in order to prevent the conflict.+++Note that these steps show the bare-bones setup. Most of the steps can be made+easier with simple commands in Crossplane CLI and YAML templates you can edit &+use.++Users can discover what resources available to them and how they+can consume them from their remote cluster.++List the published APIs:+```bash+kubectl get infrastructurepublications+```++See what keys are included in the connection `Secret` of a specific API so that+they know what keys to use in mounting process:+```+kubectl get infrastructuredefinition mysqlinstance.database.acmeco.org -o yaml+```++See what `Composition`s are available to select from:+```+kubectl get compositions+```++## Future Considerations++### Additional Crossplane CLI Commands++Crossplane CLI can have simple commands to do most of the setup. For example,+with one command it should be possible to create the `ServiceAccount` in the+central cluster together with all of its RBAC resources. Also, agent+installation could be a very smooth process if we use Crossplane CLI instead of+Helm.++### RBAC Controller++A controller with its own types to manage the `ServiceAccount`s in the central+cluster could be a boost to UX. You'd create a custom resource that will result+in all RBAC resources that are needed for the agent to work with all the APIs in+the central cluster and write the credentials to a secret. Then the user can+copy this secret into their remote cluster and refer to it.++### Dynamic Updates to Agent's Role++In the remote cluster, we provide the `Role` that has the static set of+`apiGroup`s we'd like agent to be able to manage in the remote cluster. There+could be a controller that is separately installed and it could add new+`apiGroup`s as they appear as `InfrastructurePublication`s in the remote+cluster.

Would this be something like https://github.com/crossplane/crossplane/issues/1637, but at the remote cluster?

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a+platform team in an organization, you might want to publish a set of APIs that+are audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you+would also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.

How should we enforce the read-only nature of Compositions, InfraDefs, and InfraPubs?

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a+platform team in an organization, you might want to publish a set of APIs that+are audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you+would also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's available as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster+    will be needed.++Here is an illustration of how synchronization will work:++![Synchronization Flows][sync-diagram]++### RBAC++As we have two different Kubernetes API servers, there will be two separate security+domains and because of that, the `ServiceAccount`s in the remote cluster will+not be able to do any operation in the central cluster. Since the entity that+will execute the operations in the central cluster is the agent, we need to+define how we can deliver the necessary credentials to the agent so that it can+connect to the central cluster. Additionally, it will need some permissions to+execute operations in the remote cluster like `Secret` and+`CustomResourceDefinition` creation. We will look at how the agent will be+granted permissions to do its job in two separate domains with different+mechanisms.++#### Authenticating to The Central Cluster++In order to execute any operation, a `ServiceAccount` needs to exist in the+central cluster with appropriate permissions to read during pull and write+during push operations while synchronizing with central cluster. Since the agent+is running in the remote cluster, the credentials of this `ServiceAccount` will+be stored in a `Secret` in the remote cluster. Alongside the credentials, the+agent needs to know the namespace that it should sync to in the central cluster.++The easiest configuration would be the one where we specify the `Secret` and a+target namespace via installation commands of the agent. Both of these inputs+will act as default and they can be overriden for each `Namespace`+independently. For example, multiple namespaces can have different requirements+with the same name which could cause conficts in the central cluster because all+namespaces are rolled up into one namespace. In order to prevent conflicts, the+agent will annotate the requirements in the central cluster with the UID of the+namespace in the remote cluster and do the necessary checks to avoid conflicts.++An illustrative installation command:+```bash+helm install crossplane/agent \+  --set default-credentials-secret-name=my-sa-creds \+  --set default-target-namespace=ns-in-central+```++While this installation time configuration provides a simple setup, it comes+with the restriction that you cannot have the requirements in different+namespaces with the same name. You can either try using different names or you+can specify which namespace in the remote cluster should be synced to which+namespace in the central cluster. In the remote cluster, `Namespace` can be+annotated as such:++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar+```++The agent then will try to sync the requirements in `foo` namespace of the+remote cluster into `bar` namespace of the central cluster instead of the+default target namespace which is `ns-in-central` in this example. But it will+keep using the default credentials `Secret`. In case you don't want different+namespaces to share the same credentials `Secret`, then you can override this+setting, too:++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar+    "agent.crossplane.io/credentials-secret-name": my-other-sa-creds+---+apiVersion: v1+kind: Secret+metadata:+  name: my-sa-creds+  namespace: foo+type: Opaque+data:+  kubeconfig: MWYyZDFlMmU2N2Rm...+```++Now, all the requirements in the `foo` namespace of the remote cluster will be+synced to `bar` namespace of the central cluster and all of the operations will+be done using the credentials in `my-other-sa-creds`.++#### Authorization++##### Remote Cluster++Since there will be one agent for the whole cluster, its own mounted+`ServiceAccount` in that remote cluster needs to have read & write permissions+for all of the following kinds in the remote cluster listed below:++* `CustomResourceDefinition`+* `Composition`+* `InfrastructureDefinition`+* `InfrastructurePublication`+* `Secret`+* All `*Requirement` types++The last one is a bit tricky because the exact list of `*Requirement` types on+kind level is not known during installation and it's not static; new published+APIs should be available in the remote cluster dynamically. One option is to+allow agent to grant `Role` and `RoleBinding`s to itself as it creates the+necessary `CustomResourceDefinition`s in the remote cluster. However, an entity+that is able to grant permissions to itself could greatly increase the security+posture.

Nit: "increase the security posture" reads to me like "makes it more secure", whereas I think this means "makes it less secure". Perhaps "impact the security posture" would better imply the latter?

muvaf

comment created time in a month

Pull request review commentcrossplane/oam-kubernetes-runtime

[WIP] Add repo machinery to allow oam-kubernetes-runtime to publish artifacts using Jenkins

 PLATFORMS ?= linux_amd64 linux_arm64 -include build/makelib/common.mk  # ====================================================================================-# Setup Images+# Setup Output -# even though this repo doesn't build images (note the no-op img.build target below),-# some of the init is needed for the cross build container, e.g. setting BUILD_REGISTRY--include build/makelib/image.mk-img.build:+# S3_BUCKET ?= crossplane.releases

I believe this is where the Helm chart lives. I presume the image would live at https://hub.docker.com/u/crossplane.

hasheddan

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a+platform team in an organization, you might want to publish a set of APIs that+are audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you+would also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster+    will be needed.++### RBAC++As we have two different Kubernetes clusters, there will be two separate security+domains and because of that, the `ServiceAccount`s in the remote cluster will+not be able to do any operation in the central cluster. Since the entity that+will execute the operations in the central cluster is the agent, we need to+define how we can deliver the necessary credentials to the agent so that it can+connect to the central cluster. Additionally, it will need some permissions to+execute operations in the remote cluster like `Secret` and+`CustomResourceDefinition` creation. We will look at how the agent will be+granted permissions to do its job in two separate domains with different+mechanisms.++#### Authenticating to The Central Cluster++In order to execute any operation, a `ServiceAccount` needs to exist in the+central cluster with appropriate permissions to read during pull and write+during push operations while synchronizing with central cluster. Since the agent+is running in the remote cluster, the credentials of this `ServiceAccount` will+be stored in a `Secret` in the remote cluster. Alongside the credentials, the+agent needs to know the namespace that it should sync to in the central cluster.++The easiest configuration would be the one where we specify the `Secret` and a+target namespace via installation commands of the agent. However, the namespace+pairing list could change dynamically after the installation. So, we need to be+able to supply the pairings after the installation. In order to specify which+namespace in the remote cluster should be synced to which namespace in the+central cluster, `Namespace` resource needs to be annotated.++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar

Let the admin of the remote cluster choose one namespace to be the default

A default that can be overridden at the namespace level seems reasonable. This would promote a pattern in which all namespaces of a remote cluster map to one namespace in the central cluster by default, but enable other patterns with administrator intervention.

It does seem like we could also support the one-to-one mapping by default if we wanted to though - perhaps this could be enabled as part of the agent's config? i.e. Whether to map every remote namespace to one central namespace if there was no annotation, or to map every remote namespace to the central namespace with the same name if there was no annotation. I would suggest that even if we choose not to support the latter case we should structure whatever flags/config file controls this such that we could add support for both if there was user demand.

I still couldn't convince myself for this

Fair enough - I'm happy to go with identical names that will be rejected if they happen to conflict.

muvaf

comment created time in a month

issue openedcrossplane/crossplane

Composed resources should respect external name annotation by default

<!-- Thank you for helping to improve Crossplane!

Please be sure to search for open issues before raising a new one. We use issues for bug reports and feature requests. Please find us at https://slack.crossplane.io for questions, support, and discussion. -->

What problem are you facing?

<!-- Please tell us a little about your use case - it's okay if it's hypothetical! Leading with this context helps frame the feature request so we can ensure we implement it sensibly. ---> Crossplane resources support an annotation - crossplane.io/external-name - that may dictate or reflect the 'external' name of a resource - i.e. the unique identifier of that resource an the external system such as a cloud provider that Crossplane is orchestrating. This annotation works roughly like:

  • If the external system generates its own unique ID, we set the annotation to that ID.
  • If the external system allows the caller to specify a unique ID and the managed resource author...
    • specifies the annotation, we use its value as the unique ID.
    • omits the annotation, we set it to the managed resource's metadata.name.

When resource claims are being used this annotation is propagated between the claim and the managed resource it's bound to. This allows the resource claim author to influence the external name (aka unique ID) of the external resource by setting the annotation on their resource claim.

When composition is being used the annotation is propagated from the requirement (if any) to the composite resource. Whether the annotation is propagated from the composite resource to the composed resource to the composed resources is determined by the author of the Composition. If the author does not explicitly configure patches from the composite resource's annotation to the composed resources' annotations, the annotation will be non-functional. I believe this behaviour is likely to surprise users of Crossplane. crossplane.io/external-name sounds like a standard annotation that will be honored, but whether or not it has meaning above the managed resource level is likely to be highly inconsistent given that it's dependent on humans remembering to write configuration that propagates it.

How could Crossplane help solve your problem?

<!-- Let us know how you think Crossplane could help with your use case. --> I believe Crossplane should propagate the external name annotation from the composite resource to all composed resources by default. This ensures the annotation will always be honored.

The potential one-to-many relationship between a composite resource and its composed resources means there would be some gotches with this approach. Namely:

  1. Two composed resources of the same kind would very likely experience name conflicts by default. For example a KubeCluster composite resource that composed two GKE NodePool managed resources would attempt to name them both the same thing by default.
  2. There's no obvious way to propagate the annotation back from the composed resources to the composite resource in the case where the cloud provider does not allow us to specify an external name. If there are three composed resources, which resource's external name should be considered the external name of the one composite?

Both of these issues could be addressed by allowing Composition authors to override the default behaviour - to override the default relationship between the external name annotations. The first issue could be addressed by allowing a Composition to patch over the annotation that was set by default:

apiVersion: apiextensions.crossplane.io/v1alpha1
kind: Composition
metadata:
  name: mycomposition
spec:
  from:
    apiVersion: example.org/v1alpha1
    kind: KubeCluster
  to:
  - base:
      apiVersion: container.gcp.crossplane.io/v1alpha3
      kind: NodePool
      spec:
        # ...
  - patches:
    - fromFieldPath: metadata.annotations[crossplane.io/external-name]
      toFieldPath: metadata.annotations[crossplane.io/external-name]
      transforms:
        type: string
        string:
          fmt: "%s-a"
  - base:
      apiVersion: container.gcp.crossplane.io/v1alpha3
      kind: NodePool
      spec:
        # ...
  - patches:
    - fromFieldPath: metadata.annotations[crossplane.io/external-name]
      toFieldPath: metadata.annotations[crossplane.io/external-name]
      transforms:
        type: string
        string:
          fmt: "%s-b"

The second could be addressed effectively in reverse of the first, by letting the Composition author specify which composed resource (if any) should map its external name back to the composite resource. This would require something like https://github.com/crossplane/crossplane/issues/1639 to be implemented.

Ultimately this seems like a case of "which default behaviour is less bad", with a choice of:

  • The annotation has no effect by default.
  • The annotation is at risk of causing a naming conflict by default.

I prefer the latter choice because I believe is less surprising and it has less risk of human error. If the annotation has no effect unless the Composition author remembers to propagate it, a user could reasonably annotate their requirement or composite and have everything appear to work without issue. Their composed resources would create and the composite would appear to be ready, but in fact their annotation was being ignored. If the annotation causes a conflict when there are multiple resources of the same kind and the Composition author did not address this then any composite resources using that Composition would always break, making it obvious that the Composition needed to be updated.

created time in a month

issue openedcrossplane/crossplane

Composite resources should support bidirectional patching

<!-- Thank you for helping to improve Crossplane!

Please be sure to search for open issues before raising a new one. We use issues for bug reports and feature requests. Please find us at https://slack.crossplane.io for questions, support, and discussion. -->

What problem are you facing?

<!-- Please tell us a little about your use case - it's okay if it's hypothetical! Leading with this context helps frame the feature request so we can ensure we implement it sensibly. ---> Currently a Composition configures how the values of a composite resource's fields should be used to influence composed resources; the composite resource's spec values may be 'overlaid' or 'patched' onto the composed resources. For example the below Composition patches the external name annotation from the composite resource (the KubeCluster) to the composed resource (the NodePool):

apiVersion: apiextensions.crossplane.io/v1alpha1
kind: Composition
metadata:
  name: mycomposition
spec:
  from:
    apiVersion: example.org/v1alpha1
    kind: KubeCluster
  to:
  - base:
      apiVersion: container.gcp.crossplane.io/v1alpha3
      kind: NodePool
      spec:
        # ...
  - patches:
    - fromFieldPath: metadata.annotations[crossplane.io/external-name]
      toFieldPath: metadata.annotations[crossplane.io/external-name]
      transforms:
        type: string
        string:
          fmt: "%s-a"

This allows a composite resource to control the desired state of its composed resources. Ideally it would also be possible to reflect the actual state of composed resources on the composite resource. For example a KubeCluster resource that composed a GKECluster might want to reflect the status.atProvider.status (RUNNING, PROVISIONING, etc) of the GKECluster in its own status.

How could Crossplane help solve your problem?

<!-- Let us know how you think Crossplane could help with your use case. --> Crossplane could support bidirectional patching - both from composite to composed resource and from composed to composite resource. Presumably this would require us to rethink the from and to convention in the Composition schema.

This could look something like:

apiVersion: apiextensions.crossplane.io/v1alpha1
kind: Composition
metadata:
  name: mycomposition
spec:
  composite:
    apiVersion: example.org/v1alpha1
    kind: KubeCluster
  composed:
  - base:
      apiVersion: container.gcp.crossplane.io/v1beta1
      kind: GKECluster
      spec:
        # ...
  - patches:
    - fromFieldPath: metadata.annotations[crossplane.io/external-name]
      toFieldPath: metadata.annotations[crossplane.io/external-name]
      direction: CompositeToComposed  # Perhaps this is the default?
      transforms:
        type: string
        string:
          fmt: "%s-a"
    - fromFieldPath: status.atProvider.status
      toFieldPath: status.state
      direction: ComposedToComposite

created time in a month

Pull request review commentcrossplane/crossplane

[WIP] Package Manager refactor design doc

+# Packages v2+* Owner: Dan Mangum (@hasheddan)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane currently supports installing controllers and new CRDs into a+Kubernetes cluster that the Crossplane package manager is running in. While+there are many other packaging formats in the Kubernetes ecosystem, Crossplane+supports its own for the following reasons:++- Crossplane is [opinionated about the capabilities of a+  controller](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#allowed-resource-access)+  that can be installed to extend its functionality. For instance, controllers+  [may not run as+  root](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#package-deployment-privileges)+  or request cluster admin RBAC.+- Crossplane [allocates and aggregates various+  ClusterRoles](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#crossplane-clusterroles--rbac)+  to automatically provide permissions for users in the Kubernetes cluster to+  view / edit / create / delete CRDs installed by a package.+- Crossplane guards against conflicting CRDs being installed into a cluster.+- Crossplane adds [additional metadata to+  CRDs](https://github.com/crossplane/crossplane/blob/master/design/one-pager-stack-ui-metadata.md#crd-annotation-example)+  to provide additional context for displaying their configuration in a UI.+- Crossplane [adds labels to+  CRDs](https://github.com/crossplane/crossplane/blob/master/design/one-pager-stack-relationship-labels.md#example-wordpress-crdyaml-parented-by-stackinstall)+  in an attempt to establish parent-child relationships between CRDs.++In addition, the following unimplemented features are goals of the Crossplane+package manager:++- The ability to resolve dependencies listed in a package for CRDs that do not+  yet exist in the cluster.+- The ability for a user to upgrade installed package versions without manual+  intervention.++As part of these guarantees, Crossplane supports installing packages at both the+cluster (`ClusterPackageInstall`) and namespace scope (`PackageInstall`). The+primary difference between these two installation units is that the a+`ClusterPackageInstall` can only be installed once, and the controller's+`ServiceAccount` is bound to its `ClusterRole` with a `ClusterRoleBinding`,+meaning it can watch the resources for which ir requested RBAC at the cluster+scope. A `PackageInstall`, on the other hand, has its controller's+`ServiceAccount` bound to its `ClusterRole` with a `RoleBinding` that only+allows the controller to watch resources in its own namespace.++The advantage of a `PackageInstall` is that is theoretically allows for multiple+versions of the same package to be installed in a cluster at the same time+because its controller is only watching for objects in its own namespace.+However, because CRDs are cluster-scoped objects, there cannot be conflicting+CRDs installed by two different versions of a `PackageInstall`. So while the+goal of the `PackageInstall` is to enable multiple versions of a package to+exist simultaneously in a cluster, the limitations of CRDs makes it less+effective than desired in practice.++## Goals++The current package infrastructure, though well thought out, has become somewhat+convoluted and redundant with the introduction of+[composition](https://github.com/crossplane/crossplane/blob/master/design/design-doc-composition.md)+into the Crossplane ecosystem.++Composition solves the following goals originally intended to be addressed by a+`PackageInstall` and [template+stacks](https://github.com/crossplane/crossplane/blob/master/design/one-pager-template-stacks.md):++- Ability to publish infrastructure abstractions to specific namespaces.+  - The `PackageInstall` allowed packages to install a namespace-scoped CRD and+    a controller that only watched for resources in its namespace. Composition+    enables this by creating a namespace-scoped CRD using an+    `InfrastructurePublication` and restricting the namespace in which it can be+    provisioned using RBAC.+- Ability to create new packages without writing any code.+  - The templating controller enabled this by allowing a user to create a+    `behavior.yaml` file and automatically using the templating controller image+    (configured with the `behavior.yaml`) when one was not supplied in the+    package. Composition enables this without having to install a new+    controller. The composition controllers run as part of core Crossplane and+    dynamically reconcile new types that are created in response to the creation+    of an `InfrastructureDefinition` / `InfrastructurePublication`.++Because composition serves the same purposes as these existing packaging+primitives, the current packaging system can be narrowed in scope while also+supporting a similar set of use-cases. The immediate goals of this refactor are:++- Deprecation and subsequent removal of the `PackageInstall` and+  `StackDefinition` types.+- Standardization on a single `Package` type as a unit of installation.+- Support for upgrading a `Package` that is installed and running in a cluster.+- Support for native installation of composition types, including+  `InfrastructureDefinition`, `InfrastructurePublication`, and `Composition`.++The refactor will involve abandoning the following goals of the current model:++- Supporting running multiple versions of the same package in a cluster at the+  same time. This includes installing the same CRDs from multiple packages.++## Proposal++Related Issues:+- [Merge Package and+  PackageInstall](https://github.com/crossplane/crossplane/issues/1089)++This proposal broadly encompasses the following components:++- `Package`: a cluster-scoped CRD that represents the existence of a package in+  a cluster. Users create / update / delete `Package` instances to manage the+  packages present in their cluster.+- `Module`: a cluster-scoped CRD that represents the state of packages, CRDs and+  composition types in a cluster. Only one instance of the `Module` type exists+  in a cluster.+- `PackageRevision`: a cluster-scoped CRD that represents a version of a package+  that may or may not be active in a cluster. Many `PackageRevision` instances+  may exist for a single `Package`, but only one can be active at a given time.+  `PackageRevision` instances are named the `sha256` hash of the image they are+  represent.+- _`Package` controller_: a controller responsible for observing events on+  `Package` instances, creating new `PackageRevision` instances, and resolving+  dependencies.+- _`PackageRevision` controller_: a controller responsible for observing events+  on `PackageRevision` instances, installing new CRDs /+  `InfrastructureDefinition` / `InfrastructurePublication` / `Composition`+  objects, starting package controllers, and cleaning up these objects when a+  package is upgraded.++These types and controllers overlap in functionality with some of the existing+types and controllers that are currently present in the package manager+ecosystem. However, their implementation should not immediately break any of the+current features supported by the package manager. Removal of+`ClusterPackageInstall`, `PackageInstall`, and `StackDefinition`, as well as+refactor of `Package` and all controllers should follow a deprecation schedule+that allows users to move to the new format.++## Current Workflow++In order to accurately describe the steps required for implementing this+refactor. It is important to understand, at least at a high-level, the current+workflow the package manager uses for installing a `Package`. We will use a+`ClusterPackageInstall` for the example here as the namespace-scoped+`PackageInstall` is intended to be removed as part of this refactor.++1. User creates a `ClusterPackageInstall`.+2. The _`ClusterPackageInstall` controller_ observes the `ClusterPackageInstall`+   creation and creates a `Job` that runs a `Pod` with an initContainer running+   the image supplied on the `ClusterPackageInstall` and a container running the+   same image the package manager itself is running. The `initContainer` serves+   to only copy the contents of the package directory from the supplied image+   into a `Volume`. That `Volume` is then mounted on the `container` running the+   package manager image, which is executed with the `unpack`+   [argument](https://github.com/crossplane/crossplane/blob/a0d139f7cf269599ba916ed15af3fd68ffeabbdf/cmd/crossplane/package/unpack/unpack.go#L52).+3. The `unpack` container walks the filepath of the package directory, using the+   information provided to construct a `Package` object. It then+   [writes](https://github.com/crossplane/crossplane/blob/a0d139f7cf269599ba916ed15af3fd68ffeabbdf/pkg/packages/unpack.go#L204)+   the `Package` object and all CRDs in the package to+   [stdout](https://github.com/crossplane/crossplane/blob/a0d139f7cf269599ba916ed15af3fd68ffeabbdf/cmd/crossplane/package/unpack/unpack.go#L53).+4. The _`ClusterPackageInstall` controller_ waits for the `Job` to complete+   successfully before reading the logs from the `Pod`. When the `Job` is+   complete, it reads the the logs and creates all of the objects that were+   printed, making a [few modifications as well as annotating and labelling+   appropriately](https://github.com/crossplane/crossplane/blob/6fc50822fbf11a7d31f8a9dabde5c8948c3b36ac/pkg/controller/packages/install/installjob.go#L259).+5. The _`Package` controller_ observes the `Package` creation and assumes the+   following responsibilities:+  - Setting up all RBAC described in the [Background](#background) section.+  - Annotating all CRDs with the proper labels.+  - Creating the `ServiceAccount` for the controller `Deployment`, binding it to+    its `ClusterRole` and starting the controller (i.e. creating the+    `Deployment`).+  - Making any modifications, such as+    [syncing](https://github.com/crossplane/crossplane/blob/6fc50822fbf11a7d31f8a9dabde5c8948c3b36ac/pkg/controller/packages/pkg/pkg.go#L696)+    the `Secret` for the `ServiceAccount` that are required for running the+    controller in [host aware+    mode](https://github.com/crossplane/crossplane/blob/master/design/one-pager-host-aware-stack-manager.md).++The process for a `PackageInstall` is very similar, but the packages using the+templating controller have the additional step of first producing a+`StackDefinition` in the install `Job`, then translating that to a `Package` in+the [_`StackDefinition`+controller_](https://github.com/crossplane/crossplane/blob/6fc50822fbf11a7d31f8a9dabde5c8948c3b36ac/pkg/controller/packages/templates/stackdefinition_controller.go#L53).++## Implementation++This following sections describe the changes to the current package manager that+are required to support the goals listed above.++- [Package Format](#package-format)+- [API Changes](#api-changes)+- [Package Installation Configuration](#package-installation-configuration)+- [Dependency Management](#dependency-management)+- [Installation](#installation)+- [Scenarios](#scenarios)+- [Phase 1](#phase-1)+- [Phase 2](#phase-2)+- [Phase 3](#phase-3)++### Package Format++TODO(hasheddan): This section is still a WIP++A Crossplane package is an OCI image that contains a `.registry/` directory in+its filesystem. While the image is really just serving as a file format, it has+a few properties that make it advantageous:++- Allows integration with existing OCI image registries and tooling.+- Allows a package to also include a binary that it intends to be run after+  install is complete (i.e. the controller).+- Allows package metadata schema and filesystem format to change over time+  without having to modify the package manager because the logic can be included+  in the package image to allow it to "unpack" itself in a format that the+  package manager understands.++In reality, the second two properties are somewhat mutually exclusive.+Currently, a package does not unpack itself, but rather has its `.registry/`+directory copied into the filesystem of an unpacking container that provides the+logic for translating the manifests into a format that the package manager+understands. This means that the `ENTRYPOINT` of the package image can be+invoked directly in the `Deployment` that is created as part of installation,+but also means that the package author must conform to whatever "unpack" image+is being used, rather than building it into the package image itself.++OCI images are valuable in part because they are essentially an open playground+that allows you to do almost anything. In contrast, Crossplane packages are+extremely scoped and are opinionated about how you structure your image. For+this reason, it makes sense to abstract the image building process from the+package author, such that they only define their package contents and use the+Crossplane CLI to build and push the package.++*Note: this would not preclude someone from building the package image+themselves, but it would make it much easier for them to defer that+responsibility.*++As part of this abstraction, the ability to also include the binary that is+intended to be executed as the package's controller would be removed. The+original purpose of this functionality was to not require a package author to+build and push a controller image, then have to reference it in the package's+`install.yaml`, which defines the `Deployment` that should be created.++```go+// PackageSpec specifies the desired state of a Package.+type PackageSpec struct {+  AppMetadataSpec `json:",inline"`+  CRDs            CRDList         `json:"customresourcedefinitions,omitempty"`+  Controller      ControllerSpec  `json:"controller,omitempty"`+  Permissions     PermissionsSpec `json:"permissions,omitempty"`+}+```++```go+// AppMetadataSpec defines metadata about the package application+type AppMetadataSpec struct {+  Title         string            `json:"title,omitempty"`+  OverviewShort string            `json:"overviewShort,omitempty"`+  Overview      string            `json:"overview,omitempty"`+  Readme        string            `json:"readme,omitempty"`+  Version       string            `json:"version,omitempty"`+  Icons         []IconSpec        `json:"icons,omitempty"`+  Maintainers   []ContributorSpec `json:"maintainers,omitempty"`+  Owners        []ContributorSpec `json:"owners,omitempty"`+  Company       string            `json:"company,omitempty"`+  Category      string            `json:"category,omitempty"`+  Keywords      []string          `json:"keywords,omitempty"`+  Website       string            `json:"website,omitempty"`+  Source        string            `json:"source,omitempty"`+  License       string            `json:"license,omitempty"`++  // DependsOn is the list of CRDs that this package depends on. This data+  // drives the RBAC generation process.+  DependsOn []PackageInstallSpec `json:"dependsOn,omitempty"`++  // +kubebuilder:validation:Enum=Provider;Stack;Application;Addon+  PackageType string `json:"packageType,omitempty"`++  // +kubebuilder:validation:Enum=Cluster;Namespaced+  PermissionScope string `json:"permissionScope,omitempty"`+}+```++```go+// PackageInstallSpec specifies details about a request to install a package to+// Crossplane.+type PackageInstallSpec struct {+  PackageControllerOptions `json:",inline"`++  // Source is the domain name for the package registry hosting the package+  // being requested, e.g., registry.crossplane.io+  Source string `json:"source,omitempty"`++  // Package is the name of the package package that is being requested, e.g.,+  // myapp. Either Package or CustomResourceDefinition can be specified.+  Package string `json:"package,omitempty"`++  // CustomResourceDefinition is the full name of a CRD that is owned by the+  // package being requested. This can be a convenient way of installing a+  // package when the desired CRD is known, but the package name that contains+  // it is not known. Either Package or CustomResourceDefinition can be+  // specified.+  CustomResourceDefinition string `json:"crd,omitempty"`+}++// PackageControllerOptions allow for changes in the Package extraction and+// deployment controllers. These can affect how images are fetched and how+// Package derived resources are created.+type PackageControllerOptions struct {+  // ImagePullSecrets are named secrets in the same workspace that can be used+  // to fetch Packages from private repositories and to run controllers from+  // private repositories+  ImagePullSecrets []corev1.LocalObjectReference `json:"imagePullSecrets,omitempty"`++  // ImagePullPolicy defines the pull policy for all images used during+  // Package extraction and when running the Package controller.+  // https://kubernetes.io/docs/concepts/configuration/overview/#container-images+  ImagePullPolicy corev1.PullPolicy `json:"imagePullPolicy,omitempty"`++  // ServiceAccount options allow for changes to the ServiceAccount the+  // Package Manager creates for the Package's controller+  ServiceAccount *ServiceAccountOptions `json:"serviceAccount,omitempty"`+}+```++```go+// CRDList is the full list of CRDs that this package owns and depends on+type CRDList []metav1.TypeMeta+```++```go+// ControllerSpec defines the controller that implements the logic for a+// package, which can come in different flavors.+type ControllerSpec struct {+  // ServiceAccount options allow for changes to the ServiceAccount the+  // Package Manager creates for the Package's controller+  ServiceAccount *ServiceAccountOptions `json:"serviceAccount,omitempty"`++  Deployment *ControllerDeployment `json:"deployment,omitempty"`+}+```++```go+// PermissionsSpec defines the permissions that a package will require to+// operate.+type PermissionsSpec struct {+  Rules []rbac.PolicyRule `json:"rules,omitempty"`+}+```++**Action Items**++- Determine if a `Job` that runs the `unpack` image is the best way to ++### Package Installation Configuration++Related Issues:+- [Env var support for packages to enable users to supply proxy+  credentials](https://github.com/crossplane/crossplane/issues/1638)++TODO(hasheddan): section on allowing some values to be passed from `Package` to+controller in `PackageRevision`++### API Changes++The first step towards supporting the upgrade of packages is by introducing the+`PackageRevision` type. The `PackageRevision` type will look very similar to the+current `Package`, but will add the following fields:++- `spec.infrastructureDefinitions`: list of `InfrastructureDefinition`s+  installed by the package.+- `spec.compositions`: list of `Composition`s installed by the package.+- `spec.state`: the desired state, which can be one of `Pending`, `Active`,+  `Stopped` or `Inactive`.++The following fields will be removed:++- `spec.packageType`: currently is one of `Provider`, `Stack`, `Application`, or+  `Addon`.+- `spec.permissionType`: currently is one of `Cluster` or `Namespace`.++In addition, `PackageRevision` will be a cluster-scoped type, rather than the+current namespace-scoped `Package`. The current schema for `Package` can be+viewed+[here](https://doc.crds.dev/github.com/crossplane/crossplane/packages.crossplane.io/Package/v1alpha1@v0.12.0).++In order to continue to support the current functionality of+`ClusterPackageInstall`, `PackageInstall`, `StackDefinition` the underlying+`Package` type will be renamed to `LegacyPackage`. The existing `Package` type+will be refactored to look like the current `ClusterPackageInstall` schema with+the following modifications.++This field will be added:++- `status.currentHash`: a string that is the `sha256` hash of the image in the+  current `PackageRevision`.+- `status.currentImage`: a string that is the name of the current+  `PackageRevision` image.++The following fields will be removed:++- `status.installJob`: a reference to the install `Job` for the+  `ClusterPackageInstall`. +- `status.packageRecord`: a reference to the `Package` that represents the+  installation unit for the `ClusterPackageInstall`. The `PackageRevision` is+  now discoverable by name at the cluster scope.++Before outlining the functionality of the new `Package` and `PackageRevision`+controllers, it is useful to establish the responsibilities of each.++The `Package` controller is responsible for:++- Managing the lifecycle of `PackageRevision`s+- Resolving dependencies between `Package`s++The `PackageRevision` controller is responsible for:++- Managing the lifecycle of any controller is installed by a package image+- Managing the creation and ownership of objects included in the package++### Dependency Management++Dependency management of packages can be boiled down to three questions:++1. Does the package have any dependencies?+2. Do those dependencies exist in my cluster?+3. Do compatible versions of those dependencies exist in my cluster?++If a `Package` is created that has no dependencies, it should always be+installed successfully unless it attempts to install objects that are already+present and owned in the cluster. However, most packages will depend on at least+on other package. Every package that does have dependencies should follow a+graph that terminates with leaf dependencies that in turn do not have any+dependencies themselves.++**Naive Approach**++Let's first consider a naive approach for dependency resolution to illustrate+the problem space. In the following example, a user is installing a package into+a fresh Kubernetes cluster.++User creates a `Package` with `spec.package: crossplane/pkg-a:v0.0.1`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: pkg-a+spec:+  package: crossplane/pkg-a:v0.0.1+```++When `crossplane/pkg-a:v0.0.1` is unpacked, it is shown to depend on+`crossplane/pkg-b`, `crossplane/pkg-c`, and `crossplane/pkg-d`++*Note: see [this documentation](https://nodesource.com/blog/semver-a-primer/)+for more information on the semver ranges used in the dependencies below.*++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: PackageRevision+metadata:+  name: <crossplane/pkg-a:v0.0.1-hash>+  ownerReferences:+  - apiVersion: packages.crossplane.io/v1alpha1+    controller: true+    blockOwnerDeletion: true+    kind: Package+    name: pkg-a+    uid: d9607e19-f88f-11e6-a518-42010a800195+spec:+  ...+  dependsOn:+    packages:+    - package: registry.upbound.io/crossplane/pkg-b:v1.*+    - package: registry.upbound.io/crossplane/pkg-b:v1.2.3-v1.3.5+    - package: registry.upbound.io/crossplane/pkg-b:~v2.5.7+  title: Package A+  ...+```++As mentioned before, this is a fresh Kubernetes cluster, so none of these+dependencies will have already been installed. Therefore, they need to be+installed before `pkg-a` can complete its installation. Three new `Package`s+would be created.++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-pkg-b # note: it will be suggested that users also name their packages in this format, but it cannot be strictly required+spec:+  package: registry.upbound.io/crossplane/pkg-b:v1.9.15 # latest v1.* version+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-pkg-c+spec:+  package: registry.upbound.io/crossplane/pkg-c:v1.3.5 # latest in range+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-pkg-d+spec:+  package: registry.upbound.io/crossplane/pkg-d:v2.5.22 # latest patch version+```++Each of these packages now must be unpacked and their dependencies evaluated as+well.++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: PackageRevision+metadata:+  name: <crossplane/pkg-b:1.9.15-hash>+  ownerReferences:+  - apiVersion: packages.crossplane.io/v1alpha1+    controller: true+    blockOwnerDeletion: true+    kind: Package+    name: registry.upbound.io-crossplane-pkg-b+    uid: d9607e19-f88f-11e6-a518-42010a800195+spec:+  ...+  dependsOn:+    packages:+    - package: registry.upbound.io/crossplane/provider-aws:v10.*+    - package: registry.upbound.io/crossplane/provider-gcp:v10.*+  title: Package B+  ...+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: PackageRevision+metadata:+  name: <crossplane/pkg-c:v1.3.5-hash>+  ownerReferences:+  - apiVersion: packages.crossplane.io/v1alpha1+    controller: true+    blockOwnerDeletion: true+    kind: Package+    name: registry.upbound.io-crossplane-pkg-c+    uid: d9607e19-f88f-11e6-a518-42010a800195+spec:+  ...+  dependsOn:+    packages:+    - package: registry.upbound.io/crossplane/provider-aws:v10.1.1+    - package: registry.upbound.io/crossplane/provider-gcp:v10.*+  title: Package C+  ...+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: PackageRevision+metadata:+  name: <crossplane/pkg-d:2.5.22-hash>+  ownerReferences:+  - apiVersion: packages.crossplane.io/v1alpha1+    controller: true+    blockOwnerDeletion: true+    kind: Package+    name: registry.upbound.io-crossplane-pkg-d+    uid: d9607e19-f88f-11e6-a518-42010a800195+spec:+  ...+  dependsOn:+    packages:+    - package: registry.upbound.io/crossplane/provider-aws:v10.1.*+    - package: registry.upbound.io/crossplane/provider-gcp:v10.3.*+  title: Package D+  ...+```++As seen in the `dependsOn` sections above, all three of these packages depend on+`provider-aws` and `provider-gcp`. Ideally, we would like for these controllers+to be able to reconcile independently of each other, each arriving at a stable+state with the correct versions of `provider-aws` and `provider-gcp` installed.+This is difficult because each of the three packages have different constraints+on their version requirements. For this example, let's say that `provider-gcp`'s+latest version is `v10.9.12` and `provider-aws`'s latest version is `v10.4.15`.+If `pkg-b` is reconciled first, we will see the following `Packages`:++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-provider-aws+spec:+  package: registry.upbound.io/crossplane/provider-aws:v10.4.15+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-provider-gcp+spec:+  package: registry.upbound.io/crossplane/provider-gcp:v10.9.12+```++These versions work great for `pkg-b`, but they will conflict with the required+versions for `pkg-c` and `pkg-d`. In fact, no matter which package is reconciled+first, there will be a conflict for another. What we want to do here is+establish consensus about what version should be installed. Let's look at more+sophisticated approach.++**Recursive Dependency Resolution**++Enter the `Module` type. It is a global representation of packages installed in+a cluster. The _`Package` controller_ watches for `Package`s and resolves+dependencies. Let's go back to the creation of the `pkg-a` `Package`. The+_`Package` controller_ would observe the `Package` creation, unpack the image,+check that there are no conflicts with the objects it is attempting to install,+then resolve its dependencies. Resolving dependencies would required recursively+unpacking all dependencies until encountering a conflict or reaching a leaf of+the dependency graph.++Original state of `Module`.++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Module+metadata:+  name: packages+```++*Reconcile 1 - Start Unpack*++1. Observe creation of `pkg-a`+2. Get `Module` and check if `pkg-a` exists (N)+3. Check for existing `Job` for `pkg-a` (N)+3. Create `Job` with name `pkg-a` in `crossplane-system` to unpack the image+4. Return with short wait++*Reconcile 2 - Add to Module*++1. Observe `pkg-a`+2. Get `Module` and check if `pkg-a` exists (N)+3. Check for existing `Job` for `pkg-a` (Y)+4. Read output of unpack `Job`+5. Check to see if all objects it desires to create are not already present and+   owned by another package (Y)+6. Add to `Module`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Module+metadata:+  name: packages+spec:+  packages:+  - name: pkg-a+    image: registry.upbound.io/crossplane/pkg-a:v0.0.1@abcdef+    dependencies:+    - registry.upbound.io/crossplane/pkg-b:v1.*+    - registry.upbound.io/crossplane/pkg-c:v1.2.3-v1.3.5+    - registry.upbound.io/crossplane/pkg-d:~v2.5.7+  compositions:+  - veryabstract.binfra.pkga.coolorg.io: pkg-a+  - veryabstract.cinfra.pkga.coolorg.io: pkg-a+  - veryabstract.dinfra.pkga.coolorg.io: pkg-a+```++7. Return with short wait++*Reconcile 3 - Start Resolution of Direct Dependencies*++1. Observe `pkg-a`+2. Get `Module` and check if `pkg-a` exists (Y)+3. Check if `pkg-a` is the a dependency of another package (N)+4. Check for existing `PackageRevision` owned by `pkg-a` (N)+5. Create `PackageRevision` with name equivalent to hash of image and state+   `Pending`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: PackageRevision+metadata:+  name: abcdef+  ownerReferences:+  - apiVersion: packages.crossplane.io/v1alpha1+    controller: true+    blockOwnerDeletion: true+    kind: Package+    name: pkg-a+    uid: d9607e19-f88f-11e6-a518-42010a800195+spec:+  compositions: # pkg-a is only a bundle of compositions+  - veryabstract.binfra.pkga.coolorg.io+  - veryabstract.cinfra.pkga.coolorg.io+  - veryabstract.dinfra.pkga.coolorg.io+  ...+```++6. Check to see if all dependencies in `PackageRevision` exist (N)+7. Create a `Package` for each dependency++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-pkg-b+spec:+  package: registry.upbound.io/crossplane/pkg-b:v1.9.15 # latest v1.* version+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-pkg-c+spec:+  package: registry.upbound.io/crossplane/pkg-c:v1.3.5 # latest in range+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-pkg-d+spec:+  package: registry.upbound.io/crossplane/pkg-d:v2.5.22 # latest patch version+```++8. Return with short wait++*Reconcile 4 - Observe Progress of Dependency Resolution*++1. Observe `pkg-a`+2. Get `Module` and check if `pkg-a` exists (Y)+3. Check if `pkg-a` is the a dependency of another package (N)+4. Check for existing `PackageRevision` owned by `pkg-a` (Y)+5. Check for dependencies in `Module` required by `PackageRevision` (Y)+  - Assuming those packages have now gotten to the stage where they have been+    added to `Module`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Module+metadata:+  name: packages+spec:+  packages:+  - name: pkg-a+    image: registry.upbound.io/crossplane/pkg-a:v0.0.1@abcdef+    dependencies:+    - registry.upbound.io/crossplane/pkg-b:v1.*+    - registry.upbound.io/crossplane/pkg-c:v1.2.3-v1.3.5+    - registry.upbound.io/crossplane/pkg-d:~v2.5.7+  - name: registry.upbound.io-crossplane-pkg-b+    image: registry.upbound.io/crossplane/pkg-b:1.9.15@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.*+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  - name: registry.upbound.io-crossplane-pkg-c+    image: registry.upbound.io/crossplane/pkg-c:v1.3.5@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.1.1+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  - name: registry.upbound.io-crossplane-pkg-d+    image: registry.upbound.io/crossplane/pkg-d:2.5.22@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.1.*+    - registry.upbound.io/crossplane/provider-gcp:v10.3.*+  compositions:+  - veryabstract.binfra.pkga.coolorg.io: pkg-a+  - veryabstract.cinfra.pkga.coolorg.io: pkg-a+  - veryabstract.dinfra.pkga.coolorg.io: pkg-a+  crds: # Each of the dependency packages (b,c,d) just bring a single InfrastructureDefinition+  - abstract.pkgb.coolorg.io: pkg-b+  - abstract.pkgc.coolorg.io: pkg-c+  - abstract.pkgd.coolorg.io: pkg-d

I notice that PackageRevision appears to have compositions and infrastructureDefinitions, while this type has compositons and crds (which look like they might be from InfrastructureDefinitions?).

In general I'm wondering whether it's possible for us to be consistent with regard to whether packages have knowledge of only Crossplane specific types (composition, infrapub, infradef, managed resource, provider) or deals strictly in generic concepts (CR, CRD).

hasheddan

comment created time in a month

Pull request review commentcrossplane/crossplane

[WIP] Package Manager refactor design doc

+# Packages v2+* Owner: Dan Mangum (@hasheddan)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane currently supports installing controllers and new CRDs into a+Kubernetes cluster that the Crossplane package manager is running in. While+there are many other packaging formats in the Kubernetes ecosystem, Crossplane+supports its own for the following reasons:++- Crossplane is [opinionated about the capabilities of a+  controller](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#allowed-resource-access)+  that can be installed to extend its functionality. For instance, controllers+  [may not run as+  root](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#package-deployment-privileges)+  or request cluster admin RBAC.+- Crossplane [allocates and aggregates various+  ClusterRoles](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#crossplane-clusterroles--rbac)+  to automatically provide permissions for users in the Kubernetes cluster to+  view / edit / create / delete CRDs installed by a package.+- Crossplane guards against conflicting CRDs being installed into a cluster.+- Crossplane adds [additional metadata to+  CRDs](https://github.com/crossplane/crossplane/blob/master/design/one-pager-stack-ui-metadata.md#crd-annotation-example)+  to provide additional context for displaying their configuration in a UI.+- Crossplane [adds labels to+  CRDs](https://github.com/crossplane/crossplane/blob/master/design/one-pager-stack-relationship-labels.md#example-wordpress-crdyaml-parented-by-stackinstall)+  in an attempt to establish parent-child relationships between CRDs.++In addition, the following unimplemented features are goals of the Crossplane+package manager:++- The ability to resolve dependencies listed in a package for CRDs that do not+  yet exist in the cluster.+- The ability for a user to upgrade installed package versions without manual+  intervention.++As part of these guarantees, Crossplane supports installing packages at both the+cluster (`ClusterPackageInstall`) and namespace scope (`PackageInstall`). The+primary difference between these two installation units is that the a+`ClusterPackageInstall` can only be installed once, and the controller's+`ServiceAccount` is bound to its `ClusterRole` with a `ClusterRoleBinding`,+meaning it can watch the resources for which ir requested RBAC at the cluster+scope. A `PackageInstall`, on the other hand, has its controller's+`ServiceAccount` bound to its `ClusterRole` with a `RoleBinding` that only+allows the controller to watch resources in its own namespace.++The advantage of a `PackageInstall` is that is theoretically allows for multiple+versions of the same package to be installed in a cluster at the same time+because its controller is only watching for objects in its own namespace.+However, because CRDs are cluster-scoped objects, there cannot be conflicting+CRDs installed by two different versions of a `PackageInstall`. So while the+goal of the `PackageInstall` is to enable multiple versions of a package to+exist simultaneously in a cluster, the limitations of CRDs makes it less+effective than desired in practice.++## Goals++The current package infrastructure, though well thought out, has become somewhat+convoluted and redundant with the introduction of+[composition](https://github.com/crossplane/crossplane/blob/master/design/design-doc-composition.md)+into the Crossplane ecosystem.++Composition solves the following goals originally intended to be addressed by a+`PackageInstall` and [template+stacks](https://github.com/crossplane/crossplane/blob/master/design/one-pager-template-stacks.md):++- Ability to publish infrastructure abstractions to specific namespaces.+  - The `PackageInstall` allowed packages to install a namespace-scoped CRD and+    a controller that only watched for resources in its namespace. Composition+    enables this by creating a namespace-scoped CRD using an+    `InfrastructurePublication` and restricting the namespace in which it can be+    provisioned using RBAC.+- Ability to create new packages without writing any code.+  - The templating controller enabled this by allowing a user to create a+    `behavior.yaml` file and automatically using the templating controller image+    (configured with the `behavior.yaml`) when one was not supplied in the+    package. Composition enables this without having to install a new+    controller. The composition controllers run as part of core Crossplane and+    dynamically reconcile new types that are created in response to the creation+    of an `InfrastructureDefinition` / `InfrastructurePublication`.++Because composition serves the same purposes as these existing packaging+primitives, the current packaging system can be narrowed in scope while also+supporting a similar set of use-cases. The immediate goals of this refactor are:++- Deprecation and subsequent removal of the `PackageInstall` and+  `StackDefinition` types.+- Standardization on a single `Package` type as a unit of installation.+- Support for upgrading a `Package` that is installed and running in a cluster.+- Support for native installation of composition types, including+  `InfrastructureDefinition`, `InfrastructurePublication`, and `Composition`.++The refactor will involve abandoning the following goals of the current model:++- Supporting running multiple versions of the same package in a cluster at the+  same time. This includes installing the same CRDs from multiple packages.++## Proposal++Related Issues:+- [Merge Package and+  PackageInstall](https://github.com/crossplane/crossplane/issues/1089)++This proposal broadly encompasses the following components:++- `Package`: a cluster-scoped CRD that represents the existence of a package in+  a cluster. Users create / update / delete `Package` instances to manage the+  packages present in their cluster.+- `Module`: a cluster-scoped CRD that represents the state of packages, CRDs and+  composition types in a cluster. Only one instance of the `Module` type exists+  in a cluster.+- `PackageRevision`: a cluster-scoped CRD that represents a version of a package+  that may or may not be active in a cluster. Many `PackageRevision` instances+  may exist for a single `Package`, but only one can be active at a given time.+  `PackageRevision` instances are named the `sha256` hash of the image they are+  represent.+- _`Package` controller_: a controller responsible for observing events on+  `Package` instances, creating new `PackageRevision` instances, and resolving+  dependencies.+- _`PackageRevision` controller_: a controller responsible for observing events+  on `PackageRevision` instances, installing new CRDs /+  `InfrastructureDefinition` / `InfrastructurePublication` / `Composition`+  objects, starting package controllers, and cleaning up these objects when a+  package is upgraded.++These types and controllers overlap in functionality with some of the existing+types and controllers that are currently present in the package manager+ecosystem. However, their implementation should not immediately break any of the+current features supported by the package manager. Removal of+`ClusterPackageInstall`, `PackageInstall`, and `StackDefinition`, as well as+refactor of `Package` and all controllers should follow a deprecation schedule+that allows users to move to the new format.++## Current Workflow++In order to accurately describe the steps required for implementing this+refactor. It is important to understand, at least at a high-level, the current+workflow the package manager uses for installing a `Package`. We will use a+`ClusterPackageInstall` for the example here as the namespace-scoped+`PackageInstall` is intended to be removed as part of this refactor.++1. User creates a `ClusterPackageInstall`.+2. The _`ClusterPackageInstall` controller_ observes the `ClusterPackageInstall`+   creation and creates a `Job` that runs a `Pod` with an initContainer running+   the image supplied on the `ClusterPackageInstall` and a container running the+   same image the package manager itself is running. The `initContainer` serves+   to only copy the contents of the package directory from the supplied image+   into a `Volume`. That `Volume` is then mounted on the `container` running the+   package manager image, which is executed with the `unpack`+   [argument](https://github.com/crossplane/crossplane/blob/a0d139f7cf269599ba916ed15af3fd68ffeabbdf/cmd/crossplane/package/unpack/unpack.go#L52).+3. The `unpack` container walks the filepath of the package directory, using the+   information provided to construct a `Package` object. It then+   [writes](https://github.com/crossplane/crossplane/blob/a0d139f7cf269599ba916ed15af3fd68ffeabbdf/pkg/packages/unpack.go#L204)+   the `Package` object and all CRDs in the package to+   [stdout](https://github.com/crossplane/crossplane/blob/a0d139f7cf269599ba916ed15af3fd68ffeabbdf/cmd/crossplane/package/unpack/unpack.go#L53).+4. The _`ClusterPackageInstall` controller_ waits for the `Job` to complete+   successfully before reading the logs from the `Pod`. When the `Job` is+   complete, it reads the the logs and creates all of the objects that were+   printed, making a [few modifications as well as annotating and labelling+   appropriately](https://github.com/crossplane/crossplane/blob/6fc50822fbf11a7d31f8a9dabde5c8948c3b36ac/pkg/controller/packages/install/installjob.go#L259).+5. The _`Package` controller_ observes the `Package` creation and assumes the+   following responsibilities:+  - Setting up all RBAC described in the [Background](#background) section.+  - Annotating all CRDs with the proper labels.+  - Creating the `ServiceAccount` for the controller `Deployment`, binding it to+    its `ClusterRole` and starting the controller (i.e. creating the+    `Deployment`).+  - Making any modifications, such as+    [syncing](https://github.com/crossplane/crossplane/blob/6fc50822fbf11a7d31f8a9dabde5c8948c3b36ac/pkg/controller/packages/pkg/pkg.go#L696)+    the `Secret` for the `ServiceAccount` that are required for running the+    controller in [host aware+    mode](https://github.com/crossplane/crossplane/blob/master/design/one-pager-host-aware-stack-manager.md).++The process for a `PackageInstall` is very similar, but the packages using the+templating controller have the additional step of first producing a+`StackDefinition` in the install `Job`, then translating that to a `Package` in+the [_`StackDefinition`+controller_](https://github.com/crossplane/crossplane/blob/6fc50822fbf11a7d31f8a9dabde5c8948c3b36ac/pkg/controller/packages/templates/stackdefinition_controller.go#L53).++## Implementation++This following sections describe the changes to the current package manager that+are required to support the goals listed above.++- [Package Format](#package-format)+- [API Changes](#api-changes)+- [Package Installation Configuration](#package-installation-configuration)+- [Dependency Management](#dependency-management)+- [Installation](#installation)+- [Scenarios](#scenarios)+- [Phase 1](#phase-1)+- [Phase 2](#phase-2)+- [Phase 3](#phase-3)++### Package Format++TODO(hasheddan): This section is still a WIP++A Crossplane package is an OCI image that contains a `.registry/` directory in+its filesystem. While the image is really just serving as a file format, it has+a few properties that make it advantageous:++- Allows integration with existing OCI image registries and tooling.+- Allows a package to also include a binary that it intends to be run after+  install is complete (i.e. the controller).+- Allows package metadata schema and filesystem format to change over time+  without having to modify the package manager because the logic can be included+  in the package image to allow it to "unpack" itself in a format that the+  package manager understands.++In reality, the second two properties are somewhat mutually exclusive.+Currently, a package does not unpack itself, but rather has its `.registry/`+directory copied into the filesystem of an unpacking container that provides the+logic for translating the manifests into a format that the package manager+understands. This means that the `ENTRYPOINT` of the package image can be+invoked directly in the `Deployment` that is created as part of installation,+but also means that the package author must conform to whatever "unpack" image+is being used, rather than building it into the package image itself.++OCI images are valuable in part because they are essentially an open playground+that allows you to do almost anything. In contrast, Crossplane packages are+extremely scoped and are opinionated about how you structure your image. For+this reason, it makes sense to abstract the image building process from the+package author, such that they only define their package contents and use the+Crossplane CLI to build and push the package.++*Note: this would not preclude someone from building the package image+themselves, but it would make it much easier for them to defer that+responsibility.*++As part of this abstraction, the ability to also include the binary that is+intended to be executed as the package's controller would be removed. The+original purpose of this functionality was to not require a package author to+build and push a controller image, then have to reference it in the package's+`install.yaml`, which defines the `Deployment` that should be created.++```go+// PackageSpec specifies the desired state of a Package.+type PackageSpec struct {+  AppMetadataSpec `json:",inline"`+  CRDs            CRDList         `json:"customresourcedefinitions,omitempty"`+  Controller      ControllerSpec  `json:"controller,omitempty"`+  Permissions     PermissionsSpec `json:"permissions,omitempty"`+}+```++```go+// AppMetadataSpec defines metadata about the package application+type AppMetadataSpec struct {+  Title         string            `json:"title,omitempty"`+  OverviewShort string            `json:"overviewShort,omitempty"`+  Overview      string            `json:"overview,omitempty"`+  Readme        string            `json:"readme,omitempty"`+  Version       string            `json:"version,omitempty"`+  Icons         []IconSpec        `json:"icons,omitempty"`+  Maintainers   []ContributorSpec `json:"maintainers,omitempty"`+  Owners        []ContributorSpec `json:"owners,omitempty"`+  Company       string            `json:"company,omitempty"`+  Category      string            `json:"category,omitempty"`+  Keywords      []string          `json:"keywords,omitempty"`+  Website       string            `json:"website,omitempty"`+  Source        string            `json:"source,omitempty"`+  License       string            `json:"license,omitempty"`++  // DependsOn is the list of CRDs that this package depends on. This data+  // drives the RBAC generation process.+  DependsOn []PackageInstallSpec `json:"dependsOn,omitempty"`++  // +kubebuilder:validation:Enum=Provider;Stack;Application;Addon+  PackageType string `json:"packageType,omitempty"`++  // +kubebuilder:validation:Enum=Cluster;Namespaced+  PermissionScope string `json:"permissionScope,omitempty"`+}+```++```go+// PackageInstallSpec specifies details about a request to install a package to+// Crossplane.+type PackageInstallSpec struct {+  PackageControllerOptions `json:",inline"`++  // Source is the domain name for the package registry hosting the package+  // being requested, e.g., registry.crossplane.io+  Source string `json:"source,omitempty"`++  // Package is the name of the package package that is being requested, e.g.,+  // myapp. Either Package or CustomResourceDefinition can be specified.+  Package string `json:"package,omitempty"`++  // CustomResourceDefinition is the full name of a CRD that is owned by the+  // package being requested. This can be a convenient way of installing a+  // package when the desired CRD is known, but the package name that contains+  // it is not known. Either Package or CustomResourceDefinition can be+  // specified.+  CustomResourceDefinition string `json:"crd,omitempty"`+}++// PackageControllerOptions allow for changes in the Package extraction and+// deployment controllers. These can affect how images are fetched and how+// Package derived resources are created.+type PackageControllerOptions struct {+  // ImagePullSecrets are named secrets in the same workspace that can be used+  // to fetch Packages from private repositories and to run controllers from+  // private repositories+  ImagePullSecrets []corev1.LocalObjectReference `json:"imagePullSecrets,omitempty"`++  // ImagePullPolicy defines the pull policy for all images used during+  // Package extraction and when running the Package controller.+  // https://kubernetes.io/docs/concepts/configuration/overview/#container-images+  ImagePullPolicy corev1.PullPolicy `json:"imagePullPolicy,omitempty"`++  // ServiceAccount options allow for changes to the ServiceAccount the+  // Package Manager creates for the Package's controller+  ServiceAccount *ServiceAccountOptions `json:"serviceAccount,omitempty"`+}+```++```go+// CRDList is the full list of CRDs that this package owns and depends on+type CRDList []metav1.TypeMeta+```++```go+// ControllerSpec defines the controller that implements the logic for a+// package, which can come in different flavors.+type ControllerSpec struct {+  // ServiceAccount options allow for changes to the ServiceAccount the+  // Package Manager creates for the Package's controller+  ServiceAccount *ServiceAccountOptions `json:"serviceAccount,omitempty"`++  Deployment *ControllerDeployment `json:"deployment,omitempty"`+}+```++```go+// PermissionsSpec defines the permissions that a package will require to+// operate.+type PermissionsSpec struct {+  Rules []rbac.PolicyRule `json:"rules,omitempty"`+}+```++**Action Items**++- Determine if a `Job` that runs the `unpack` image is the best way to ++### Package Installation Configuration++Related Issues:+- [Env var support for packages to enable users to supply proxy+  credentials](https://github.com/crossplane/crossplane/issues/1638)++TODO(hasheddan): section on allowing some values to be passed from `Package` to+controller in `PackageRevision`++### API Changes++The first step towards supporting the upgrade of packages is by introducing the+`PackageRevision` type. The `PackageRevision` type will look very similar to the+current `Package`, but will add the following fields:++- `spec.infrastructureDefinitions`: list of `InfrastructureDefinition`s+  installed by the package.+- `spec.compositions`: list of `Composition`s installed by the package.+- `spec.state`: the desired state, which can be one of `Pending`, `Active`,+  `Stopped` or `Inactive`.++The following fields will be removed:++- `spec.packageType`: currently is one of `Provider`, `Stack`, `Application`, or+  `Addon`.+- `spec.permissionType`: currently is one of `Cluster` or `Namespace`.++In addition, `PackageRevision` will be a cluster-scoped type, rather than the+current namespace-scoped `Package`. The current schema for `Package` can be+viewed+[here](https://doc.crds.dev/github.com/crossplane/crossplane/packages.crossplane.io/Package/v1alpha1@v0.12.0).++In order to continue to support the current functionality of+`ClusterPackageInstall`, `PackageInstall`, `StackDefinition` the underlying+`Package` type will be renamed to `LegacyPackage`. The existing `Package` type+will be refactored to look like the current `ClusterPackageInstall` schema with+the following modifications.++This field will be added:++- `status.currentHash`: a string that is the `sha256` hash of the image in the+  current `PackageRevision`.+- `status.currentImage`: a string that is the name of the current+  `PackageRevision` image.++The following fields will be removed:++- `status.installJob`: a reference to the install `Job` for the+  `ClusterPackageInstall`. +- `status.packageRecord`: a reference to the `Package` that represents the+  installation unit for the `ClusterPackageInstall`. The `PackageRevision` is+  now discoverable by name at the cluster scope.++Before outlining the functionality of the new `Package` and `PackageRevision`+controllers, it is useful to establish the responsibilities of each.++The `Package` controller is responsible for:++- Managing the lifecycle of `PackageRevision`s+- Resolving dependencies between `Package`s++The `PackageRevision` controller is responsible for:++- Managing the lifecycle of any controller is installed by a package image+- Managing the creation and ownership of objects included in the package++### Dependency Management++Dependency management of packages can be boiled down to three questions:++1. Does the package have any dependencies?+2. Do those dependencies exist in my cluster?+3. Do compatible versions of those dependencies exist in my cluster?++If a `Package` is created that has no dependencies, it should always be+installed successfully unless it attempts to install objects that are already+present and owned in the cluster. However, most packages will depend on at least+on other package. Every package that does have dependencies should follow a+graph that terminates with leaf dependencies that in turn do not have any+dependencies themselves.++**Naive Approach**++Let's first consider a naive approach for dependency resolution to illustrate+the problem space. In the following example, a user is installing a package into+a fresh Kubernetes cluster.++User creates a `Package` with `spec.package: crossplane/pkg-a:v0.0.1`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: pkg-a+spec:+  package: crossplane/pkg-a:v0.0.1+```++When `crossplane/pkg-a:v0.0.1` is unpacked, it is shown to depend on+`crossplane/pkg-b`, `crossplane/pkg-c`, and `crossplane/pkg-d`++*Note: see [this documentation](https://nodesource.com/blog/semver-a-primer/)+for more information on the semver ranges used in the dependencies below.*++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: PackageRevision+metadata:+  name: <crossplane/pkg-a:v0.0.1-hash>+  ownerReferences:+  - apiVersion: packages.crossplane.io/v1alpha1+    controller: true+    blockOwnerDeletion: true+    kind: Package+    name: pkg-a+    uid: d9607e19-f88f-11e6-a518-42010a800195+spec:+  ...+  dependsOn:+    packages:+    - package: registry.upbound.io/crossplane/pkg-b:v1.*+    - package: registry.upbound.io/crossplane/pkg-b:v1.2.3-v1.3.5+    - package: registry.upbound.io/crossplane/pkg-b:~v2.5.7+  title: Package A+  ...+```++As mentioned before, this is a fresh Kubernetes cluster, so none of these+dependencies will have already been installed. Therefore, they need to be+installed before `pkg-a` can complete its installation. Three new `Package`s+would be created.++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-pkg-b # note: it will be suggested that users also name their packages in this format, but it cannot be strictly required+spec:+  package: registry.upbound.io/crossplane/pkg-b:v1.9.15 # latest v1.* version+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-pkg-c+spec:+  package: registry.upbound.io/crossplane/pkg-c:v1.3.5 # latest in range+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-pkg-d+spec:+  package: registry.upbound.io/crossplane/pkg-d:v2.5.22 # latest patch version+```++Each of these packages now must be unpacked and their dependencies evaluated as+well.++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: PackageRevision+metadata:+  name: <crossplane/pkg-b:1.9.15-hash>+  ownerReferences:+  - apiVersion: packages.crossplane.io/v1alpha1+    controller: true+    blockOwnerDeletion: true+    kind: Package+    name: registry.upbound.io-crossplane-pkg-b+    uid: d9607e19-f88f-11e6-a518-42010a800195+spec:+  ...+  dependsOn:+    packages:+    - package: registry.upbound.io/crossplane/provider-aws:v10.*+    - package: registry.upbound.io/crossplane/provider-gcp:v10.*+  title: Package B+  ...+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: PackageRevision+metadata:+  name: <crossplane/pkg-c:v1.3.5-hash>+  ownerReferences:+  - apiVersion: packages.crossplane.io/v1alpha1+    controller: true+    blockOwnerDeletion: true+    kind: Package+    name: registry.upbound.io-crossplane-pkg-c+    uid: d9607e19-f88f-11e6-a518-42010a800195+spec:+  ...+  dependsOn:+    packages:+    - package: registry.upbound.io/crossplane/provider-aws:v10.1.1+    - package: registry.upbound.io/crossplane/provider-gcp:v10.*+  title: Package C+  ...+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: PackageRevision+metadata:+  name: <crossplane/pkg-d:2.5.22-hash>+  ownerReferences:+  - apiVersion: packages.crossplane.io/v1alpha1+    controller: true+    blockOwnerDeletion: true+    kind: Package+    name: registry.upbound.io-crossplane-pkg-d+    uid: d9607e19-f88f-11e6-a518-42010a800195+spec:+  ...+  dependsOn:+    packages:+    - package: registry.upbound.io/crossplane/provider-aws:v10.1.*+    - package: registry.upbound.io/crossplane/provider-gcp:v10.3.*+  title: Package D+  ...+```++As seen in the `dependsOn` sections above, all three of these packages depend on+`provider-aws` and `provider-gcp`. Ideally, we would like for these controllers+to be able to reconcile independently of each other, each arriving at a stable+state with the correct versions of `provider-aws` and `provider-gcp` installed.+This is difficult because each of the three packages have different constraints+on their version requirements. For this example, let's say that `provider-gcp`'s+latest version is `v10.9.12` and `provider-aws`'s latest version is `v10.4.15`.+If `pkg-b` is reconciled first, we will see the following `Packages`:++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-provider-aws+spec:+  package: registry.upbound.io/crossplane/provider-aws:v10.4.15+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-provider-gcp+spec:+  package: registry.upbound.io/crossplane/provider-gcp:v10.9.12+```++These versions work great for `pkg-b`, but they will conflict with the required+versions for `pkg-c` and `pkg-d`. In fact, no matter which package is reconciled+first, there will be a conflict for another. What we want to do here is+establish consensus about what version should be installed. Let's look at more+sophisticated approach.++**Recursive Dependency Resolution**++Enter the `Module` type. It is a global representation of packages installed in+a cluster. The _`Package` controller_ watches for `Package`s and resolves+dependencies. Let's go back to the creation of the `pkg-a` `Package`. The+_`Package` controller_ would observe the `Package` creation, unpack the image,+check that there are no conflicts with the objects it is attempting to install,+then resolve its dependencies. Resolving dependencies would required recursively+unpacking all dependencies until encountering a conflict or reaching a leaf of+the dependency graph.++Original state of `Module`.++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Module+metadata:+  name: packages+```++*Reconcile 1 - Start Unpack*++1. Observe creation of `pkg-a`+2. Get `Module` and check if `pkg-a` exists (N)+3. Check for existing `Job` for `pkg-a` (N)+3. Create `Job` with name `pkg-a` in `crossplane-system` to unpack the image+4. Return with short wait++*Reconcile 2 - Add to Module*++1. Observe `pkg-a`+2. Get `Module` and check if `pkg-a` exists (N)+3. Check for existing `Job` for `pkg-a` (Y)+4. Read output of unpack `Job`+5. Check to see if all objects it desires to create are not already present and+   owned by another package (Y)+6. Add to `Module`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Module+metadata:+  name: packages+spec:+  packages:+  - name: pkg-a+    image: registry.upbound.io/crossplane/pkg-a:v0.0.1@abcdef+    dependencies:+    - registry.upbound.io/crossplane/pkg-b:v1.*+    - registry.upbound.io/crossplane/pkg-c:v1.2.3-v1.3.5+    - registry.upbound.io/crossplane/pkg-d:~v2.5.7+  compositions:+  - veryabstract.binfra.pkga.coolorg.io: pkg-a+  - veryabstract.cinfra.pkga.coolorg.io: pkg-a+  - veryabstract.dinfra.pkga.coolorg.io: pkg-a+```++7. Return with short wait++*Reconcile 3 - Start Resolution of Direct Dependencies*++1. Observe `pkg-a`+2. Get `Module` and check if `pkg-a` exists (Y)+3. Check if `pkg-a` is the a dependency of another package (N)+4. Check for existing `PackageRevision` owned by `pkg-a` (N)+5. Create `PackageRevision` with name equivalent to hash of image and state+   `Pending`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: PackageRevision+metadata:+  name: abcdef+  ownerReferences:+  - apiVersion: packages.crossplane.io/v1alpha1+    controller: true+    blockOwnerDeletion: true+    kind: Package+    name: pkg-a+    uid: d9607e19-f88f-11e6-a518-42010a800195+spec:+  compositions: # pkg-a is only a bundle of compositions+  - veryabstract.binfra.pkga.coolorg.io+  - veryabstract.cinfra.pkga.coolorg.io+  - veryabstract.dinfra.pkga.coolorg.io+  ...+```++6. Check to see if all dependencies in `PackageRevision` exist (N)+7. Create a `Package` for each dependency++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-pkg-b+spec:+  package: registry.upbound.io/crossplane/pkg-b:v1.9.15 # latest v1.* version+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-pkg-c+spec:+  package: registry.upbound.io/crossplane/pkg-c:v1.3.5 # latest in range+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-pkg-d+spec:+  package: registry.upbound.io/crossplane/pkg-d:v2.5.22 # latest patch version+```++8. Return with short wait++*Reconcile 4 - Observe Progress of Dependency Resolution*++1. Observe `pkg-a`+2. Get `Module` and check if `pkg-a` exists (Y)+3. Check if `pkg-a` is the a dependency of another package (N)+4. Check for existing `PackageRevision` owned by `pkg-a` (Y)+5. Check for dependencies in `Module` required by `PackageRevision` (Y)+  - Assuming those packages have now gotten to the stage where they have been+    added to `Module`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Module+metadata:+  name: packages+spec:+  packages:+  - name: pkg-a+    image: registry.upbound.io/crossplane/pkg-a:v0.0.1@abcdef+    dependencies:+    - registry.upbound.io/crossplane/pkg-b:v1.*+    - registry.upbound.io/crossplane/pkg-c:v1.2.3-v1.3.5+    - registry.upbound.io/crossplane/pkg-d:~v2.5.7+  - name: registry.upbound.io-crossplane-pkg-b+    image: registry.upbound.io/crossplane/pkg-b:1.9.15@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.*+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  - name: registry.upbound.io-crossplane-pkg-c+    image: registry.upbound.io/crossplane/pkg-c:v1.3.5@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.1.1+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  - name: registry.upbound.io-crossplane-pkg-d+    image: registry.upbound.io/crossplane/pkg-d:2.5.22@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.1.*+    - registry.upbound.io/crossplane/provider-gcp:v10.3.*+  compositions:+  - veryabstract.binfra.pkga.coolorg.io: pkg-a+  - veryabstract.cinfra.pkga.coolorg.io: pkg-a+  - veryabstract.dinfra.pkga.coolorg.io: pkg-a+  crds: # Each of the dependency packages (b,c,d) just bring a single InfrastructureDefinition+  - abstract.pkgb.coolorg.io: pkg-b+  - abstract.pkgc.coolorg.io: pkg-c+  - abstract.pkgd.coolorg.io: pkg-d+```++6. Get `PackageRevision`s for each dependency package+7. Are all of them `Active`? (N)+8. Return with short wait++*Reconcile 5 - Activate `PackageRevision`*++1. Observe `pkg-a`+2. Get `Module` and check if `pkg-a` exists (Y)+3. Check if `pkg-a` is the a dependency of another package (N)+4. Check for existing `PackageRevision` owned by `pkg-a` (Y)+5. Check for dependencies in `Module` required by `PackageRevision` (Y)++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Module+metadata:+  name: packages+spec:+  packages:+  - name: pkg-a+    image: registry.upbound.io/crossplane/pkg-a:v0.0.1@abcdef+    dependencies:+    - registry.upbound.io/crossplane/pkg-b:v1.*+    - registry.upbound.io/crossplane/pkg-c:v1.2.3-v1.3.5+    - registry.upbound.io/crossplane/pkg-d:~v2.5.7+  - name: registry.upbound.io-crossplane-pkg-b+    image: registry.upbound.io/crossplane/pkg-b:1.9.15@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.*+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  - name: registry.upbound.io-crossplane-pkg-c+    image: registry.upbound.io/crossplane/pkg-c:v1.3.5@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.1.1+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  - name: registry.upbound.io-crossplane-pkg-d+    image: registry.upbound.io/crossplane/pkg-d:2.5.22@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.1.*+    - registry.upbound.io/crossplane/provider-gcp:v10.3.*+  - name: registry.upbound.io-crossplane-provider-aws+    image: registry.upbound.io/crossplane/provider-aws:v10.1.1@abcdef+    # provider-aws has no dependencies+  - name: registry.upbound.io-crossplane-provider-gcp+    image: registry.upbound.io/crossplane/provider-aws:v10.3.32@abcdef # Latest patch on v10.3+    # provider-gcp has no dependencies+  compositions:+  - veryabstract.binfra.pkga.coolorg.io: pkg-a+  - veryabstract.cinfra.pkga.coolorg.io: pkg-a+  - veryabstract.dinfra.pkga.coolorg.io: pkg-a+  crds:+  - mycoolinfra.pkgb.coolorg.io/v1alpha1: registry.upbound.io-crossplane-pkg-b+  - mycoolinfra.pkgc.coolorg.io/v1alpha1: registry.upbound.io-crossplane-pkg-c+  - mycoolinfra.pkgd.coolorg.io/v1alpha1: registry.upbound.io-crossplane-pkg-d+  - rdsinstance.database.aws.crossplane.io/v1beta1: registry.upbound.io-crossplane-provider-aws # provider-aws brings only direct CRDs+  ...+  - cloudsqlinstance.database.gcp.crossplane.io/v1beta1: registry.upbound.io-crossplane-provider-gcp # provider-gcp brings only direct CRDs+  ...+```++6. Get `PackageRevision`s for each dependency package+7. Are all of them `Active` and `Ready`? (Y)+  - We will get to what magic led this to happen in a moment+8. Set the `PackageRevision` for `pkg-a` to `Active`+9. Return with long wait++**What about the magic? - Reconciling Package B**++In our last look at the `Module`, we saw that `provider-gcp` and `provider-aws`+had been installed successfully with the correct version that satisfied all 3+dependencies. To examine how this happened, let's take a look at the lifecycle+of `pkg-b`, assuming it was the first direct dependency to be reconciled.++*Reconcile 1 - Start Unpack*++1. Observe creation of `registry.upbound.io-crossplane-pkg-b`+2. Get `Module` and check if `registry.upbound.io-crossplane-pkg-b` exists (N)+3. Check for existing `Job` for `registry.upbound.io-crossplane-pkg-b` (N)+3. Create `Job` with name `registry.upbound.io-crossplane-pkg-b` in+   `crossplane-system` to unpack the image+4. Return with short wait++*Reconcile 2 - Add to Module*++1. Observe `registry.upbound.io-crossplane-pkg-b`+2. Get `Module` and check if `pkg-b` exists (N)+3. Check for existing `Job` for `pkg-b` (Y)+4. Read output of unpack `Job`+5. Check to see if all objects it desires to create are not already present and+   owned by another package (Y)+6. Add to `Module`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Module+metadata:+  name: packages+spec:+  packages:+  - name: pkg-a+    image: registry.upbound.io/crossplane/pkg-a:v0.0.1@abcdef+    dependencies:+    - registry.upbound.io/crossplane/pkg-b:v1.*+    - registry.upbound.io/crossplane/pkg-c:v1.2.3-v1.3.5+    - registry.upbound.io/crossplane/pkg-d:~v2.5.7+  - name: registry.upbound.io-crossplane-pkg-b+    image: registry.upbound.io/crossplane/pkg-b:1.9.15@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.*+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  compositions:+  - veryabstract.binfra.pkga.coolorg.io: pkg-a+  - veryabstract.cinfra.pkga.coolorg.io: pkg-a+  - veryabstract.dinfra.pkga.coolorg.io: pkg-a+  crds: # pkg-b brings a single InfraDef+  - abstract.pkgb.coolorg.io: registry.upbound.io-crossplane-pkg-b+```++7. Return with short wait++*Reconcile 3 - Wait For Presence of Sibling Dependencies*++This step is to ensure that the ++1. Observe `registry.upbound.io-crossplane-pkg-b`+2. Get `Module` and check if `registry.upbound.io-crossplane-pkg-b` exists (Y)+3. Check if `registry.upbound.io-crossplane-pkg-b` is a dependency of another+   package (Y)+4. Check and see if all sibling dependencies have been added to `Module` (N)+  - We can see that `registry.upbound.io-crossplane-pkg-c` is a sibling and has+    been added, but `registry.upbound.io-crossplane-pkg-d` has not++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Module+metadata:+  name: packages+spec:+  packages:+  - name: pkg-a+    image: registry.upbound.io/crossplane/pkg-a:v0.0.1@abcdef+    dependencies:+    - registry.upbound.io/crossplane/pkg-b:v1.*+    - registry.upbound.io/crossplane/pkg-c:v1.2.3-v1.3.5+    - registry.upbound.io/crossplane/pkg-d:~v2.5.7+  - name: registry.upbound.io-crossplane-pkg-b+    image: registry.upbound.io/crossplane/pkg-b:1.9.15@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.*+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  - name: registry.upbound.io-crossplane-pkg-c+    image: registry.upbound.io/crossplane/pkg-c:v1.3.5@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.1.1+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  compositions:+  - veryabstract.binfra.pkga.coolorg.io: pkg-a+  - veryabstract.cinfra.pkga.coolorg.io: pkg-a+  - veryabstract.dinfra.pkga.coolorg.io: pkg-a+  crds:+  - abstract.pkgb.coolorg.io: registry.upbound.io-crossplane-pkg-b+  - abstract.pkgc.coolorg.io: registry.upbound.io-crossplane-pkg-c+```++5. Return with short wait++*Reconcile 4 - Start Resolution of Direct Dependencies*++1. Observe `registry.upbound.io-crossplane-pkg-b`+2. Get `Module` and check if `registry.upbound.io-crossplane-pkg-b` exists (Y)+3. Check if `registry.upbound.io-crossplane-pkg-b` is the a dependency of+   another package (N)+4. Check for existing `PackageRevision` owned by+   `registry.upbound.io-crossplane-pkg-b` (N)+5. Create `PackageRevision` with name equivalent to hash of image and state+   `Pending`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: PackageRevision+metadata:+  name: abcdef+  ownerReferences:+  - apiVersion: packages.crossplane.io/v1alpha1+    controller: true+    blockOwnerDeletion: true+    kind: Package+    name: registry.upbound.io-crossplane-pkg-b+    uid: d9607e19-f88f-11e6-a518-42010a800195+spec:+  infrastructureDefinitions:+  - abstract.pkgb.coolorg.io+  ...+```++6. Check to see if all dependencies in `PackageRevision` exist (N)+  - Both `provider-gcp` and `provider-aws` are missing. However, now that all+    sibling dependencies have been added to the `Module`, we have enough+    information to determine the correct version that should be used for each+    shared dependency.+  - Note that there is nothing about Package B vs C and D that makes it carry+    out this operation. In fact, it could be that one package creates the+    `Package` for `provider-aws` and one creates the `Package` for+    `provider-gcp`. Because we have achieved consensus on what versions should+    exist, we don't care which creates them.+  - If there were conflicts in the dependencies, no `Package`s would be created,+    and the status would be propagated up the dependency tree (more on this+    later).+7. Create a `Package` for each dependency++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-provider-aws+spec:+  package: registry.upbound.io/crossplane/provider-aws:v10.1.1+```++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Package+metadata:+  name: registry.upbound.io-crossplane-provider-gcp+spec:+  package: registry.upbound.io/crossplane/provider-gcp:v10.3.32+```++8. Return with short wait++*Reconcile 5 - Observe Progress of Dependency Resolution*++1. Observe `registry.upbound.io-crossplane-pkg-b`+2. Get `Module` and check if `registry.upbound.io-crossplane-pkg-b` exists (Y)+3. Check if `registry.upbound.io-crossplane-pkg-b` is a dependency of another+   package (Y)+4. Check and see if all sibling dependencies have been added to `Module` (Y)+4. Check for existing `PackageRevision` owned by+   `registry.upbound.io-crossplane-pkg-b` (Y)+5. Check for dependencies in `Module` required by `PackageRevision` (Y)+  - Assuming those packages have now gotten to the stage where they have been+    added to `Module`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Module+metadata:+  name: packages+spec:+  packages:+  - name: pkg-a+    image: registry.upbound.io/crossplane/pkg-a:v0.0.1@abcdef+    dependencies:+    - registry.upbound.io/crossplane/pkg-b:v1.*+    - registry.upbound.io/crossplane/pkg-c:v1.2.3-v1.3.5+    - registry.upbound.io/crossplane/pkg-d:~v2.5.7+  - name: registry.upbound.io-crossplane-pkg-b+    image: registry.upbound.io/crossplane/pkg-b:1.9.15@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.*+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  - name: registry.upbound.io-crossplane-pkg-c+    image: registry.upbound.io/crossplane/pkg-c:v1.3.5@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.1.1+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  - name: registry.upbound.io-crossplane-pkg-d+    image: registry.upbound.io/crossplane/pkg-d:2.5.22@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.1.*+    - registry.upbound.io/crossplane/provider-gcp:v10.3.*+  - name: registry.upbound.io-crossplane-provider-aws+    image: registry.upbound.io/crossplane/provider-aws:v10.1.1@abcdef+    # provider-aws has no dependencies+  - name: registry.upbound.io-crossplane-provider-gcp+    image: registry.upbound.io/crossplane/provider-aws:v10.3.32@abcdef # Latest patch on v10.3+    # provider-gcp has no dependencies+  compositions:+  - veryabstract.binfra.pkga.coolorg.io: pkg-a+  - veryabstract.cinfra.pkga.coolorg.io: pkg-a+  - veryabstract.dinfra.pkga.coolorg.io: pkg-a+  crds:+  - mycoolinfra.pkgb.coolorg.io/v1alpha1: registry.upbound.io-crossplane-pkg-b+  - mycoolinfra.pkgc.coolorg.io/v1alpha1: registry.upbound.io-crossplane-pkg-c+  - mycoolinfra.pkgd.coolorg.io/v1alpha1: registry.upbound.io-crossplane-pkg-d+  - rdsinstance.database.aws.crossplane.io/v1beta1: registry.upbound.io-crossplane-provider-aws # provider-aws brings only direct CRDs+  ...+  - cloudsqlinstance.database.gcp.crossplane.io/v1beta1: registry.upbound.io-crossplane-provider-gcp # provider-gcp brings only direct CRDs+  ...+```++6. Get `PackageRevision`s for each dependency package+7. Are all of them `Active` and `Ready`? (Y)+  - `provider-aws` and `provider-gcp` did not have any dependencies, so they+    should have been installed and activated+8. Set the `PackageRevision` for `registry.upbound.io-crossplane-pkg-b` to+   `Active`+9. Return with long wait++**Updating a Package**++We have shown that creating a `Package` with dependencies in a fresh Kubernetes+cluster can be handled, but updating any packages in that chain must also be+supported. Let's first look at updating `pkg-a` to `v0.0.2`.++*Reconcile 1 - Observe Update*++1. Observe update of `pkg-a`+2. Get `Module` and check if `pkg-a` exists (Y)+3. Check if `image` in `Module` matches `spec.package` on `Package` (N)+4. Check if `Job` exists for `spec.package` of `pkg-a` (N)+5. Create new `Job` to unpack new `pkg-a` image+6. Return with short wait++*Reconcile 2 - Update Module*++1. Observe `pkg-a`+2. Get `Module` and check if `pkg-a` exists (Y)+3. Check if `image` in `Module` matches `spec.package` on `Package` (N)+4. Check if `Job` exists for `spec.package` of `pkg-a` (Y)+4. Read output of unpack `Job`+5. Check to see if objects to be installed are compatible for update (Y)+  - This includes checking that it does not introduce any new types that are+    already present and owned by another package+6. Check to see if any dependencies are updated, and if they are compatible with+   any existing dependencies (Y)+  - A scenario in which this would not be true would be if `pkg-a` updated its+    dependency on `pkg-b` to `v2.*` because the installed version is not+    compatible, and other packages or users may require the installed version.+7. Update `Module`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: Module+metadata:+  name: packages+spec:+  packages:+  - name: pkg-a+    image: registry.upbound.io/crossplane/pkg-a:v0.0.2@xyzlmnop # CHANGED+    dependencies:+    - registry.upbound.io/crossplane/pkg-b:v1.*+    - registry.upbound.io/crossplane/pkg-c:v1.3.3-v1.3.5 # CHANGED+    - registry.upbound.io/crossplane/pkg-d:~v2.5.7+  - name: registry.upbound.io-crossplane-pkg-b+    image: registry.upbound.io/crossplane/pkg-b:1.9.15@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.*+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  - name: registry.upbound.io-crossplane-pkg-c+    image: registry.upbound.io/crossplane/pkg-c:v1.3.5@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.1.1+    - registry.upbound.io/crossplane/provider-gcp:v10.*+  - name: registry.upbound.io-crossplane-pkg-d+    image: registry.upbound.io/crossplane/pkg-d:2.5.22@abcdef+    dependencies:+    - registry.upbound.io/crossplane/provider-aws:v10.1.*+    - registry.upbound.io/crossplane/provider-gcp:v10.3.*+  - name: registry.upbound.io-crossplane-provider-aws+    image: registry.upbound.io/crossplane/provider-aws:v10.1.1@abcdef+    # provider-aws has no dependencies+  - name: registry.upbound.io-crossplane-provider-gcp+    image: registry.upbound.io/crossplane/provider-aws:v10.3.32@abcdef # Latest patch on v10.3+    # provider-gcp has no dependencies+  compositions:+  - veryabstract.binfra.pkga.coolorg.io: pkg-a+  - veryabstract.cinfra.pkga.coolorg.io: pkg-a+  - veryabstract.dinfra.pkga.coolorg.io: pkg-a+  crds:+  - mycoolinfra.pkga.coolorg.io/v1alpha1: pkg-a # ADDED+  - mycoolinfra.pkgb.coolorg.io/v1alpha1: registry.upbound.io-crossplane-pkg-b+  - mycoolinfra.pkgc.coolorg.io/v1alpha1: registry.upbound.io-crossplane-pkg-c+  - mycoolinfra.pkgd.coolorg.io/v1alpha1: registry.upbound.io-crossplane-pkg-d+  - rdsinstance.database.aws.crossplane.io/v1beta1: registry.upbound.io-crossplane-provider-aws+  ...+  - cloudsqlinstance.database.gcp.crossplane.io/v1beta1: registry.upbound.io-crossplane-provider-gcp+  ...+```++8. Return with short wait++*Reconcile 3 - Create New Package Revision*++1. Observe `pkg-a`+2. Get `Module` and check if `pkg-a` exists (Y)+3. Check if `pkg-a` is the a dependency of another package (N)+4. Check for existing `PackageRevision` owned by `pkg-a` with name of current+   image hash (N)+5. Create `PackageRevision` with name equivalent to hash of image and state+   `Pending`++```yaml+apiVersion: packages.crossplane.io/v1alpha1+kind: PackageRevision+metadata:+  name: xyzlmnop+  ownerReferences:+  - apiVersion: packages.crossplane.io/v1alpha1+    controller: true+    blockOwnerDeletion: true+    kind: Package+    name: pkg-a+    uid: d9607e19-f88f-11e6-a518-42010a800195+spec:+  infrastructureDefinitions:+  - mycoolinfra.pkga.coolorg.io+  compositions:+  - veryabstract.binfra.pkga.coolorg.io+  - veryabstract.cinfra.pkga.coolorg.io+  - veryabstract.dinfra.pkga.coolorg.io+  ...+```++6. Return with short wait++*Reconcile 4 - Observe Progress of New Dependency Resolution*++1. Observe `pkg-a`+2. Get `Module` and check if `pkg-a` exists (Y)+3. Check if `pkg-a` is the a dependency of another package (N)+4. Check for existing `PackageRevision` owned by `pkg-a` with name of current+   image hash (Y)+5. Check for dependencies in `Module` required by `PackageRevision` (Y)+6. Get `PackageRevision`s for each dependency package+7. Are all of them `Active` and `Ready`? (Y)+  - No dependencies were updated+8. Set the `Active` `PackageRevision` for `registry.upbound.io-crossplane-pkg-b`+   to `Stopped`+9. Return with short wait++*Reconcile 5 - Start New PackageRevision*++1. Observe `pkg-a`+2. Get `Module` and check if `pkg-a` exists (Y)+3. Check if `pkg-a` is the a dependency of another package (N)+4. Check for existing `PackageRevision` owned by `pkg-a` with name of current+   image hash (Y)+5. Check for dependencies in `Module` required by `PackageRevision` (Y)+6. Get `PackageRevision`s for each dependency package+7. Are all of them `Active` and `Ready`? (Y)+  - No dependencies were updated+8. Check for any `Active` or `Stopped` `PackageRevision` for+   `registry.upbound.io-crossplane-pkg-b` (Y)+9. Check the `Stopped` `PackageRevision` to see if it has finished clean up (Y)+8. Set the `Pending` `PackageRevision` that matches name of current image hash+   for `registry.upbound.io-crossplane-pkg-b` to `Active`+9. Return with long wait++**Unwinding Dependencies**++A robust dependency management model must also resolve dependencies when+packages are uninstalled. Let's first take a look at how the uninstall of+`pkg-a` would be handled.++*Reconcile 1 - Observe Delete*++1. Observe delete of `pkg-a`+2. Get `Module` and check if `pkg-a` exists (Y)+3. Check if `pkg-a` is depended on by any other packages (N)+4. Check if any `PackageRevision`s exist (Y)+5. Delete all `PackageRevision`s for `Package`+6. Return++*Reconcile 2 - Remove Finalizer*++1. Observe delete of `pkg-a`+2. Get `Module` and check if `pkg-a` exists (Y)+3. Check if `pkg-a` is depended on by any other packages (N)+4. Check if any `PackageRevision`s exist (N)+  - If a `PackageRevision` has been deleted but is failing to clean up, that+    status should be propagated to `Package`+5. Remove finalizer from `Package`+6. Return++Note that deleting a parent package had no effect on the packages that we+installed as its dependencies. Furthermore, the intermediate packages (B, C, D)+are still able to resolve the correct version that is to be used for+`provider-aws` and `provider-gcp`. Users in the cluster could continue to+interact with each package in any way they were before. Let's now consider if+`pkg-a` had not been deleted, but a user tried to delete+`registry.upbound.io-crossplane-pkg-b`.++*Reconcile 1 - Observe Delete*++1. Observe delete of `registry.upbound.io-crossplane-pkg-b`+2. Get `Module` and check if `registry.upbound.io-crossplane-pkg-b` exists (Y)+3. Check if `registry.upbound.io-crossplane-pkg-b` is depended on by any other+   packages (Y)+4. Do not proceed and propagate invalid operation to status of+   `registry.upbound.io-crossplane-pkg-b`+6. Return++For more scenarios that are required to be supported by the dependency+management model, see the [Scenarios](#scenarios) section.++Action Items++- [ ] Explore the limitations of including all dependency information in a+  single instance of an object (i.e. `Module`)+  - [ ] How cumbersome would it be to "overflow" to other `Module` instances and+    build a single pane on reconcile?+  - [ ] Could `Module` state be an interface that allows for different backends+    other than an object in `etcd` to be used?++### Installation++Once a `PackageRevision` has been created, the _`Package` controller_ hands+control to the _`PackageRevision` controller_. The _`PackageRevision`+controller_ will operate similarly to the current implementation the _`Package`+controller_, with a few modifications. In total, the _`PackageRevision`_+controller will execute the following steps:++*`PackageRevision` Observed with State `Active`*++1. Observe `PackageRevision` with state `Active`.+3. Create all objects listed in `spec.customResourceDefinitions`,+   `spec.infrastructureDefinitions`, `spec.InfrastructurePublications`, and+   `spec.compositions`.+4. Label the objects to indicate that they are currently owned by this+   `PackageRevision`.+5. Ensure that all dependent (listed in `spec.dependsOn`) CRDs are present in+   the cluster.+   - This should have been checked by the _`Package` controller_, but we check+     here again.

What's the thinking behind doing this again?

hasheddan

comment created time in a month

Pull request review commentcrossplane/crossplane

[WIP] Package Manager refactor design doc

+# Packages v2+* Owner: Dan Mangum (@hasheddan)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane currently supports installing controllers and new CRDs into a+Kubernetes cluster that the Crossplane package manager is running in. While+there are many other packaging formats in the Kubernetes ecosystem, Crossplane+supports its own for the following reasons:++- Crossplane is [opinionated about the capabilities of a+  controller](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#allowed-resource-access)+  that can be installed to extend its functionality. For instance, controllers+  [may not run as+  root](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#package-deployment-privileges)+  or request cluster admin RBAC.+- Crossplane [allocates and aggregates various+  ClusterRoles](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#crossplane-clusterroles--rbac)+  to automatically provide permissions for users in the Kubernetes cluster to+  view / edit / create / delete CRDs installed by a package.+- Crossplane guards against conflicting CRDs being installed into a cluster.+- Crossplane adds [additional metadata to+  CRDs](https://github.com/crossplane/crossplane/blob/master/design/one-pager-stack-ui-metadata.md#crd-annotation-example)+  to provide additional context for displaying their configuration in a UI.+- Crossplane [adds labels to+  CRDs](https://github.com/crossplane/crossplane/blob/master/design/one-pager-stack-relationship-labels.md#example-wordpress-crdyaml-parented-by-stackinstall)+  in an attempt to establish parent-child relationships between CRDs.++In addition, the following unimplemented features are goals of the Crossplane+package manager:++- The ability to resolve dependencies listed in a package for CRDs that do not+  yet exist in the cluster.+- The ability for a user to upgrade installed package versions without manual+  intervention.++As part of these guarantees, Crossplane supports installing packages at both the+cluster (`ClusterPackageInstall`) and namespace scope (`PackageInstall`). The+primary difference between these two installation units is that the a+`ClusterPackageInstall` can only be installed once, and the controller's+`ServiceAccount` is bound to its `ClusterRole` with a `ClusterRoleBinding`,+meaning it can watch the resources for which ir requested RBAC at the cluster+scope. A `PackageInstall`, on the other hand, has its controller's+`ServiceAccount` bound to its `ClusterRole` with a `RoleBinding` that only+allows the controller to watch resources in its own namespace.++The advantage of a `PackageInstall` is that is theoretically allows for multiple+versions of the same package to be installed in a cluster at the same time+because its controller is only watching for objects in its own namespace.+However, because CRDs are cluster-scoped objects, there cannot be conflicting+CRDs installed by two different versions of a `PackageInstall`. So while the+goal of the `PackageInstall` is to enable multiple versions of a package to+exist simultaneously in a cluster, the limitations of CRDs makes it less+effective than desired in practice.++## Goals++The current package infrastructure, though well thought out, has become somewhat+convoluted and redundant with the introduction of+[composition](https://github.com/crossplane/crossplane/blob/master/design/design-doc-composition.md)+into the Crossplane ecosystem.++Composition solves the following goals originally intended to be addressed by a+`PackageInstall` and [template+stacks](https://github.com/crossplane/crossplane/blob/master/design/one-pager-template-stacks.md):++- Ability to publish infrastructure abstractions to specific namespaces.+  - The `PackageInstall` allowed packages to install a namespace-scoped CRD and+    a controller that only watched for resources in its namespace. Composition+    enables this by creating a namespace-scoped CRD using an+    `InfrastructurePublication` and restricting the namespace in which it can be+    provisioned using RBAC.+- Ability to create new packages without writing any code.+  - The templating controller enabled this by allowing a user to create a+    `behavior.yaml` file and automatically using the templating controller image+    (configured with the `behavior.yaml`) when one was not supplied in the+    package. Composition enables this without having to install a new+    controller. The composition controllers run as part of core Crossplane and+    dynamically reconcile new types that are created in response to the creation+    of an `InfrastructureDefinition` / `InfrastructurePublication`.++Because composition serves the same purposes as these existing packaging+primitives, the current packaging system can be narrowed in scope while also+supporting a similar set of use-cases. The immediate goals of this refactor are:++- Deprecation and subsequent removal of the `PackageInstall` and+  `StackDefinition` types.+- Standardization on a single `Package` type as a unit of installation.+- Support for upgrading a `Package` that is installed and running in a cluster.+- Support for native installation of composition types, including+  `InfrastructureDefinition`, `InfrastructurePublication`, and `Composition`.++The refactor will involve abandoning the following goals of the current model:++- Supporting running multiple versions of the same package in a cluster at the+  same time. This includes installing the same CRDs from multiple packages.++## Proposal++Related Issues:+- [Merge Package and+  PackageInstall](https://github.com/crossplane/crossplane/issues/1089)++This proposal broadly encompasses the following components:++- `Package`: a cluster-scoped CRD that represents the existence of a package in+  a cluster. Users create / update / delete `Package` instances to manage the+  packages present in their cluster.+- `Module`: a cluster-scoped CRD that represents the state of packages, CRDs and+  composition types in a cluster. Only one instance of the `Module` type exists+  in a cluster.+- `PackageRevision`: a cluster-scoped CRD that represents a version of a package+  that may or may not be active in a cluster. Many `PackageRevision` instances+  may exist for a single `Package`, but only one can be active at a given time.+  `PackageRevision` instances are named the `sha256` hash of the image they are+  represent.+- _`Package` controller_: a controller responsible for observing events on+  `Package` instances, creating new `PackageRevision` instances, and resolving+  dependencies.+- _`PackageRevision` controller_: a controller responsible for observing events+  on `PackageRevision` instances, installing new CRDs /+  `InfrastructureDefinition` / `InfrastructurePublication` / `Composition`+  objects, starting package controllers, and cleaning up these objects when a

Nit: I think this means "starting the packaged controllers" not "starting the Package" controller, but I read it as the latter at first, which caught me off guard.

hasheddan

comment created time in a month

Pull request review commentcrossplane/crossplane

[WIP] Package Manager refactor design doc

+# Packages v2+* Owner: Dan Mangum (@hasheddan)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane currently supports installing controllers and new CRDs into a+Kubernetes cluster that the Crossplane package manager is running in. While+there are many other packaging formats in the Kubernetes ecosystem, Crossplane+supports its own for the following reasons:++- Crossplane is [opinionated about the capabilities of a+  controller](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#allowed-resource-access)+  that can be installed to extend its functionality. For instance, controllers+  [may not run as+  root](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#package-deployment-privileges)+  or request cluster admin RBAC.+- Crossplane [allocates and aggregates various+  ClusterRoles](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#crossplane-clusterroles--rbac)+  to automatically provide permissions for users in the Kubernetes cluster to+  view / edit / create / delete CRDs installed by a package.+- Crossplane guards against conflicting CRDs being installed into a cluster.+- Crossplane adds [additional metadata to+  CRDs](https://github.com/crossplane/crossplane/blob/master/design/one-pager-stack-ui-metadata.md#crd-annotation-example)+  to provide additional context for displaying their configuration in a UI.+- Crossplane [adds labels to+  CRDs](https://github.com/crossplane/crossplane/blob/master/design/one-pager-stack-relationship-labels.md#example-wordpress-crdyaml-parented-by-stackinstall)+  in an attempt to establish parent-child relationships between CRDs.++In addition, the following unimplemented features are goals of the Crossplane+package manager:++- The ability to resolve dependencies listed in a package for CRDs that do not+  yet exist in the cluster.+- The ability for a user to upgrade installed package versions without manual+  intervention.++As part of these guarantees, Crossplane supports installing packages at both the+cluster (`ClusterPackageInstall`) and namespace scope (`PackageInstall`). The+primary difference between these two installation units is that the a+`ClusterPackageInstall` can only be installed once, and the controller's+`ServiceAccount` is bound to its `ClusterRole` with a `ClusterRoleBinding`,+meaning it can watch the resources for which ir requested RBAC at the cluster+scope. A `PackageInstall`, on the other hand, has its controller's+`ServiceAccount` bound to its `ClusterRole` with a `RoleBinding` that only+allows the controller to watch resources in its own namespace.++The advantage of a `PackageInstall` is that is theoretically allows for multiple+versions of the same package to be installed in a cluster at the same time+because its controller is only watching for objects in its own namespace.+However, because CRDs are cluster-scoped objects, there cannot be conflicting+CRDs installed by two different versions of a `PackageInstall`. So while the+goal of the `PackageInstall` is to enable multiple versions of a package to+exist simultaneously in a cluster, the limitations of CRDs makes it less+effective than desired in practice.++## Goals++The current package infrastructure, though well thought out, has become somewhat+convoluted and redundant with the introduction of+[composition](https://github.com/crossplane/crossplane/blob/master/design/design-doc-composition.md)+into the Crossplane ecosystem.++Composition solves the following goals originally intended to be addressed by a+`PackageInstall` and [template+stacks](https://github.com/crossplane/crossplane/blob/master/design/one-pager-template-stacks.md):++- Ability to publish infrastructure abstractions to specific namespaces.+  - The `PackageInstall` allowed packages to install a namespace-scoped CRD and+    a controller that only watched for resources in its namespace. Composition+    enables this by creating a namespace-scoped CRD using an+    `InfrastructurePublication` and restricting the namespace in which it can be+    provisioned using RBAC.+- Ability to create new packages without writing any code.+  - The templating controller enabled this by allowing a user to create a+    `behavior.yaml` file and automatically using the templating controller image+    (configured with the `behavior.yaml`) when one was not supplied in the+    package. Composition enables this without having to install a new+    controller. The composition controllers run as part of core Crossplane and+    dynamically reconcile new types that are created in response to the creation+    of an `InfrastructureDefinition` / `InfrastructurePublication`.++Because composition serves the same purposes as these existing packaging+primitives, the current packaging system can be narrowed in scope while also+supporting a similar set of use-cases. The immediate goals of this refactor are:++- Deprecation and subsequent removal of the `PackageInstall` and+  `StackDefinition` types.+- Standardization on a single `Package` type as a unit of installation.+- Support for upgrading a `Package` that is installed and running in a cluster.+- Support for native installation of composition types, including+  `InfrastructureDefinition`, `InfrastructurePublication`, and `Composition`.++The refactor will involve abandoning the following goals of the current model:++- Supporting running multiple versions of the same package in a cluster at the+  same time. This includes installing the same CRDs from multiple packages.++## Proposal++Related Issues:+- [Merge Package and+  PackageInstall](https://github.com/crossplane/crossplane/issues/1089)++This proposal broadly encompasses the following components:++- `Package`: a cluster-scoped CRD that represents the existence of a package in+  a cluster. Users create / update / delete `Package` instances to manage the+  packages present in their cluster.+- `Module`: a cluster-scoped CRD that represents the state of packages, CRDs and+  composition types in a cluster. Only one instance of the `Module` type exists+  in a cluster.+- `PackageRevision`: a cluster-scoped CRD that represents a version of a package+  that may or may not be active in a cluster. Many `PackageRevision` instances+  may exist for a single `Package`, but only one can be active at a given time.+  `PackageRevision` instances are named the `sha256` hash of the image they are+  represent.+- _`Package` controller_: a controller responsible for observing events on+  `Package` instances, creating new `PackageRevision` instances, and resolving+  dependencies.+- _`PackageRevision` controller_: a controller responsible for observing events+  on `PackageRevision` instances, installing new CRDs /+  `InfrastructureDefinition` / `InfrastructurePublication` / `Composition`+  objects, starting package controllers, and cleaning up these objects when a+  package is upgraded.++These types and controllers overlap in functionality with some of the existing+types and controllers that are currently present in the package manager+ecosystem. However, their implementation should not immediately break any of the+current features supported by the package manager. Removal of+`ClusterPackageInstall`, `PackageInstall`, and `StackDefinition`, as well as+refactor of `Package` and all controllers should follow a deprecation schedule+that allows users to move to the new format.++## Current Workflow++In order to accurately describe the steps required for implementing this+refactor. It is important to understand, at least at a high-level, the current+workflow the package manager uses for installing a `Package`. We will use a+`ClusterPackageInstall` for the example here as the namespace-scoped+`PackageInstall` is intended to be removed as part of this refactor.++1. User creates a `ClusterPackageInstall`.+2. The _`ClusterPackageInstall` controller_ observes the `ClusterPackageInstall`+   creation and creates a `Job` that runs a `Pod` with an initContainer running+   the image supplied on the `ClusterPackageInstall` and a container running the+   same image the package manager itself is running. The `initContainer` serves+   to only copy the contents of the package directory from the supplied image+   into a `Volume`. That `Volume` is then mounted on the `container` running the+   package manager image, which is executed with the `unpack`+   [argument](https://github.com/crossplane/crossplane/blob/a0d139f7cf269599ba916ed15af3fd68ffeabbdf/cmd/crossplane/package/unpack/unpack.go#L52).+3. The `unpack` container walks the filepath of the package directory, using the+   information provided to construct a `Package` object. It then+   [writes](https://github.com/crossplane/crossplane/blob/a0d139f7cf269599ba916ed15af3fd68ffeabbdf/pkg/packages/unpack.go#L204)+   the `Package` object and all CRDs in the package to+   [stdout](https://github.com/crossplane/crossplane/blob/a0d139f7cf269599ba916ed15af3fd68ffeabbdf/cmd/crossplane/package/unpack/unpack.go#L53).+4. The _`ClusterPackageInstall` controller_ waits for the `Job` to complete+   successfully before reading the logs from the `Pod`. When the `Job` is+   complete, it reads the the logs and creates all of the objects that were+   printed, making a [few modifications as well as annotating and labelling+   appropriately](https://github.com/crossplane/crossplane/blob/6fc50822fbf11a7d31f8a9dabde5c8948c3b36ac/pkg/controller/packages/install/installjob.go#L259).+5. The _`Package` controller_ observes the `Package` creation and assumes the+   following responsibilities:+  - Setting up all RBAC described in the [Background](#background) section.+  - Annotating all CRDs with the proper labels.+  - Creating the `ServiceAccount` for the controller `Deployment`, binding it to+    its `ClusterRole` and starting the controller (i.e. creating the+    `Deployment`).+  - Making any modifications, such as+    [syncing](https://github.com/crossplane/crossplane/blob/6fc50822fbf11a7d31f8a9dabde5c8948c3b36ac/pkg/controller/packages/pkg/pkg.go#L696)+    the `Secret` for the `ServiceAccount` that are required for running the+    controller in [host aware+    mode](https://github.com/crossplane/crossplane/blob/master/design/one-pager-host-aware-stack-manager.md).++The process for a `PackageInstall` is very similar, but the packages using the+templating controller have the additional step of first producing a+`StackDefinition` in the install `Job`, then translating that to a `Package` in+the [_`StackDefinition`+controller_](https://github.com/crossplane/crossplane/blob/6fc50822fbf11a7d31f8a9dabde5c8948c3b36ac/pkg/controller/packages/templates/stackdefinition_controller.go#L53).++## Implementation++This following sections describe the changes to the current package manager that+are required to support the goals listed above.++- [Package Format](#package-format)+- [API Changes](#api-changes)+- [Package Installation Configuration](#package-installation-configuration)+- [Dependency Management](#dependency-management)+- [Installation](#installation)+- [Scenarios](#scenarios)+- [Phase 1](#phase-1)+- [Phase 2](#phase-2)+- [Phase 3](#phase-3)++### Package Format++TODO(hasheddan): This section is still a WIP++A Crossplane package is an OCI image that contains a `.registry/` directory in+its filesystem. While the image is really just serving as a file format, it has+a few properties that make it advantageous:++- Allows integration with existing OCI image registries and tooling.+- Allows a package to also include a binary that it intends to be run after+  install is complete (i.e. the controller).+- Allows package metadata schema and filesystem format to change over time+  without having to modify the package manager because the logic can be included+  in the package image to allow it to "unpack" itself in a format that the+  package manager understands.++In reality, the second two properties are somewhat mutually exclusive.+Currently, a package does not unpack itself, but rather has its `.registry/`+directory copied into the filesystem of an unpacking container that provides the+logic for translating the manifests into a format that the package manager+understands. This means that the `ENTRYPOINT` of the package image can be+invoked directly in the `Deployment` that is created as part of installation,+but also means that the package author must conform to whatever "unpack" image+is being used, rather than building it into the package image itself.

Do I follow correctly that there's a bit of a disconnect between how we envisioned a package would work (separate image from the packaged thing, built in entrypoint that emits package metadata to stdout) and how it actually works today (packaged thing included in the package image, package metadata must be a file at a known path within the image)?

hasheddan

comment created time in a month

Pull request review commentcrossplane/crossplane

[WIP] Package Manager refactor design doc

+# Packages v2+* Owner: Dan Mangum (@hasheddan)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane currently supports installing controllers and new CRDs into a+Kubernetes cluster that the Crossplane package manager is running in. While+there are many other packaging formats in the Kubernetes ecosystem, Crossplane+supports its own for the following reasons:++- Crossplane is [opinionated about the capabilities of a+  controller](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#allowed-resource-access)+  that can be installed to extend its functionality. For instance, controllers+  [may not run as+  root](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#package-deployment-privileges)+  or request cluster admin RBAC.+- Crossplane [allocates and aggregates various+  ClusterRoles](https://github.com/crossplane/crossplane/blob/master/design/one-pager-packages-security-isolation.md#crossplane-clusterroles--rbac)+  to automatically provide permissions for users in the Kubernetes cluster to+  view / edit / create / delete CRDs installed by a package.+- Crossplane guards against conflicting CRDs being installed into a cluster.+- Crossplane adds [additional metadata to+  CRDs](https://github.com/crossplane/crossplane/blob/master/design/one-pager-stack-ui-metadata.md#crd-annotation-example)+  to provide additional context for displaying their configuration in a UI.+- Crossplane [adds labels to+  CRDs](https://github.com/crossplane/crossplane/blob/master/design/one-pager-stack-relationship-labels.md#example-wordpress-crdyaml-parented-by-stackinstall)+  in an attempt to establish parent-child relationships between CRDs.++In addition, the following unimplemented features are goals of the Crossplane+package manager:++- The ability to resolve dependencies listed in a package for CRDs that do not+  yet exist in the cluster.+- The ability for a user to upgrade installed package versions without manual+  intervention.++As part of these guarantees, Crossplane supports installing packages at both the+cluster (`ClusterPackageInstall`) and namespace scope (`PackageInstall`). The+primary difference between these two installation units is that the a+`ClusterPackageInstall` can only be installed once, and the controller's+`ServiceAccount` is bound to its `ClusterRole` with a `ClusterRoleBinding`,+meaning it can watch the resources for which ir requested RBAC at the cluster+scope. A `PackageInstall`, on the other hand, has its controller's+`ServiceAccount` bound to its `ClusterRole` with a `RoleBinding` that only+allows the controller to watch resources in its own namespace.++The advantage of a `PackageInstall` is that is theoretically allows for multiple+versions of the same package to be installed in a cluster at the same time+because its controller is only watching for objects in its own namespace.+However, because CRDs are cluster-scoped objects, there cannot be conflicting+CRDs installed by two different versions of a `PackageInstall`. So while the+goal of the `PackageInstall` is to enable multiple versions of a package to+exist simultaneously in a cluster, the limitations of CRDs makes it less+effective than desired in practice.++## Goals++The current package infrastructure, though well thought out, has become somewhat+convoluted and redundant with the introduction of+[composition](https://github.com/crossplane/crossplane/blob/master/design/design-doc-composition.md)+into the Crossplane ecosystem.++Composition solves the following goals originally intended to be addressed by a+`PackageInstall` and [template+stacks](https://github.com/crossplane/crossplane/blob/master/design/one-pager-template-stacks.md):++- Ability to publish infrastructure abstractions to specific namespaces.+  - The `PackageInstall` allowed packages to install a namespace-scoped CRD and+    a controller that only watched for resources in its namespace. Composition+    enables this by creating a namespace-scoped CRD using an+    `InfrastructurePublication` and restricting the namespace in which it can be+    provisioned using RBAC.+- Ability to create new packages without writing any code.+  - The templating controller enabled this by allowing a user to create a+    `behavior.yaml` file and automatically using the templating controller image+    (configured with the `behavior.yaml`) when one was not supplied in the+    package. Composition enables this without having to install a new+    controller. The composition controllers run as part of core Crossplane and+    dynamically reconcile new types that are created in response to the creation+    of an `InfrastructureDefinition` / `InfrastructurePublication`.++Because composition serves the same purposes as these existing packaging+primitives, the current packaging system can be narrowed in scope while also+supporting a similar set of use-cases. The immediate goals of this refactor are:++- Deprecation and subsequent removal of the `PackageInstall` and+  `StackDefinition` types.+- Standardization on a single `Package` type as a unit of installation.+- Support for upgrading a `Package` that is installed and running in a cluster.+- Support for native installation of composition types, including+  `InfrastructureDefinition`, `InfrastructurePublication`, and `Composition`.++The refactor will involve abandoning the following goals of the current model:++- Supporting running multiple versions of the same package in a cluster at the+  same time. This includes installing the same CRDs from multiple packages.++## Proposal++Related Issues:+- [Merge Package and+  PackageInstall](https://github.com/crossplane/crossplane/issues/1089)++This proposal broadly encompasses the following components:++- `Package`: a cluster-scoped CRD that represents the existence of a package in+  a cluster. Users create / update / delete `Package` instances to manage the+  packages present in their cluster.+- `Module`: a cluster-scoped CRD that represents the state of packages, CRDs and+  composition types in a cluster. Only one instance of the `Module` type exists+  in a cluster.+- `PackageRevision`: a cluster-scoped CRD that represents a version of a package+  that may or may not be active in a cluster. Many `PackageRevision` instances+  may exist for a single `Package`, but only one can be active at a given time.+  `PackageRevision` instances are named the `sha256` hash of the image they are+  represent.+- _`Package` controller_: a controller responsible for observing events on+  `Package` instances, creating new `PackageRevision` instances, and resolving+  dependencies.+- _`PackageRevision` controller_: a controller responsible for observing events+  on `PackageRevision` instances, installing new CRDs /+  `InfrastructureDefinition` / `InfrastructurePublication` / `Composition`+  objects, starting package controllers, and cleaning up these objects when a+  package is upgraded.++These types and controllers overlap in functionality with some of the existing+types and controllers that are currently present in the package manager+ecosystem. However, their implementation should not immediately break any of the+current features supported by the package manager. Removal of+`ClusterPackageInstall`, `PackageInstall`, and `StackDefinition`, as well as+refactor of `Package` and all controllers should follow a deprecation schedule+that allows users to move to the new format.++## Current Workflow++In order to accurately describe the steps required for implementing this+refactor. It is important to understand, at least at a high-level, the current+workflow the package manager uses for installing a `Package`. We will use a+`ClusterPackageInstall` for the example here as the namespace-scoped+`PackageInstall` is intended to be removed as part of this refactor.++1. User creates a `ClusterPackageInstall`.+2. The _`ClusterPackageInstall` controller_ observes the `ClusterPackageInstall`+   creation and creates a `Job` that runs a `Pod` with an initContainer running+   the image supplied on the `ClusterPackageInstall` and a container running the+   same image the package manager itself is running. The `initContainer` serves+   to only copy the contents of the package directory from the supplied image+   into a `Volume`. That `Volume` is then mounted on the `container` running the+   package manager image, which is executed with the `unpack`+   [argument](https://github.com/crossplane/crossplane/blob/a0d139f7cf269599ba916ed15af3fd68ffeabbdf/cmd/crossplane/package/unpack/unpack.go#L52).+3. The `unpack` container walks the filepath of the package directory, using the+   information provided to construct a `Package` object. It then+   [writes](https://github.com/crossplane/crossplane/blob/a0d139f7cf269599ba916ed15af3fd68ffeabbdf/pkg/packages/unpack.go#L204)+   the `Package` object and all CRDs in the package to+   [stdout](https://github.com/crossplane/crossplane/blob/a0d139f7cf269599ba916ed15af3fd68ffeabbdf/cmd/crossplane/package/unpack/unpack.go#L53).+4. The _`ClusterPackageInstall` controller_ waits for the `Job` to complete+   successfully before reading the logs from the `Pod`. When the `Job` is+   complete, it reads the the logs and creates all of the objects that were+   printed, making a [few modifications as well as annotating and labelling+   appropriately](https://github.com/crossplane/crossplane/blob/6fc50822fbf11a7d31f8a9dabde5c8948c3b36ac/pkg/controller/packages/install/installjob.go#L259).+5. The _`Package` controller_ observes the `Package` creation and assumes the+   following responsibilities:+  - Setting up all RBAC described in the [Background](#background) section.+  - Annotating all CRDs with the proper labels.+  - Creating the `ServiceAccount` for the controller `Deployment`, binding it to+    its `ClusterRole` and starting the controller (i.e. creating the+    `Deployment`).+  - Making any modifications, such as+    [syncing](https://github.com/crossplane/crossplane/blob/6fc50822fbf11a7d31f8a9dabde5c8948c3b36ac/pkg/controller/packages/pkg/pkg.go#L696)+    the `Secret` for the `ServiceAccount` that are required for running the+    controller in [host aware+    mode](https://github.com/crossplane/crossplane/blob/master/design/one-pager-host-aware-stack-manager.md).++The process for a `PackageInstall` is very similar, but the packages using the+templating controller have the additional step of first producing a+`StackDefinition` in the install `Job`, then translating that to a `Package` in+the [_`StackDefinition`+controller_](https://github.com/crossplane/crossplane/blob/6fc50822fbf11a7d31f8a9dabde5c8948c3b36ac/pkg/controller/packages/templates/stackdefinition_controller.go#L53).++## Implementation++This following sections describe the changes to the current package manager that+are required to support the goals listed above.++- [Package Format](#package-format)+- [API Changes](#api-changes)+- [Package Installation Configuration](#package-installation-configuration)+- [Dependency Management](#dependency-management)+- [Installation](#installation)+- [Scenarios](#scenarios)+- [Phase 1](#phase-1)+- [Phase 2](#phase-2)+- [Phase 3](#phase-3)++### Package Format++TODO(hasheddan): This section is still a WIP++A Crossplane package is an OCI image that contains a `.registry/` directory in+its filesystem. While the image is really just serving as a file format, it has+a few properties that make it advantageous:++- Allows integration with existing OCI image registries and tooling.+- Allows a package to also include a binary that it intends to be run after+  install is complete (i.e. the controller).+- Allows package metadata schema and filesystem format to change over time+  without having to modify the package manager because the logic can be included+  in the package image to allow it to "unpack" itself in a format that the+  package manager understands.++In reality, the second two properties are somewhat mutually exclusive.+Currently, a package does not unpack itself, but rather has its `.registry/`+directory copied into the filesystem of an unpacking container that provides the+logic for translating the manifests into a format that the package manager+understands. This means that the `ENTRYPOINT` of the package image can be+invoked directly in the `Deployment` that is created as part of installation,+but also means that the package author must conform to whatever "unpack" image+is being used, rather than building it into the package image itself.++OCI images are valuable in part because they are essentially an open playground+that allows you to do almost anything. In contrast, Crossplane packages are+extremely scoped and are opinionated about how you structure your image. For+this reason, it makes sense to abstract the image building process from the+package author, such that they only define their package contents and use the+Crossplane CLI to build and push the package.++*Note: this would not preclude someone from building the package image+themselves, but it would make it much easier for them to defer that+responsibility.*++As part of this abstraction, the ability to also include the binary that is+intended to be executed as the package's controller would be removed. The+original purpose of this functionality was to not require a package author to+build and push a controller image, then have to reference it in the package's+`install.yaml`, which defines the `Deployment` that should be created.++```go+// PackageSpec specifies the desired state of a Package.+type PackageSpec struct {+  AppMetadataSpec `json:",inline"`+  CRDs            CRDList         `json:"customresourcedefinitions,omitempty"`+  Controller      ControllerSpec  `json:"controller,omitempty"`+  Permissions     PermissionsSpec `json:"permissions,omitempty"`+}+```++```go+// AppMetadataSpec defines metadata about the package application+type AppMetadataSpec struct {+  Title         string            `json:"title,omitempty"`+  OverviewShort string            `json:"overviewShort,omitempty"`+  Overview      string            `json:"overview,omitempty"`+  Readme        string            `json:"readme,omitempty"`+  Version       string            `json:"version,omitempty"`+  Icons         []IconSpec        `json:"icons,omitempty"`+  Maintainers   []ContributorSpec `json:"maintainers,omitempty"`+  Owners        []ContributorSpec `json:"owners,omitempty"`+  Company       string            `json:"company,omitempty"`+  Category      string            `json:"category,omitempty"`+  Keywords      []string          `json:"keywords,omitempty"`+  Website       string            `json:"website,omitempty"`+  Source        string            `json:"source,omitempty"`+  License       string            `json:"license,omitempty"`++  // DependsOn is the list of CRDs that this package depends on. This data+  // drives the RBAC generation process.+  DependsOn []PackageInstallSpec `json:"dependsOn,omitempty"`++  // +kubebuilder:validation:Enum=Provider;Stack;Application;Addon+  PackageType string `json:"packageType,omitempty"`++  // +kubebuilder:validation:Enum=Cluster;Namespaced+  PermissionScope string `json:"permissionScope,omitempty"`+}+```++```go+// PackageInstallSpec specifies details about a request to install a package to+// Crossplane.+type PackageInstallSpec struct {+  PackageControllerOptions `json:",inline"`++  // Source is the domain name for the package registry hosting the package+  // being requested, e.g., registry.crossplane.io+  Source string `json:"source,omitempty"`++  // Package is the name of the package package that is being requested, e.g.,+  // myapp. Either Package or CustomResourceDefinition can be specified.+  Package string `json:"package,omitempty"`++  // CustomResourceDefinition is the full name of a CRD that is owned by the+  // package being requested. This can be a convenient way of installing a+  // package when the desired CRD is known, but the package name that contains+  // it is not known. Either Package or CustomResourceDefinition can be+  // specified.+  CustomResourceDefinition string `json:"crd,omitempty"`+}++// PackageControllerOptions allow for changes in the Package extraction and+// deployment controllers. These can affect how images are fetched and how+// Package derived resources are created.+type PackageControllerOptions struct {+  // ImagePullSecrets are named secrets in the same workspace that can be used+  // to fetch Packages from private repositories and to run controllers from+  // private repositories+  ImagePullSecrets []corev1.LocalObjectReference `json:"imagePullSecrets,omitempty"`++  // ImagePullPolicy defines the pull policy for all images used during+  // Package extraction and when running the Package controller.+  // https://kubernetes.io/docs/concepts/configuration/overview/#container-images+  ImagePullPolicy corev1.PullPolicy `json:"imagePullPolicy,omitempty"`++  // ServiceAccount options allow for changes to the ServiceAccount the+  // Package Manager creates for the Package's controller+  ServiceAccount *ServiceAccountOptions `json:"serviceAccount,omitempty"`+}+```++```go+// CRDList is the full list of CRDs that this package owns and depends on+type CRDList []metav1.TypeMeta+```++```go+// ControllerSpec defines the controller that implements the logic for a+// package, which can come in different flavors.+type ControllerSpec struct {+  // ServiceAccount options allow for changes to the ServiceAccount the+  // Package Manager creates for the Package's controller+  ServiceAccount *ServiceAccountOptions `json:"serviceAccount,omitempty"`++  Deployment *ControllerDeployment `json:"deployment,omitempty"`+}+```++```go+// PermissionsSpec defines the permissions that a package will require to+// operate.+type PermissionsSpec struct {+  Rules []rbac.PolicyRule `json:"rules,omitempty"`+}+```++**Action Items**++- Determine if a `Job` that runs the `unpack` image is the best way to 

Just reiterating that despite my poking at this, I'm totally fine sticking with the status quo (using a Job) for this quarter. I wonder if there could be simplifications to be had, but to my knowledge this isn't user facing or broken, and thus is a low priority to address.

hasheddan

comment created time in a month

pull request commentcrossplane/provider-aws

Add ELB resource

This looks pretty close to done to me; most of my comments were fairly minor. Can you please resolve any conversations that you feel are address and also comment here to let me know when it's ready for another look? I'll be offline next week, so I'd like to get this merged by Friday if possible.

sahil-lakhwani

comment created time in a month

Pull request review commentcrossplane/provider-aws

Add ELB resource

+/*+Copyright 2020 The Crossplane Authors.++Licensed under the Apache License, Version 2.0 (the "License");+you may not use this file except in compliance with the License.+You may obtain a copy of the License at++    http://www.apache.org/licenses/LICENSE-2.0++Unless required by applicable law or agreed to in writing, software+distributed under the License is distributed on an "AS IS" BASIS,+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+See the License for the specific language governing permissions and+limitations under the License.+*/++package elb++import (+	"context"++	"github.com/aws/aws-sdk-go-v2/aws"+	awselb "github.com/aws/aws-sdk-go-v2/service/elasticloadbalancing"+	"github.com/google/go-cmp/cmp"+	"github.com/pkg/errors"+	corev1 "k8s.io/api/core/v1"+	"k8s.io/apimachinery/pkg/types"+	ctrl "sigs.k8s.io/controller-runtime"+	"sigs.k8s.io/controller-runtime/pkg/client"++	runtimev1alpha1 "github.com/crossplane/crossplane-runtime/apis/core/v1alpha1"+	"github.com/crossplane/crossplane-runtime/pkg/event"+	"github.com/crossplane/crossplane-runtime/pkg/logging"+	"github.com/crossplane/crossplane-runtime/pkg/meta"+	"github.com/crossplane/crossplane-runtime/pkg/reconciler/managed"+	"github.com/crossplane/crossplane-runtime/pkg/resource"++	v1alpha1 "github.com/crossplane/provider-aws/apis/elasticloadbalancing/v1alpha1"+	awsv1alpha3 "github.com/crossplane/provider-aws/apis/v1alpha3"+	awsclients "github.com/crossplane/provider-aws/pkg/clients"+	"github.com/crossplane/provider-aws/pkg/clients/elasticloadbalancing/elb"+)++const (+	errUnexpectedObject = "The managed resource is not an ELB resource"++	errCreateELBClient   = "cannot create ELB client"+	errGetProvider       = "cannot get provider"+	errGetProviderSecret = "cannot get provider secret"++	errDescribe      = "failed to describe ELB with given name"+	errDescribeTags  = "failed to describe tags for ELB with given name"+	errMultipleItems = "retrieved multiple ELBs for the given name"+	errCreate        = "failed to create the ELB resource"+	errUpdate        = "failed to update ELB resource"+	errDelete        = "failed to delete the ELB resource"+	errSpecUpdate    = "cannot update spec of ELB custom resource"+)++// SetupELB adds a controller that reconciles ELBs.+func SetupELB(mgr ctrl.Manager, l logging.Logger) error {+	name := managed.ControllerName(v1alpha1.ELBGroupKind)+	return ctrl.NewControllerManagedBy(mgr).+		Named(name).+		For(&v1alpha1.ELB{}).+		Complete(managed.NewReconciler(mgr,+			resource.ManagedKind(v1alpha1.ELBGroupVersionKind),+			managed.WithExternalConnecter(&connector{kube: mgr.GetClient(), newClientFn: elb.NewClient}),+			managed.WithConnectionPublishers(),+			managed.WithLogger(l.WithValues("controller", name)),+			managed.WithRecorder(event.NewAPIRecorder(mgr.GetEventRecorderFor(name)))))+}++type connector struct {+	kube        client.Client+	newClientFn func(ctx context.Context, credentials []byte, region string, auth awsclients.AuthMethod) (elb.Client, error)+}++func (c *connector) Connect(ctx context.Context, mg resource.Managed) (managed.ExternalClient, error) {+	cr, ok := mg.(*v1alpha1.ELB)+	if !ok {+		return nil, errors.New(errUnexpectedObject)+	}++	p := &awsv1alpha3.Provider{}+	if err := c.kube.Get(ctx, types.NamespacedName{Name: cr.Spec.ProviderReference.Name}, p); err != nil {+		return nil, errors.Wrap(err, errGetProvider)+	}++	if aws.BoolValue(p.Spec.UseServiceAccount) {+		elbClient, err := c.newClientFn(ctx, []byte{}, p.Spec.Region, awsclients.UsePodServiceAccount)+		return &external{client: elbClient, kube: c.kube}, errors.Wrap(err, errCreateELBClient)+	}++	if p.GetCredentialsSecretReference() == nil {+		return nil, errors.New(errGetProviderSecret)+	}++	s := &corev1.Secret{}+	n := types.NamespacedName{Namespace: p.Spec.CredentialsSecretRef.Namespace, Name: p.Spec.CredentialsSecretRef.Name}+	if err := c.kube.Get(ctx, n, s); err != nil {+		return nil, errors.Wrap(err, errGetProviderSecret)+	}++	elbClient, err := c.newClientFn(ctx, s.Data[p.Spec.CredentialsSecretRef.Key], p.Spec.Region, awsclients.UseProviderSecret)+	return &external{client: elbClient, kube: c.kube}, errors.Wrap(err, errCreateELBClient)+}++type external struct {+	kube   client.Client+	client elb.Client+}++func (e *external) Observe(ctx context.Context, mgd resource.Managed) (managed.ExternalObservation, error) { // nolint:gocyclo+	cr, ok := mgd.(*v1alpha1.ELB)+	if !ok {+		return managed.ExternalObservation{}, errors.New(errUnexpectedObject)+	}++	response, err := e.client.DescribeLoadBalancersRequest(&awselb.DescribeLoadBalancersInput{+		LoadBalancerNames: []string{meta.GetExternalName(cr)},+	}).Send(ctx)+	if err != nil {+		return managed.ExternalObservation{}, errors.Wrap(resource.Ignore(elb.IsELBNotFound, err), errDescribe)+	}++	// in a successful response, there should be one and only one object+	if len(response.LoadBalancerDescriptions) != 1 {+		return managed.ExternalObservation{}, errors.New(errMultipleItems)+	}++	observed := response.LoadBalancerDescriptions[0]++	tagsResponse, err := e.client.DescribeTagsRequest(&awselb.DescribeTagsInput{+		LoadBalancerNames: []string{meta.GetExternalName(cr)},+	}).Send(ctx)+	if err != nil {+		return managed.ExternalObservation{}, errors.Wrap(resource.Ignore(elb.IsELBNotFound, err), errDescribeTags)+	}++	// update the CRD spec for any new values from provider+	current := cr.Spec.ForProvider.DeepCopy()+	elb.LateInitializeELB(&cr.Spec.ForProvider, &observed, tagsResponse.TagDescriptions[0].Tags)+	if !cmp.Equal(current, &cr.Spec.ForProvider) {+		if err := e.kube.Update(ctx, cr); err != nil {+			return managed.ExternalObservation{}, errors.Wrap(err, errSpecUpdate)+		}+	}++	cr.Status.SetConditions(runtimev1alpha1.Available())++	cr.Status.AtProvider = elb.GenerateELBObservation(observed)++	upToDate, err := elb.IsUpToDate(cr.Spec.ForProvider, observed, tagsResponse.TagDescriptions[0].Tags)+	if err != nil {+		return managed.ExternalObservation{}, errors.Wrap(err, errUpdate)+	}++	return managed.ExternalObservation{+		ResourceExists:   true,+		ResourceUpToDate: upToDate,+	}, nil+}++func (e *external) Create(ctx context.Context, mgd resource.Managed) (managed.ExternalCreation, error) {+	cr, ok := mgd.(*v1alpha1.ELB)+	if !ok {+		return managed.ExternalCreation{}, errors.New(errUnexpectedObject)+	}++	cr.Status.SetConditions(runtimev1alpha1.Creating())++	_, err := e.client.CreateLoadBalancerRequest(elb.GenerateCreateELBInput(meta.GetExternalName(cr),+		cr.Spec.ForProvider)).Send(ctx)++	return managed.ExternalCreation{}, errors.Wrap(err, errCreate)+}++func (e *external) Update(ctx context.Context, mgd resource.Managed) (managed.ExternalUpdate, error) { // // nolint:gocyclo+	cr, ok := mgd.(*v1alpha1.ELB)+	if !ok {+		return managed.ExternalUpdate{}, errors.New(errUnexpectedObject)+	}++	response, err := e.client.DescribeLoadBalancersRequest(&awselb.DescribeLoadBalancersInput{+		LoadBalancerNames: []string{meta.GetExternalName(cr)},+	}).Send(ctx)+	if err != nil {+		return managed.ExternalUpdate{}, errors.Wrap(resource.Ignore(elb.IsELBNotFound, err), errUpdate)+	}++	if len(response.LoadBalancerDescriptions) != 1 {+		return managed.ExternalUpdate{}, errors.New(errMultipleItems)+	}++	observed := response.LoadBalancerDescriptions[0]++	tagsResponse, err := e.client.DescribeTagsRequest(&awselb.DescribeTagsInput{+		LoadBalancerNames: []string{meta.GetExternalName(cr)},+	}).Send(ctx)+	if err != nil {+		return managed.ExternalUpdate{}, errors.Wrap(resource.Ignore(elb.IsELBNotFound, err), errDescribeTags)+	}++	patch, err := elb.CreatePatch(observed, cr.Spec.ForProvider, tagsResponse.TagDescriptions[0].Tags)

I found the names patch and CreatePatch a little unintuitive here. Do I follow correctly that:

  • patch is a JSON merge patch unmarshalled into a *v1alpha1.ELBParameters.
  • Because merge patches replace rather than update arrays all slice fields like patch.AvailabilityZones will contain the desired state of the entire array - i.e. they should match cr.Spec.ForProvider.AvailabilityZones?

It might be worth adding a comment to clarify this behaviour.

sahil-lakhwani

comment created time in a month

Pull request review commentcrossplane/provider-aws

Add ELB resource

+/*+Copyright 2020 The Crossplane Authors.++Licensed under the Apache License, Version 2.0 (the "License");+you may not use this file except in compliance with the License.+You may obtain a copy of the License at++    http://www.apache.org/licenses/LICENSE-2.0++Unless required by applicable law or agreed to in writing, software+distributed under the License is distributed on an "AS IS" BASIS,+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+See the License for the specific language governing permissions and+limitations under the License.+*/++package elb++import (+	"context"++	"github.com/aws/aws-sdk-go-v2/aws"+	awselb "github.com/aws/aws-sdk-go-v2/service/elasticloadbalancing"+	"github.com/google/go-cmp/cmp"+	"github.com/pkg/errors"+	corev1 "k8s.io/api/core/v1"+	"k8s.io/apimachinery/pkg/types"+	ctrl "sigs.k8s.io/controller-runtime"+	"sigs.k8s.io/controller-runtime/pkg/client"++	runtimev1alpha1 "github.com/crossplane/crossplane-runtime/apis/core/v1alpha1"+	"github.com/crossplane/crossplane-runtime/pkg/event"+	"github.com/crossplane/crossplane-runtime/pkg/logging"+	"github.com/crossplane/crossplane-runtime/pkg/meta"+	"github.com/crossplane/crossplane-runtime/pkg/reconciler/managed"+	"github.com/crossplane/crossplane-runtime/pkg/resource"++	v1alpha1 "github.com/crossplane/provider-aws/apis/elasticloadbalancing/v1alpha1"+	awsv1alpha3 "github.com/crossplane/provider-aws/apis/v1alpha3"+	awsclients "github.com/crossplane/provider-aws/pkg/clients"+	"github.com/crossplane/provider-aws/pkg/clients/elasticloadbalancing/elb"+)++const (+	errUnexpectedObject = "The managed resource is not an ELB resource"++	errCreateELBClient   = "cannot create ELB client"+	errGetProvider       = "cannot get provider"+	errGetProviderSecret = "cannot get provider secret"++	errDescribe      = "failed to describe ELB with given name"+	errDescribeTags  = "failed to describe tags for ELB with given name"+	errMultipleItems = "retrieved multiple ELBs for the given name"+	errCreate        = "failed to create the ELB resource"+	errUpdate        = "failed to update ELB resource"+	errDelete        = "failed to delete the ELB resource"+	errSpecUpdate    = "cannot update spec of ELB custom resource"

Nit: It would be nice if these error messages were phrased consistently, either "cannot ..." or "failed to ...". I believe "cannot" is more prevalent in our code base.

sahil-lakhwani

comment created time in a month

Pull request review commentcrossplane/provider-aws

Add ELB resource

+/*+Copyright 2020 The Crossplane Authors.++Licensed under the Apache License, Version 2.0 (the "License");+you may not use this file except in compliance with the License.+You may obtain a copy of the License at++    http://www.apache.org/licenses/LICENSE-2.0++Unless required by applicable law or agreed to in writing, software+distributed under the License is distributed on an "AS IS" BASIS,+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+See the License for the specific language governing permissions and+limitations under the License.+*/++package elb++import (+	"context"++	"github.com/aws/aws-sdk-go-v2/aws"+	awselb "github.com/aws/aws-sdk-go-v2/service/elasticloadbalancing"+	"github.com/google/go-cmp/cmp"+	"github.com/pkg/errors"+	corev1 "k8s.io/api/core/v1"+	"k8s.io/apimachinery/pkg/types"+	ctrl "sigs.k8s.io/controller-runtime"+	"sigs.k8s.io/controller-runtime/pkg/client"++	runtimev1alpha1 "github.com/crossplane/crossplane-runtime/apis/core/v1alpha1"+	"github.com/crossplane/crossplane-runtime/pkg/event"+	"github.com/crossplane/crossplane-runtime/pkg/logging"+	"github.com/crossplane/crossplane-runtime/pkg/meta"+	"github.com/crossplane/crossplane-runtime/pkg/reconciler/managed"+	"github.com/crossplane/crossplane-runtime/pkg/resource"++	v1alpha1 "github.com/crossplane/provider-aws/apis/elasticloadbalancing/v1alpha1"+	awsv1alpha3 "github.com/crossplane/provider-aws/apis/v1alpha3"+	awsclients "github.com/crossplane/provider-aws/pkg/clients"+	"github.com/crossplane/provider-aws/pkg/clients/elasticloadbalancing/elb"+)++const (+	errUnexpectedObject = "The managed resource is not an ELB resource"++	errCreateELBClient   = "cannot create ELB client"+	errGetProvider       = "cannot get provider"+	errGetProviderSecret = "cannot get provider secret"++	errDescribe      = "failed to describe ELB with given name"+	errDescribeTags  = "failed to describe tags for ELB with given name"+	errMultipleItems = "retrieved multiple ELBs for the given name"+	errCreate        = "failed to create the ELB resource"+	errUpdate        = "failed to update ELB resource"+	errDelete        = "failed to delete the ELB resource"+	errSpecUpdate    = "cannot update spec of ELB custom resource"+)++// SetupELB adds a controller that reconciles ELBs.+func SetupELB(mgr ctrl.Manager, l logging.Logger) error {+	name := managed.ControllerName(v1alpha1.ELBGroupKind)+	return ctrl.NewControllerManagedBy(mgr).+		Named(name).+		For(&v1alpha1.ELB{}).+		Complete(managed.NewReconciler(mgr,+			resource.ManagedKind(v1alpha1.ELBGroupVersionKind),+			managed.WithExternalConnecter(&connector{kube: mgr.GetClient(), newClientFn: elb.NewClient}),+			managed.WithConnectionPublishers(),+			managed.WithLogger(l.WithValues("controller", name)),+			managed.WithRecorder(event.NewAPIRecorder(mgr.GetEventRecorderFor(name)))))+}++type connector struct {+	kube        client.Client+	newClientFn func(ctx context.Context, credentials []byte, region string, auth awsclients.AuthMethod) (elb.Client, error)+}++func (c *connector) Connect(ctx context.Context, mg resource.Managed) (managed.ExternalClient, error) {+	cr, ok := mg.(*v1alpha1.ELB)+	if !ok {+		return nil, errors.New(errUnexpectedObject)+	}++	p := &awsv1alpha3.Provider{}+	if err := c.kube.Get(ctx, types.NamespacedName{Name: cr.Spec.ProviderReference.Name}, p); err != nil {+		return nil, errors.Wrap(err, errGetProvider)+	}++	if aws.BoolValue(p.Spec.UseServiceAccount) {+		elbClient, err := c.newClientFn(ctx, []byte{}, p.Spec.Region, awsclients.UsePodServiceAccount)+		return &external{client: elbClient, kube: c.kube}, errors.Wrap(err, errCreateELBClient)+	}++	if p.GetCredentialsSecretReference() == nil {+		return nil, errors.New(errGetProviderSecret)+	}++	s := &corev1.Secret{}+	n := types.NamespacedName{Namespace: p.Spec.CredentialsSecretRef.Namespace, Name: p.Spec.CredentialsSecretRef.Name}+	if err := c.kube.Get(ctx, n, s); err != nil {+		return nil, errors.Wrap(err, errGetProviderSecret)+	}++	elbClient, err := c.newClientFn(ctx, s.Data[p.Spec.CredentialsSecretRef.Key], p.Spec.Region, awsclients.UseProviderSecret)+	return &external{client: elbClient, kube: c.kube}, errors.Wrap(err, errCreateELBClient)+}++type external struct {+	kube   client.Client+	client elb.Client+}++func (e *external) Observe(ctx context.Context, mgd resource.Managed) (managed.ExternalObservation, error) { // nolint:gocyclo+	cr, ok := mgd.(*v1alpha1.ELB)+	if !ok {+		return managed.ExternalObservation{}, errors.New(errUnexpectedObject)+	}++	response, err := e.client.DescribeLoadBalancersRequest(&awselb.DescribeLoadBalancersInput{+		LoadBalancerNames: []string{meta.GetExternalName(cr)},+	}).Send(ctx)+	if err != nil {+		return managed.ExternalObservation{}, errors.Wrap(resource.Ignore(elb.IsELBNotFound, err), errDescribe)+	}++	// in a successful response, there should be one and only one object+	if len(response.LoadBalancerDescriptions) != 1 {+		return managed.ExternalObservation{}, errors.New(errMultipleItems)+	}++	observed := response.LoadBalancerDescriptions[0]++	tagsResponse, err := e.client.DescribeTagsRequest(&awselb.DescribeTagsInput{+		LoadBalancerNames: []string{meta.GetExternalName(cr)},+	}).Send(ctx)+	if err != nil {+		return managed.ExternalObservation{}, errors.Wrap(resource.Ignore(elb.IsELBNotFound, err), errDescribeTags)+	}++	// update the CRD spec for any new values from provider+	current := cr.Spec.ForProvider.DeepCopy()+	elb.LateInitializeELB(&cr.Spec.ForProvider, &observed, tagsResponse.TagDescriptions[0].Tags)+	if !cmp.Equal(current, &cr.Spec.ForProvider) {+		if err := e.kube.Update(ctx, cr); err != nil {+			return managed.ExternalObservation{}, errors.Wrap(err, errSpecUpdate)+		}+	}++	cr.Status.SetConditions(runtimev1alpha1.Available())++	cr.Status.AtProvider = elb.GenerateELBObservation(observed)++	upToDate, err := elb.IsUpToDate(cr.Spec.ForProvider, observed, tagsResponse.TagDescriptions[0].Tags)+	if err != nil {+		return managed.ExternalObservation{}, errors.Wrap(err, errUpdate)

Should this be errUpdate, or is the error in checking that the resource is up-to-date?

sahil-lakhwani

comment created time in a month

Pull request review commentcrossplane/provider-aws

Add ELB resource

+/*+Copyright 2020 The Crossplane Authors.++Licensed under the Apache License, Version 2.0 (the "License");+you may not use this file except in compliance with the License.+You may obtain a copy of the License at++    http://www.apache.org/licenses/LICENSE-2.0++Unless required by applicable law or agreed to in writing, software+distributed under the License is distributed on an "AS IS" BASIS,+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+See the License for the specific language governing permissions and+limitations under the License.+*/++package elb++import (+	"testing"++	"github.com/aws/aws-sdk-go-v2/aws"+	elb "github.com/aws/aws-sdk-go-v2/service/elasticloadbalancing"+	"github.com/google/go-cmp/cmp"++	"github.com/crossplane/provider-aws/apis/elasticloadbalancing/v1alpha1"+)++var (+	elbName           = "someELB"+	availabilityZones = []string{"us-east-1a", "us-east-1b"}+	listener          = v1alpha1.Listener{+		InstancePort:     80,+		InstanceProtocol: aws.String("HTTP"),+		LoadBalancerPort: 80,+		Protocol:         "HTTP",+	}+	elbListener = elb.Listener{+		InstancePort:     aws.Int64(80),+		InstanceProtocol: aws.String("HTTP"),+		LoadBalancerPort: aws.Int64(80),+		Protocol:         aws.String("HTTP"),+	}+	scheme  = "internal"+	elbTags = []elb.Tag{+		{+			Key:   aws.String("k1"),+			Value: aws.String("v1"),+		},+		{+			Key:   aws.String("k2"),+			Value: aws.String("v2"),+		},+	}+	tags = []v1alpha1.Tag{+		{+			Key:   "k1",+			Value: aws.String("v1"),+		},+		{+			Key:   "k2",+			Value: aws.String("v2"),+		},+	}+)++func elbParams(m ...func(*v1alpha1.ELBParameters)) *v1alpha1.ELBParameters {+	o := &v1alpha1.ELBParameters{+		AvailabilityZones: availabilityZones,+		Listeners:         []v1alpha1.Listener{listener},+	}++	for _, f := range m {+		f(o)+	}++	return o+}++func loadBalancer(m ...func(*elb.LoadBalancerDescription)) *elb.LoadBalancerDescription {+	o := &elb.LoadBalancerDescription{+		AvailabilityZones:    availabilityZones,+		ListenerDescriptions: []elb.ListenerDescription{{Listener: &elbListener}},+	}++	for _, f := range m {+		f(o)+	}++	return o+}++func TestLateInitializeELB(t *testing.T) {+	type args struct {+		spec *v1alpha1.ELBParameters+		in   elb.LoadBalancerDescription+		tags []elb.Tag+	}+	cases := map[string]struct {+		args args+		want *v1alpha1.ELBParameters+	}{+		"AllFilledNoDiff": {+			args: args{+				spec: elbParams(),+				in:   *loadBalancer(),+			},+			want: elbParams(),+		},+		"AllFilledExternalDiff": {+			args: args{+				spec: elbParams(),+				in: *loadBalancer(func(lb *elb.LoadBalancerDescription) {+					lb.Scheme = aws.String(scheme)+				}),+			},+			want: elbParams(func(p *v1alpha1.ELBParameters) {+				p.Scheme = aws.String(scheme)+			}),+		},+		"PartialFilled": {+			args: args{+				spec: elbParams(func(p *v1alpha1.ELBParameters) {+					p.AvailabilityZones = nil+				}),+				in: *loadBalancer(),+			},+			want: elbParams(),+		},+		"Tags": {+			args: args{+				spec: elbParams(),+				in:   *loadBalancer(),+				tags: elbTags,+			},+			want: elbParams(func(p *v1alpha1.ELBParameters) {+				p.Tags = tags+			}),+		},+	}++	for name, tc := range cases {+		t.Run(name, func(t *testing.T) {+			LateInitializeELB(tc.args.spec, &tc.args.in, tc.args.tags)+			if diff := cmp.Diff(tc.args.spec, tc.want); diff != "" {+				t.Errorf("LateInitializeSpec(...): -want, +got:\n%s", diff)+			}+		})+	}+}++func TestGenerateCreateRoleInput(t *testing.T) {+	cases := map[string]struct {+		in  v1alpha1.ELBParameters+		out elb.CreateLoadBalancerInput+	}{+		"FilledInput": {+			in: *elbParams(),+			out: elb.CreateLoadBalancerInput{+				LoadBalancerName:  &elbName,+				AvailabilityZones: availabilityZones,+				Listeners:         []elb.Listener{elbListener},+			},+		},+	}++	for name, tc := range cases {+		t.Run(name, func(t *testing.T) {+			r := GenerateCreateELBInput(elbName, tc.in)+			if diff := cmp.Diff(r, &tc.out); diff != "" {+				t.Errorf("GenerateNetworkObservation(...): -want, +got:\n%s", diff)+			}+		})+	}+}++func TestBuildELBListeners(t *testing.T) {+	cases := map[string]struct {+		in  []v1alpha1.Listener+		out []elb.Listener+	}{+		"FilledInput": {+			in:  []v1alpha1.Listener{listener},+			out: []elb.Listener{elbListener},+		},+	}++	for name, tc := range cases {+		t.Run(name, func(t *testing.T) {+			r := BuildELBListeners(tc.in)+			if diff := cmp.Diff(r, tc.out); diff != "" {+				t.Errorf("GenerateNetworkObservation(...): -want, +got:\n%s", diff)+			}+		})+	}+}++func TestBuildELBTags(t *testing.T) {+	cases := map[string]struct {+		in  []v1alpha1.Tag+		out []elb.Tag+	}{+		"FilledInput": {+			in:  tags,+			out: elbTags,+		},+	}++	for name, tc := range cases {+		t.Run(name, func(t *testing.T) {+			r := BuildELBTags(tc.in)+			if diff := cmp.Diff(r, tc.out); diff != "" {+				t.Errorf("GenerateNetworkObservation(...): -want, +got:\n%s", diff)+			}+		})+	}+}++func TestCreatePatch(t *testing.T) {

Could you add a test (either here or on IsUpToDate) to verify our expected behaviour when an array in the AWS struct matches an array in our struct, but with a different order? Is there a chance this could cause us to generate a patch that would simply reorder an otherwise matching array?

sahil-lakhwani

comment created time in a month

Pull request review commentcrossplane/provider-aws

Add ELB resource

+/*+Copyright 2020 The Crossplane Authors.++Licensed under the Apache License, Version 2.0 (the "License");+you may not use this file except in compliance with the License.+You may obtain a copy of the License at++    http://www.apache.org/licenses/LICENSE-2.0++Unless required by applicable law or agreed to in writing, software+distributed under the License is distributed on an "AS IS" BASIS,+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+See the License for the specific language governing permissions and+limitations under the License.+*/++package elb++import (+	"context"+	"encoding/json"+	"sort"+	"strings"++	"github.com/aws/aws-sdk-go-v2/aws"+	"github.com/aws/aws-sdk-go-v2/aws/awserr"+	elb "github.com/aws/aws-sdk-go-v2/service/elasticloadbalancing"+	"github.com/aws/aws-sdk-go-v2/service/elasticloadbalancing/elasticloadbalancingiface"+	"github.com/google/go-cmp/cmp"++	"github.com/crossplane/provider-aws/apis/elasticloadbalancing/v1alpha1"+	clients "github.com/crossplane/provider-aws/pkg/clients"+)++const (+	// ELBNotFound is the code returned by AWS when there is not ELB with a specified name.+	ELBNotFound = "LoadBalancerNotFound"

Does the AWS elb package have a constant we could reuse for this?

sahil-lakhwani

comment created time in a month

Pull request review commentcrossplane/provider-aws

Add ELB resource

+/*+Copyright 2020 The Crossplane Authors.++Licensed under the Apache License, Version 2.0 (the "License");+you may not use this file except in compliance with the License.+You may obtain a copy of the License at++    http://www.apache.org/licenses/LICENSE-2.0++Unless required by applicable law or agreed to in writing, software+distributed under the License is distributed on an "AS IS" BASIS,+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+See the License for the specific language governing permissions and+limitations under the License.+*/++package v1alpha1++import (+	"context"++	"github.com/crossplane/crossplane-runtime/pkg/reference"+	"sigs.k8s.io/controller-runtime/pkg/client"+)++// ResolveReferences of this RouteTable+func (mg *ELBAttachment) ResolveReferences(ctx context.Context, c client.Reader) error {+	r := reference.NewAPIResolver(c, mg)++	// Resolve spec.vpcID

Looks like this is actually resolving spec.forProvider.elbName.

sahil-lakhwani

comment created time in a month

Pull request review commentcrossplane/provider-aws

Add ELB resource

+/*+Copyright 2020 The Crossplane Authors.++Licensed under the Apache License, Version 2.0 (the "License");+you may not use this file except in compliance with the License.+You may obtain a copy of the License at++    http://www.apache.org/licenses/LICENSE-2.0++Unless required by applicable law or agreed to in writing, software+distributed under the License is distributed on an "AS IS" BASIS,+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+See the License for the specific language governing permissions and+limitations under the License.+*/++package v1alpha1++import (+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"++	runtimev1alpha1 "github.com/crossplane/crossplane-runtime/apis/core/v1alpha1"+)++// Tag defines a key value pair that can be attached to an ELB+type Tag struct {++	// The key of the tag.+	Key string `json:"key"`++	// The value of the tag.+	// +optional+	Value *string `json:"value,omitempty"`+}++// Listener represents the port binding(s) between the ELB and EC2 instances.+type Listener struct {++	// The port on which the instance is listening.+	InstancePort int64 `json:"instancePort"`++	// The protocol to use for routing traffic to instances: HTTP, HTTPS, TCP, or+	// SSL. If not specified, the value is same as for Protocol.+	// +optional+	InstanceProtocol *string `json:"instanceProtocol,omitempty"`++	// The port on which the load balancer is listening.+	LoadBalancerPort int64 `json:"loadBalancerPort"`++	// The load balancer transport protocol to use for routing: HTTP, HTTPS, TCP,+	// or SSL.+	Protocol string `json:"protocol"`++	// The Amazon Resource Name (ARN) of the server certificate.+	// +optional+	SSLCertificateID *string `json:"sslCertificateId,omitempty"`+}++// BackendServerDescription provides information about the instances attached to the ELB.+type BackendServerDescription struct {++	// The port on which the EC2 instance is listening.+	InstancePort int64 `json:"instancePort,omitempty"`++	// The names of the policies enabled for the EC2 instance.+	PolicyNames []string `json:"policyNames,omitempty"`+}++// HealthCheck defines the rules that the ELB uses to decide if an attached instance is healthy.+type HealthCheck struct {++	// The number of consecutive health checks successes required before moving+	// the instance to the Healthy state.+	HealthyThreshold int64 `json:"healthCheck"`++	// The approximate interval, in seconds, between health checks of an individual+	// instance.+	Interval int64 `json:"interval"`++	// The instance being checked.+	Target string `json:"target"`++	// The amount of time, in seconds, during which no response means a failed health+	// check.+	Timeout int64 `json:"timeout"`++	// The number of consecutive health check failures required before moving the+	// instance to the Unhealthy state.+	UnhealthyThreshold int64 `json:"unhealthyThreshold"`+}++// ELBParameters define the desired state of an AWS ELB.+type ELBParameters struct {+	// One or more Availability Zones from the same region as the load balancer.+	AvailabilityZones []string `json:"availabilityZones"`++	// Information about the health checks conducted on the load balancer.+	HealthCheck *HealthCheck `json:"healthCheck,omitempty"`++	// The listeners for this ELB.+	Listeners []Listener `json:"listeners"`++	// The type of a load balancer. Valid only for load balancers in a VPC.+	// +optional+	// +immutable+	Scheme *string `json:"scheme,omitempty"`++	// The IDs of the security groups to assign to the load balancer.+	// +optional+	SecurityGroups []string `json:"securityGroups,omitempty"`

I believe we model SGs - could this have reference and selector variants?

sahil-lakhwani

comment created time in a month

Pull request review commentcrossplane/provider-aws

Add ELB resource

+/*+Copyright 2020 The Crossplane Authors.++Licensed under the Apache License, Version 2.0 (the "License");+you may not use this file except in compliance with the License.+You may obtain a copy of the License at++    http://www.apache.org/licenses/LICENSE-2.0++Unless required by applicable law or agreed to in writing, software+distributed under the License is distributed on an "AS IS" BASIS,+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+See the License for the specific language governing permissions and+limitations under the License.+*/++package v1alpha1++import (+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"++	runtimev1alpha1 "github.com/crossplane/crossplane-runtime/apis/core/v1alpha1"+)++// Tag defines a key value pair that can be attached to an ELB+type Tag struct {++	// The key of the tag.+	Key string `json:"key"`++	// The value of the tag.+	// +optional+	Value *string `json:"value,omitempty"`+}++// Listener represents the port binding(s) between the ELB and EC2 instances.+type Listener struct {++	// The port on which the instance is listening.+	InstancePort int64 `json:"instancePort"`++	// The protocol to use for routing traffic to instances: HTTP, HTTPS, TCP, or+	// SSL. If not specified, the value is same as for Protocol.+	// +optional+	InstanceProtocol *string `json:"instanceProtocol,omitempty"`++	// The port on which the load balancer is listening.+	LoadBalancerPort int64 `json:"loadBalancerPort"`++	// The load balancer transport protocol to use for routing: HTTP, HTTPS, TCP,+	// or SSL.+	Protocol string `json:"protocol"`++	// The Amazon Resource Name (ARN) of the server certificate.+	// +optional+	SSLCertificateID *string `json:"sslCertificateId,omitempty"`+}++// BackendServerDescription provides information about the instances attached to the ELB.+type BackendServerDescription struct {++	// The port on which the EC2 instance is listening.+	InstancePort int64 `json:"instancePort,omitempty"`++	// The names of the policies enabled for the EC2 instance.+	PolicyNames []string `json:"policyNames,omitempty"`+}++// HealthCheck defines the rules that the ELB uses to decide if an attached instance is healthy.+type HealthCheck struct {++	// The number of consecutive health checks successes required before moving+	// the instance to the Healthy state.+	HealthyThreshold int64 `json:"healthCheck"`

I notice the JSON tag is quite different from the field name here. Is that intentional?

sahil-lakhwani

comment created time in a month

Pull request review commentcrossplane/provider-aws

Add ELB resource

+/*+Copyright 2020 The Crossplane Authors.++Licensed under the Apache License, Version 2.0 (the "License");+you may not use this file except in compliance with the License.+You may obtain a copy of the License at++    http://www.apache.org/licenses/LICENSE-2.0++Unless required by applicable law or agreed to in writing, software+distributed under the License is distributed on an "AS IS" BASIS,+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+See the License for the specific language governing permissions and+limitations under the License.+*/++package v1alpha1++import (+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"++	runtimev1alpha1 "github.com/crossplane/crossplane-runtime/apis/core/v1alpha1"+)++// Tag defines a key value pair that can be attached to an ELB+type Tag struct {++	// The key of the tag.+	Key string `json:"key"`++	// The value of the tag.+	// +optional+	Value *string `json:"value,omitempty"`+}++// Listener represents the port binding(s) between the ELB and EC2 instances.+type Listener struct {++	// The port on which the instance is listening.+	InstancePort int64 `json:"instancePort"`++	// The protocol to use for routing traffic to instances: HTTP, HTTPS, TCP, or+	// SSL. If not specified, the value is same as for Protocol.+	// +optional+	InstanceProtocol *string `json:"instanceProtocol,omitempty"`++	// The port on which the load balancer is listening.+	LoadBalancerPort int64 `json:"loadBalancerPort"`++	// The load balancer transport protocol to use for routing: HTTP, HTTPS, TCP,+	// or SSL.+	Protocol string `json:"protocol"`++	// The Amazon Resource Name (ARN) of the server certificate.+	// +optional+	SSLCertificateID *string `json:"sslCertificateId,omitempty"`+}++// BackendServerDescription provides information about the instances attached to the ELB.+type BackendServerDescription struct {++	// The port on which the EC2 instance is listening.+	InstancePort int64 `json:"instancePort,omitempty"`++	// The names of the policies enabled for the EC2 instance.+	PolicyNames []string `json:"policyNames,omitempty"`+}++// HealthCheck defines the rules that the ELB uses to decide if an attached instance is healthy.+type HealthCheck struct {++	// The number of consecutive health checks successes required before moving+	// the instance to the Healthy state.+	HealthyThreshold int64 `json:"healthCheck"`++	// The approximate interval, in seconds, between health checks of an individual+	// instance.+	Interval int64 `json:"interval"`++	// The instance being checked.+	Target string `json:"target"`++	// The amount of time, in seconds, during which no response means a failed health+	// check.+	Timeout int64 `json:"timeout"`++	// The number of consecutive health check failures required before moving the+	// instance to the Unhealthy state.+	UnhealthyThreshold int64 `json:"unhealthyThreshold"`+}++// ELBParameters define the desired state of an AWS ELB.+type ELBParameters struct {+	// One or more Availability Zones from the same region as the load balancer.+	AvailabilityZones []string `json:"availabilityZones"`++	// Information about the health checks conducted on the load balancer.+	HealthCheck *HealthCheck `json:"healthCheck,omitempty"`++	// The listeners for this ELB.+	Listeners []Listener `json:"listeners"`++	// The type of a load balancer. Valid only for load balancers in a VPC.+	// +optional+	// +immutable+	Scheme *string `json:"scheme,omitempty"`++	// The IDs of the security groups to assign to the load balancer.+	// +optional+	SecurityGroups []string `json:"securityGroups,omitempty"`++	// The IDs of the subnets in your VPC to attach to the load balancer. Specify+	// one subnet per Availability Zone specified in AvailabilityZones.+	// +optional+	Subnets []string `json:"subnets,omitempty"`

Similar to the above - I think we model AWS subnets so hopefully we can have reference and selector variants.

sahil-lakhwani

comment created time in a month

Pull request review commentcrossplane/provider-aws

Add MasterPasswordSecretRef to allow specifying a password for RDS

 type RDSInstanceParameters struct { 	// +optional 	MasterUsername *string `json:"masterUsername,omitempty"` +	// MasterPasswordSecretRef references the secret that contains the password used+	// in the creation of this RDS instance. If a reference is not given, a password+	// will be auto-generated.+	//+	// The key in the secret should be "password"

Should this be configurable?

muvaf

comment created time in a month

issue commentcrossplane/crossplane

Generate RBAC roles for InfraDefs and InfraPubs

Capturing some discussion from @lukeweber in https://crossplane.slack.com/archives/CEF5N8X08/p1594095385024800.

One thing for me is if we want to automate this if there’s no ability to auto-aggregate, how would the crossplane-admin get access to these new types?

I am suggesting that auto-aggregation happens. I imagine we'd create a ClusterRole named something like crossplane-admin. All of the more granular ClusterRoles we created would be labelled such that they aggregated up to the crossplane-admin ClusterRole. That said, I am suggesting that we don't automatically bind either the crossplane-admin ClusterRole or any of the more granular ClusterRoles to any subjects.

Unless you’re full cluster admin you can’t grant yourself permissions beyond what you have. Since these RBAC roles are creating new permissions, a user who didn’t have them wouldn’t be able to assign them.

Right - the cluster-admin user (or someone with similarly broad permissions) would need to bind the crossplane-admin role to some subject(s) at install time. These subjects would henceforth be responsible for granting other subjects either the crossplane-admin role (not recommended) or any of the more granular roles it aggregates. I imagine this experience being similar to GCP IAM roles, in which someone with the project owner role can grant any of the more granular roles like storage admin, network user, etc.

negz

comment created time in a month

issue openedcrossplane/crossplane

Generate RBAC roles for InfraDefs and InfraPubs

<!-- Thank you for helping to improve Crossplane!

Please be sure to search for open issues before raising a new one. We use issues for bug reports and feature requests. Please find us at https://slack.crossplane.io for questions, support, and discussion. -->

What problem are you facing?

<!-- Please tell us a little about your use case - it's okay if it's hypothetical! Leading with this context helps frame the feature request so we can ensure we implement it sensibly. ---> A Crossplane InfrastructureDefinition defines a new API - a new CR that Crossplane may reconcile by composing other CRs. An InfrastructurePublication publishes this API at the namespace scope such that applications can easily consume it. Currently these concepts are blissfully ignorant of RBAC; a platform builder must first define and publish their API, then create and bind RBAC roles such that folks can actually use that API. I suspect the RBAC roles platform builders would define would often (but not always) be fairly standard; perhaps an RBAC role that granted * verbs to the newly defined resources, and an RBAC role that granted read-only verbs to the newly defined resources.

How could Crossplane help solve your problem?

<!-- Let us know how you think Crossplane could help with your use case. --> Crossplane could automatically create (but not bind) RBAC roles when new APIs were defined. For example if an InfrastructureDefinition defined a new mysqlinstance.example.org composite resource, Crossplane might automatically create a mysqlinstance.example.org-editor and a mysqlinstance.example.org-viewer ClusterRole that granted RO and RW access to the newly defined API. Perhaps these roles could be aggregated up to a general crossplane-admin superuser role.

This would leave the decision of whether and how to actually grant these roles up to the folks deploying Crossplane, but would automate the need to actually author roles for a lot of people. It would emulate the experience of the "baked in" IAM roles that cloud providers provide, while still allowing platform builders to use RBAC to author and bind bespoke RBAC roles if the generated roles did not fit their needs.

created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a+platform team in an organization, you might want to publish a set of APIs that+are audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you+would also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster+    will be needed.++### RBAC++As we have two different Kubernetes clusters, there will be two separate security+domains and because of that, the `ServiceAccount`s in the remote cluster will+not be able to do any operation in the central cluster. Since the entity that+will execute the operations in the central cluster is the agent, we need to+define how we can deliver the necessary credentials to the agent so that it can+connect to the central cluster. Additionally, it will need some permissions to+execute operations in the remote cluster like `Secret` and+`CustomResourceDefinition` creation. We will look at how the agent will be+granted permissions to do its job in two separate domains with different+mechanisms.++#### Authenticating to The Central Cluster++In order to execute any operation, a `ServiceAccount` needs to exist in the+central cluster with appropriate permissions to read during pull and write+during push operations while synchronizing with central cluster. Since the agent+is running in the remote cluster, the credentials of this `ServiceAccount` will+be stored in a `Secret` in the remote cluster. Alongside the credentials, the+agent needs to know the namespace that it should sync to in the central cluster.++The easiest configuration would be the one where we specify the `Secret` and a+target namespace via installation commands of the agent. However, the namespace+pairing list could change dynamically after the installation. So, we need to be+able to supply the pairings after the installation. In order to specify which+namespace in the remote cluster should be synced to which namespace in the+central cluster, `Namespace` resource needs to be annotated.++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar

Would this annotation be mandatory, or optional?

Per our out of band conversation I suggest:

  • The annotation be optional, and the agent default to trying to sync to the same namespace in the central cluster if the annotation is omitted. So if a requirement was authored in remote namespace example the agent would assume that an example namespace also existed in the central cluster and would try to create a corresponding requirement there.
  • We avoid conflicts by using metadata.generateName when we create a corresponding requirement in the central cluster. That is, if the remote requirement is {namespace: example, name: coolreq} (and there were no annotations on the namespace) the corresponding central requirement would be {namespace: example, generateName: coolreq}.

We'd need to ensure there was some way for the remote requirement to regain control of the corresponding central requirement under this approach; presumably annotations creating a relationship at each end.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a+platform team in an organization, you might want to publish a set of APIs that+are audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you+would also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.
  * `InfrastructurePublication`s to discover what's available as published API.
muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a+platform team in an organization, you might want to publish a set of APIs that+are audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you+would also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster+    will be needed.++### RBAC++As we have two different Kubernetes clusters, there will be two separate security+domains and because of that, the `ServiceAccount`s in the remote cluster will+not be able to do any operation in the central cluster. Since the entity that+will execute the operations in the central cluster is the agent, we need to+define how we can deliver the necessary credentials to the agent so that it can+connect to the central cluster. Additionally, it will need some permissions to+execute operations in the remote cluster like `Secret` and+`CustomResourceDefinition` creation. We will look at how the agent will be+granted permissions to do its job in two separate domains with different+mechanisms.++#### Authenticating to The Central Cluster++In order to execute any operation, a `ServiceAccount` needs to exist in the+central cluster with appropriate permissions to read during pull and write+during push operations while synchronizing with central cluster. Since the agent+is running in the remote cluster, the credentials of this `ServiceAccount` will+be stored in a `Secret` in the remote cluster. Alongside the credentials, the+agent needs to know the namespace that it should sync to in the central cluster.++The easiest configuration would be the one where we specify the `Secret` and a+target namespace via installation commands of the agent. However, the namespace+pairing list could change dynamically after the installation. So, we need to be+able to supply the pairings after the installation. In order to specify which+namespace in the remote cluster should be synced to which namespace in the+central cluster, `Namespace` resource needs to be annotated.++```yaml

@muvaf should confirm, but we chatted earlier today and my understanding is that the annotation would be on the namespace on the remote (agent side) cluster.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you use a single cluster for+both Crossplane and all of your applications. However, there could be many cases+that you'd like to have multiple Kubernetes clusters for different purposes+like:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+  * All applications are subject to the same user management domain, i.e. same+    api-server. This could be managed to be safe, but it's not physically+    impossible to have a `ServiceAccount` in another namespace to have access to+    resources in your namespace. So, you wouldn't really trust to have+    production in one namespace and dev in the other.++When you use multiple clusters and deploy Crossplane to each one of them gives+you more flexibility but you'd lose the ability to see all the infrastructure+from one place as the clusters are physically isolated. An example case that+you'd like to have centralized infrastructure management is that as a+platform team in an organization, you might want to publish a set of APIs that+are audited and blessed for developers in the organization to use in order to+request infrastructure. Besides from that, there are other benefits like cost+overview from one place, tracking lost/forgotten resources etc. But you+would also want to enable application teams to self-serve and have certain level of+freedom to choose the infrastructure architecture they'd like using the building+blocks you've provided.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Making existing application bundles like Helm charts are harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` in order to deploy them to a cluster that's+    different than where Crossplane itself runs.+  * For example, you'd like change only the `StatefulSet` in the Helm chart with+    `MySQLInstanceRequirement` to use Crossplane for DB provisioning but you+    have to change each resource to be a template in a+    `KubernetesApplicationResource` if you'd like to use the `Secret` of+    `MySQLInstanceRequirement` in the remote cluster.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models provide+  because of that proxy and in some cases it could be functionally detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're convinced that the current+mechanics do not provide the experience users would like to have.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into+the remote cluster, and they request the infrastructure from a central cluster+and pull the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode+as well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster+    will be needed.++### RBAC++As we have two different Kubernetes clusters, there will be two separate security+domains and because of that, the `ServiceAccount`s in the remote cluster will+not be able to do any operation in the central cluster. Since the entity that+will execute the operations in the central cluster is the agent, we need to+define how we can deliver the necessary credentials to the agent so that it can+connect to the central cluster. Additionally, it will need some permissions to+execute operations in the remote cluster like `Secret` and+`CustomResourceDefinition` creation. We will look at how the agent will be+granted permissions to do its job in two separate domains with different+mechanisms.++#### Authenticating to The Central Cluster++In order to execute any operation, a `ServiceAccount` needs to exist in the+central cluster with appropriate permissions to read during pull and write+during push operations while synchronizing with central cluster. Since the agent+is running in the remote cluster, the credentials of this `ServiceAccount` will+be stored in a `Secret` in the remote cluster. Alongside the credentials, the+agent needs to know the namespace that it should sync to in the central cluster.++The easiest configuration would be the one where we specify the `Secret` and a+target namespace via installation commands of the agent. However, the namespace+pairing list could change dynamically after the installation. So, we need to be+able to supply the pairings after the installation. In order to specify which+namespace in the remote cluster should be synced to which namespace in the+central cluster, `Namespace` resource needs to be annotated.++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar+```+++We also need to supply which `Secret` in the cluster contains the credentials to+connect to that target namespace. There are two ways to do that:++* Annotate `Namespace` with the name of the `Secret`.+  * The `Secret` has to exist in that `Namespace`.+* If name of the `Secret` is not given as annotation on the `Namespace`, use the+  `Secret` name supplied as installation configuration.++Here is an example `foo` namespace that uses its own `Secret` to connect to+target `bar` namespace.++```yaml+apiVersion: v1+kind: Namespace+metadata:+  name: foo+  annotations:+    "agent.crossplane.io/target-namespace": bar+    "agent.crossplane.io/credentials-secret-name": my-sa-creds+---+apiVersion: v1+kind: Secret+metadata:+  name: my-sa-creds+  namespace: foo+type: Opaque+data:+  kubeconfig: MWYyZDFlMmU2N2Rm...+```++In case the annotation `agent.crossplane.io/credentials-secret-name` is not+given on `Namespace`, then the installation configuration will be used which+would look like the following:++```bash+helm install --namespace crossplane crossplane/agent --set credentials-secret-name=my-sa-creds+```++#### Authorization++##### Remote Cluster++Since there will be one agent for the whole cluster, its own mounted+`ServiceAccount` in that remote cluster needs to have read & write permissions+for all of the following kinds in the remote cluster listed below:++* `CustomResourceDefinition`+* `Composition`+* `InfrastructureDefinition`+* `InfrastructurePublication`+* `Secret`+* All `*Requirement` types++The last one is a bit tricky because the exact list of `*Requirement` types on+kind level is not known during installation and it's not static; new published+APIs should be available in the remote cluster dynamically. One option is to+allow agent to grant `Role` and `RoleBinding`s to itself as it creates the+necessary `CustomResourceDefinition`s in the remote cluster. However, an entity+that is able to grant permissions to itself could greatly increase the security+posture.++When you zoom out and think about how the `Role` will look like, in most of the+cases, it's something like the following:++```yaml+apiVersion: rbac.authorization.k8s.io/v1+kind: Role+metadata:+  name: crossplane-agent+  namespace: default+rules:+  # all kinds could be under one company/organization group+- apiGroups: ["acmeco.org"] +  resources: ["*"]+  verbs: ["*"]+  # or there could be logical groupings for different sets of requirements+- apiGroups: ["database.acmeco.org"]+  resources: ["*"]+  verbs: ["*"]+```++As you can see, it's either one group for all the new APIs or a logical group+for each set of APIs. In both cases, the frequency of the need to add a new+`apiGroup` is less than one would imagine thanks to the ability of allowing a+whole group; most frequently, the platform operators will be adding new kinds to+the existing groups.++In the light of this assumption, the initial approach will be that the `Role`+bound to the agent will be populated by a static list of the groups of the+requirement types during the installation like shown above and if a new group is+introduced, then an addition to this `Role` will be needed. A separate+controller to dynamically manage the `Role` is mentioned in the [Future+Considerations](#future-considerations) section.++##### Central Cluster++The `ServiceAccount` that will be created in the central cluster needs to have+the following permissions for agent to do its operations:++* Read+  * `CustomResourceDefinition`s+  * `InfrastructureDefinition`s+  * `InfrastructurePublication`s+  * `Composition`s+  * `Secret`s in the given namespace.+* Write+  * `*Requirement` types that you'd like to allow in given namespace.++### User Experience++In this section, a walkthrough from only a central cluster to a working+application will be shown step by step to show how the user experience will+shape.++Setup:+  * The Central Cluster: A Kubernetes cluster with Crossplane deployed &+    configured with providers and some published APIs.++Steps in the central cluster:+1. A new `Namespace` called `foo` is created.+1. A `ServiceAccount` called `agent1` in that namespace are created and+   necessary RBAC resources are created.++```yaml+# The ServiceAccount whose credentials will be copied over to remote cluster+# for agent to use to connect to the central cluster.+apiVersion: v1+kind: ServiceAccount+metadata:+  name: agent1+  namespace: foo+---+# To be able to create & delete requirements in the designated namespace of+# the central cluster.+apiVersion: rbac.authorization.k8s.io/v1+kind: Role+metadata:+  name: agent-requirement+  namespace: foo+rules:+- apiGroups: [""]+  resources: ["secrets"]+  verbs: ["*"]+- apiGroups: ["database.acmeco.org"]+  resources: ["*"]+  verbs: ["*"]+- apiGroups: ["network.acmeco.org"]+  resources: ["*"]+  verbs: ["*"]+---+apiVersion: rbac.authorization.k8s.io/v1+kind: RoleBinding+metadata:+  name: agent-requirement+  namespace: foo+subjects:+- kind: ServiceAccount+  name: agent1+  apiGroup: rbac.authorization.k8s.io+roleRef:+  kind: Role+  name: agent-requirement+  apiGroup: rbac.authorization.k8s.io+```++The YAML above includes what's necessary to sync a specific namespace. The YAML+below is for cluster-scoped resources that should be read by the agent and it's+generic to be used by all agents except the `subjects` list of+`ClusterRoleBinding`:++```yaml+# To be able to read the cluster-scoped resources.+apiVersion: rbac.authorization.k8s.io/v1+kind: ClusterRole+metadata:+  name: read-for-agents+rules:+- apiGroups: ["apiextensions.kubernetes.io"]+  resources: ["customresourcedefinitions"]+  verbs: ["get", "watch", "list"]+- apiGroups: ["apiextensions.crossplane.io"]+  resources:+  - infrastructuredefinitions+  - infrastructurepublications+  - compositions+  verbs: ["get", "watch", "list"]+---+apiVersion: rbac.authorization.k8s.io/v1+kind: ClusterRoleBinding+metadata:+  name: read-for-agents+subjects:+- kind: ServiceAccount+  name: agent1+  apiGroup: rbac.authorization.k8s.io+roleRef:+  kind: ClusterRole+  name: read-for-agents+  apiGroup: rbac.authorization.k8s.io+```++At this point, we have a `ServiceAccount` with all necessary permissions in our+central cluster. You can think of it like IAM user in the public cloud+providers; we have created it and allowed it to access a certain set of APIs.+Later on, its key will be used by the agent; just like provider-aws using the+key of an IAM User.++1. User provisions a network stack and a Kubernetes cluster in that private+   network through Crossplane (or via other methods, even kind cluster would+   work). This cluster will be used as remote cluster. We'll run the rest of the+   steps in that remote cluster.+1. The `Secret` that contains the kubeconfig of the `ServiceAccount` we created+   in the central cluster is replicated in the remote cluster with name+   `agent1-creds`.+1. The agent is installed into the remote cluster. The Helm package will have+   all the necessary RBAC resources but user will need to enter the API groups+   of the published types so that it can reconcile them in the remote cluster.+   `helm install crossplane/agent --set apiGroups=database.acmeco.org,network.acmeco.org --set credentials-secret-name=agent1-creds`+1. The `Namespace`s you'd like to sync are created with the necessary annotation+   like the following:++  ```yaml+  apiVersion: v1+  kind: Namespace+  metadata:+    name: foo+    annotations:+      "agent.crossplane.io/target-namespace": bar+  ```++At this point, the setup has been completed. Now, users can use the APIs in the+remote cluster just as if they are in the local mode. An example application to+deploy:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++Note that these steps show the bare-bones setup. Most of the steps can be made+easier with simple commands in Crossplane CLI and YAML templates you can edit &+use.++Users can discover what resources available to them and how they+can consume them from their remote cluster.++List the published APIs:+```bash+kubectl get infrastructurepublications+```++See what keys are included in the connection `Secret` of a specific API so that+they know what keys to use in mounting process:+```+kubectl get infrastructuredefinition mysqlinstance.database.acmeco.org -o yaml+```++See what `Composition`s are available to select from:+```+kubectl get compositions+```++## Future Considerations++### Additional Crossplane CLI Commands++Crossplane CLI can have simple commands to do most of the setup. For example,+with one command it should be possible to create the `ServiceAccount` in the+central cluster together with all of its RBAC resources. Also, agent+installation could be a very smooth process if we use Crossplane CLI instead of+Helm.++### RBAC Controller++A controller with its own types to manage the `ServiceAccount`s in the central+cluster could be a boost to UX. You'd create a custom resource that will result+in all RBAC resources that are needed for the agent to work with all the APIs in+the central cluster and write the credentials to a secret. Then the user can+copy this secret into their remote cluster and refer to it.++### Dynamic Updates to Agent's Role++In the remote cluster, we provide the `Role` that has the static set of+`apiGroup`s we'd like agent to be able to manage in the remote cluster. There+could be a controller that is separately installed and it could add new+`apiGroup`s as they appear as `InfrastructurePublication`s in the remote+cluster.++### Agent as an Executable for VMs++Instead of remote cluster, a VM can also use the infrastructure services that+the central cluster exposes. A different version of the agent could sync the+secrets to a file in the VM that can be used as credential file for VM+workloads. Maybe a small api-server shipped with that agent binary for+requirement requests?

Super interesting idea!

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,+you might want to see all infrastructure to be managed & represented from one+place and this has various benefits like common published APIs, cost overview,+tracking lost/forgotten resources etc. But you would want to enable application+teams to self-serve and have certain level of freedom about on a certain+infrastructure architecture they'd like to have their applications run inside.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.* You cannot interact with what you deploy directly,+    i.e. always have to use `KubernetesApplicationResource` as a proxy and that+    has its own set of challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Migration from a Helm chart that has a few `Deployment`s and a `StatefulSet`+  for DB is harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` as well as `StatefulSet` of DB with a DB+    requirement CR.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models+  provide because of that proxy and in some cases it could be functionally+  detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're more convinced that the thing+users want to manage from a single central point is infrastructure (including+policy, audit etc) but not necessarily applications.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into the+remote cluster, and they request the infrastructure from central cluster and pull+the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode as+well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster will+    be needed.++### RBAC++As we have two different Kubernetes cluster, there will be two separate security+domains and because of that, the `ServiceAccount`s in the remote cluster will+not be able to do any operation in the central cluster. Since the entity that+will execute the operations in the central cluster is the agent, we need to+define how we can deliver the necessary credentials to the agent so that it can+connect to the central cluster. Additionally, it will need some permissions to+execute operations in the remote cluster like `Secret` and+`CustomResourceDefinition` creation. We will look at how the agent will be+granted permissions to do its job in two separate domains with different+mechanisms.++#### Authenticating to The Central Cluster++In order to execute any operation, a `ServiceAccount` needs to exist in the+central cluster with appropriate permissions to read during pull and write+during push operations while synchronizing with central cluster. Since the agent+is running in the remote cluster, the credentials of this `ServiceAccount` will+be stored in a `Secret` in the remote cluster. Alongside the credentials, the+agent needs to know the namespace that it should sync to in the central cluster.++The way to have the agent find the right `Secret` to use in order to+authenticate to the central cluster for synchronization of a given requirement+will consist of two steps:++* Look for specific annotations on the requirement type to specify what `Secret`+  needs to be used:+  * `agent.crossplane.io/credentials-secret-name: sa-secret` for the full+    kubeconfig to connect to the central cluster.+  * `agent.crossplane.io/namespace: my-app-1` for the namespace it should sync

something less flexible and more predictable would be ideal

@muvaf Pushed a change to this PR since I left this comment that aligns pretty closely with what I was thinking. The relationship between remote and central resources are now defined by annotations at the namespace level rather than annotations on the CRs themselves. I suspect this is about the right level of flexibility and granularity.

muvaf

comment created time in a month

PR opened negz/provider-nop

Remove malindented and incorrect reference to GCP

<!-- Thank you for helping to improve Crossplane!

We strongly recommend you look through our contributor guide at https://git.io/fj2m9 if this is your first time opening a Crossplane pull request. You can find us in https://slack.crossplane.io/messages/dev if you need any help contributing. -->

Description of your changes

<!-- Briefly describe what this pull request does. Be sure to direct your reviewers' attention to anything that needs special consideration.

We love pull requests that resolve an open Crossplane issue. If yours does, you can uncomment the below line to indicate which issue your PR fixes, for example "Fixes #500":

Fixes # -->

This line was indented too far, causing the package to fail. It also referenced GCP, which is not what this provider is.

+0 -1

0 comment

1 changed file

pr created time in a month

create barnchnegz/provider-nop

branch : spatialawareness

created branch time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,+you might want to see all infrastructure to be managed & represented from one+place and this has various benefits like common published APIs, cost overview,+tracking lost/forgotten resources etc. But you would want to enable application+teams to self-serve and have certain level of freedom about on a certain+infrastructure architecture they'd like to have their applications run inside.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.* You cannot interact with what you deploy directly,+    i.e. always have to use `KubernetesApplicationResource` as a proxy and that+    has its own set of challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Migration from a Helm chart that has a few `Deployment`s and a `StatefulSet`+  for DB is harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` as well as `StatefulSet` of DB with a DB+    requirement CR.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models+  provide because of that proxy and in some cases it could be functionally+  detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're more convinced that the thing+users want to manage from a single central point is infrastructure (including+policy, audit etc) but not necessarily applications.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into the+remote cluster, and they request the infrastructure from central cluster and pull+the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode as+well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster will+    be needed.++### RBAC++As we have two different Kubernetes cluster, there will be two separate security

I still have a slight preference for something like "Kubernetes API servers", mostly because "Kubernetes cluster" implies to me a cluster that is handling container orchestration. Just a nitpick though; feel free to ignore.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,+you might want to see all infrastructure to be managed & represented from one+place and this has various benefits like common published APIs, cost overview,+tracking lost/forgotten resources etc. But you would want to enable application+teams to self-serve and have certain level of freedom about on a certain+infrastructure architecture they'd like to have their applications run inside.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.* You cannot interact with what you deploy directly,+    i.e. always have to use `KubernetesApplicationResource` as a proxy and that+    has its own set of challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Migration from a Helm chart that has a few `Deployment`s and a `StatefulSet`+  for DB is harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` as well as `StatefulSet` of DB with a DB+    requirement CR.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models+  provide because of that proxy and in some cases it could be functionally+  detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're more convinced that the thing+users want to manage from a single central point is infrastructure (including+policy, audit etc) but not necessarily applications.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into the+remote cluster, and they request the infrastructure from central cluster and pull+the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode as+well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.

That makes sense, but I do think we should create a record (i.e. IP) on the remote cluster so that folks can have the same kubectl get infrastructurepublications experience on remote clusters as they can on the central cluster.

muvaf

comment created time in a month

push eventcrossplane/provider-aws

Rahul M Chheda

commit sha 36a0ecb3b5bfdd6fb02f6467ed2eb10309548236

Added AWS SNS Notification API (#244) * Added SNS Subscriptions and SNS Topics Signed-off-by: Rahul M Chheda <rchheda@infracloud.io>

view details

push time in a month

PR merged crossplane/provider-aws

Added AWS SNS Notification API

<!-- Thank you for helping to improve Crossplane!

We strongly recommend you look through our contributor guide at https://git.io/fj2m9 if this is your first time opening a Crossplane pull request. You can find us in https://slack.crossplane.io/messages/dev if you need any help contributing. -->

Description of your changes

<!-- Briefly describe what this pull request does. Be sure to direct your reviewers' attention to anything that needs special consideration.

We love pull requests that resolve an open Crossplane issue. If yours does, you can uncomment the below line to indicate which issue your PR fixes, for example "Fixes #500":

Fixes # --> Fixes #175

- Added SNS Topic CRUD methods + tests
- Added SNS Subscription CRUD methods + tests

Checklist

<!-- Please run through the below readiness checklist. The first two items are relevant to every Crossplane pull request. --> I have:

  • [x] Run make reviewable to ensure this PR is ready for review.
  • [x] Ensured this PR contains a neat, self documenting set of commits.
  • [ ] Updated any relevant documentation, examples, or release notes.
  • [ ] Updated the dependencies in app.yaml to include any new role permissions.
+4505 -0

10 comments

25 changed files

rahulchheda

pr closed time in a month

issue closedcrossplane/provider-aws

Simple Notification Service support

What problem are you facing?

SNS support

How could Crossplane help solve your problem?

Add SNS managed resource

References

closed time in a month

infinitecompute

push eventnegz/provider-nop

Nic Cope

commit sha 3beba6c580c898664348dd6e649476749f2f2a48

Document provider-nop and add an example. Signed-off-by: Nic Cope <negz@rk0n.org>

view details

push time in a month

push eventnegz/provider-nop

Nic Cope

commit sha 55098c5fca5593a57c15f3c091c7fe319d57e02f

Document provider-nop and add an example. Signed-off-by: Nic Cope <negz@rk0n.org>

view details

push time in a month

push eventnegz/provider-nop

Nic Cope

commit sha 052fd637344519c54a394f435a242be5f04dd160

Document provider-nop and add an example. Signed-off-by: Nic Cope <negz@rk0n.org>

view details

push time in a month

push eventnegz/provider-nop

Nic Cope

commit sha 69eec070211c4925bd0116151abbbcc88a0f2ef4

Document provider-nop and add an example. Signed-off-by: Nic Cope <negz@rk0n.org>

view details

push time in a month

push eventnegz/provider-nop

Nic Cope

commit sha d20ba704f935a51e8d349e48af83f87529d0412c

Document provider-nop and add an example. Signed-off-by: Nic Cope <negz@rk0n.org>

view details

push time in a month

push eventnegz/provider-nop

Nic Cope

commit sha 38714b1f88607187a04843391dfcb9f988ac7c9e

Bootstrap by duplicating provider-gcp Signed-off-by: Nic Cope <negz@rk0n.org>

view details

Nic Cope

commit sha 2f2415be9a5ecdb6d7b27d8da4c7ab2079d431d3

Rename provider-gcp to provider-nop Signed-off-by: Nic Cope <negz@rk0n.org>

view details

Nic Cope

commit sha e5928690afe71a4c4cd8eca5420065e74dd1615e

Remove everything except IAM ServiceAccounts Signed-off-by: Nic Cope <negz@rk0n.org>

view details

Nic Cope

commit sha f05544e18f61e23fe3c05e8dceee30ef58e2618e

Cleanup some final references to GCP Signed-off-by: Nic Cope <negz@rk0n.org>

view details

Nic Cope

commit sha 47745f2cc3158ac0b52d4d7c50f38acf729b5380

Replace ServiceAccount with a NopResource Signed-off-by: Nic Cope <negz@rk0n.org>

view details

Nic Cope

commit sha 87b9541293c6f947435109ac48d680f76ff837aa

Document provider-nop and add an example. Signed-off-by: Nic Cope <negz@rk0n.org>

view details

push time in a month

pull request commentcrossplane/provider-aws

Added AWS IAMGroup with UserMembership, and PolicyAttachment

Thanks @rahulchheda and @sahil-lakhwani! Looks like there's a little more review to go. I'll wait until @sahil-lakhwani has approved this PR before we have a maintainer review it.

rahulchheda

comment created time in a month

pull request commentcrossplane/provider-aws

Add support for AWS Redshift v1alpha1

Thanks for reviewing @sahil-lakhwani! I'll wait for you to approve this PR before we have one of our maintainers review it.

arush-sal

comment created time in a month

pull request commentcrossplane/provider-aws

Add ELB resource

@sahil-lakhwani Thanks for marking this ready for review. @arush-sal I'm going to wait for you to approve this PR before we move forward with a review from the maintainers.

sahil-lakhwani

comment created time in a month

issue openedcrossplane/provider-gcp

GCP ServiceAccounts are missing package metadata

<!-- Thank you for helping to improve Crossplane!

Please be sure to search for open issues before raising a new one. We use issues for bug reports and feature requests. Please find us at https://slack.crossplane.io for questions, support, and discussion. -->

What happened?

<!-- Please let us know what behaviour you expected and how Crossplane diverged from that behaviour. --> At the time of writing there's no package metadata for GCP Service Accounts:

https://github.com/crossplane/provider-gcp/tree/c5b62363098170b0bcf2ff340ec99167c52370af/config/package/manifests/resources

created time in a month

create barnchnegz/provider-nop

branch : master

created branch time in a month

created repositorynegz/provider-nop

The world's most useful Crossplane provider

created time in a month

pull request commentcrossplane/crossplane

add make install command to install crds to cluster

I like this target @wonderflow, but I wonder if if should have a different name? make install implies to me that it would install all of Crossplane, not just the CRDs. Is the idea here that you could do something like make install run to avoid needing to use Helm to develop against an API server?

wonderflow

comment created time in a month

issue openedcrossplane/crossplane

Consider improving the UX of defaults in Compositions

<!-- Thank you for helping to improve Crossplane!

Please be sure to search for open issues before raising a new one. We use issues for bug reports and feature requests. Please find us at https://slack.crossplane.io for questions, support, and discussion. -->

What problem are you facing?

<!-- Please tell us a little about your use case - it's okay if it's hypothetical! Leading with this context helps frame the feature request so we can ensure we implement it sensibly. ---> In Crossplane a "composite resource" is a CR whose schema is defined by an InfrastructureDefinition. The composite resource is reconciled by composing one or more "composed resources". The relationship between the composite and the composed resources is configured by a Composition. The Composition specifies templates ("bases") for the composed resources, as well as a series of "patches" that patch each base by mapping fields from the composite resource.

It's currently possible to support default values, i.e. values that will be used when a field is omitted from the composite resource:

---
apiVersion: apiextensions.crossplane.io/v1alpha1
kind: InfrastructureDefinition
metadata:
  name: clusters.example.upbound.io
spec:
  crdSpecTemplate:
    group: example.upbound.io
    version: v1alpha1
    names:
      kind: Cluster
      listKind: ClusterList
      plural: clusters
      singular: cluster
    validation:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              # Note that this field is not marked required. When it is omitted
              # the default value specified in the Composition will be used.
              clusterVersion:
                description: GKE cluster version
                type: string
---
apiVersion: apiextensions.crossplane.io/v1alpha1
kind: Composition
metadata:
  name: clusters.example.upbound.io
spec:
  writeConnectionSecretsToNamespace: crossplane-system
  reclaimPolicy: Delete
  from:
    apiVersion: example.upbound.io/v1alpha1
    kind: Cluster
  to:
    - base:
        apiVersion: container.gcp.crossplane.io/v1beta1
        kind: GKECluster
        spec:
          providerRef:
            name: gcp-provider
          forProvider:
            # This version is the 'default' version for the Cluster. It will be used if the
            # author of the Cluster CR omits the spec.clusterVersion.
            initialClusterVersion: 1.15.11-gke.17
      patches:
        - fromFieldPath: spec.clusterVersion
          toFieldPath: spec.forProvider.initialClusterVersion
    - base:
        apiVersion: container.gcp.crossplane.io/v1alpha1
        kind: NodePool
        spec:
          forProvider:
            # This version is the 'default' version for the NodePool. It will be used if the
            # author of the Cluster CR omits the spec.clusterVersion.
            version: 1.15.11-gke.17
      patches:
        - fromFieldPath: "spec.clusterVersion"
          toFieldPath: "spec.forProvider.version"

In the above abridged example both the composed GKECluster and the NodePool will default to version 1.15.11-gke.17, unless the Cluster CR specifies a different value, which would then override the version in each base.

How could Crossplane help solve your problem?

<!-- Let us know how you think Crossplane could help with your use case. --> @ktintc has been experimenting with composition and expressed that she would prefer a UX that was closer to specifying a default value for the fromFieldPath: spec.clusterVersion rather than specifying defaults for each base. This has the advantage of not repeating the default value in multiple places in cases where that would be ready (i.e. specifying the same version in the GKECluster and NodePool bases).

Note that once we update to v1 CRDs (per https://github.com/crossplane/crossplane/issues/1435) we'll get something like this for free, because v1 CRDs support default values, and thus we'd be able to easily support that in our InfrastructureDefinitions, for example:

apiVersion: apiextensions.crossplane.io/v1alpha1
kind: InfrastructureDefinition
metadata:
  name: clusters.example.upbound.io
spec:
  crdSpecTemplate:
    group: example.upbound.io
    version: v1alpha1
    names:
      kind: Cluster
      listKind: ClusterList
      plural: clusters
      singular: cluster
    validation:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              clusterVersion:
                description: GKE cluster version
                type: string
                # A default value can now be specified in the InfraDef
                default: 1.15.11-gke.17

created time in a month

issue closedcrossplane/addon-oam-kubernetes-remote

Rename this repo to multi-cloud or sth rather than remote

Hi folks,

@szihai is playing with all the repos and guides recently. According to his feedback, the local and remote in the names of these repos are most confusing part for now. It's easy to mis-read as if I want to use Crossplane on minikube, I need to install local and if I want to use Crossplane on AlibabaCloud, I need to install remote.

I'd propose we change the names based on their functionalities:

addon-oam-kubernetes-local -> addon-oam-controllers
addon-oam-kubernetes-remote -> addon-oam-controllers-multicloud

Open for discussion of course.

/cc @negz @hasheddan

closed time in a month

resouer

issue closedcrossplane/addon-oam-kubernetes-remote

Add support for ContainerizedWorkload ConfigFiles

The ContainerizedWorkload OAM core workload type specifies support for writing configuration files to a Container. In the current Crossplane implementation, the config file values can either be supplied directly (via the Value field) or from a Secret (via the fromSecret field).

In the context of a Kubernetes Deployment, this looks like creating adding a Volume on the PodSpecTemplate, then a VolumeMount on the specific container. This works fine in the fromSecret case, as Secret is a valid volume source. However, providing data directly to a Volume is not supported, which means that when the Value field is defined, a workaround must be developed.

The most straightforward workaround would likely be to add the Value string as an annotation on the Pod then access the value in the Volume using the DownwardAPI volume source.

cc @negz @hongchaodeng @ryanzhang-oss

closed time in a month

hasheddan

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,+you might want to see all infrastructure to be managed & represented from one+place and this has various benefits like common published APIs, cost overview,+tracking lost/forgotten resources etc. But you would want to enable application+teams to self-serve and have certain level of freedom about on a certain+infrastructure architecture they'd like to have their applications run inside.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.

This section is really well written, and explained! Nice work.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,+you might want to see all infrastructure to be managed & represented from one+place and this has various benefits like common published APIs, cost overview,+tracking lost/forgotten resources etc. But you would want to enable application+teams to self-serve and have certain level of freedom about on a certain+infrastructure architecture they'd like to have their applications run inside.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.* You cannot interact with what you deploy directly,+    i.e. always have to use `KubernetesApplicationResource` as a proxy and that+    has its own set of challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Migration from a Helm chart that has a few `Deployment`s and a `StatefulSet`+  for DB is harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` as well as `StatefulSet` of DB with a DB+    requirement CR.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models+  provide because of that proxy and in some cases it could be functionally+  detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're more convinced that the thing+users want to manage from a single central point is infrastructure (including+policy, audit etc) but not necessarily applications.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into the+remote cluster, and they request the infrastructure from central cluster and pull+the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode as+well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster will+    be needed.++### RBAC++As we have two different Kubernetes cluster, there will be two separate security+domains and because of that, the `ServiceAccount`s in the remote cluster will+not be able to do any operation in the central cluster. Since the entity that+will execute the operations in the central cluster is the agent, we need to+define how we can deliver the necessary credentials to the agent so that it can+connect to the central cluster. Additionally, it will need some permissions to+execute operations in the remote cluster like `Secret` and+`CustomResourceDefinition` creation. We will look at how the agent will be+granted permissions to do its job in two separate domains with different+mechanisms.++#### Authenticating to The Central Cluster++In order to execute any operation, a `ServiceAccount` needs to exist in the+central cluster with appropriate permissions to read during pull and write+during push operations while synchronizing with central cluster. Since the agent+is running in the remote cluster, the credentials of this `ServiceAccount` will+be stored in a `Secret` in the remote cluster. Alongside the credentials, the+agent needs to know the namespace that it should sync to in the central cluster.++The way to have the agent find the right `Secret` to use in order to+authenticate to the central cluster for synchronization of a given requirement+will consist of two steps:++* Look for specific annotations on the requirement type to specify what `Secret`+  needs to be used:+  * `agent.crossplane.io/credentials-secret-name: sa-secret` for the full+    kubeconfig to connect to the central cluster.+  * `agent.crossplane.io/namespace: my-app-1` for the namespace it should sync+    requirements and their secrets.+* If no annotation is given, then use the defaults.+  * Installation procedure of the agent will allow to specify a default secret+    and namespace during installation.++By this override behavior, it is possible to supply credentials of another+namespace or `ServiceAccount` for a specific `*Requirement` resource. So, users+can target different namespaces in the central cluster from different namespaces+in the remote cluster, having the ability to have the granularity of RBAC and+synchronization as however they'd like.++> Though the override behavior also paves the way to target a completely+> different central cluster, too, which may break the setup since CRDs in that+> other central cluster could be different. We accept this risk as it would have+> to be a deliberate choice of the user.++#### Authorization++##### Remote Cluster++Since there will be one agent for the whole cluster, its own mounted+`ServiceAccount` in that remote cluster needs to have read & write permissions+for all of the following kinds in the remote cluster listed below:++* `CustomResourceDefinition`+* `Composition`+* `InfrastructureDefinition`+* `InfrastructurePublication`+* `Secret`+* All `*Requirement` types++The last one is a bit tricky because the exact list of `*Requirement` types on+kind level is not known during installation and it's not static; new published+APIs should be available in the remote cluster dynamically. One option is to+allow agent to grant `Role` and `RoleBinding`s to itself as it creates the+necessary `CustomResourceDefinition`s in the remote cluster. However, an entity+that is able to grant permissions to itself could greatly increase the security+posture.++When you zoom out and think about how the `Role` will look like, in most of the+cases, it's something like the following:++```yaml+apiVersion: rbac.authorization.k8s.io/v1+kind: Role+metadata:+  name: crossplane-agent+  namespace: default+rules:+  # all kinds could be under one company/organization group+- apiGroups: ["acmeco.org"] +  resources: ["*"]+  verbs: ["*"]+  # or there could be logical groupings for different sets of requirements+- apiGroups: ["database.acmeco.org"]+  resources: ["*"]+  verbs: ["*"]+```++As you can see, it's either one group for all the new APIs or a logical group+for each set of APIs. In both cases, the frequency of the need to add a new+`apiGroup` is less than one would imagine thanks to the ability of allowing a+whole group; most frequently, the platform operators will be adding new kinds to+the existing groups.++In the light of this assumption, the initial approach will be that the `Role`+bound to the agent will be populated by a static list of the groups of the+requirement types during the installation like shown above and if a new group is+introduced, then an addition to this `Role` will be needed. A separate+controller to dynamically manage the `Role` is mentioned in the [Future+Considerations](#future-considerations) section.

It seems like this problem might overlap slightly with the need for the central Crossplane to grant itself access to manage requirements. It seems like the ideas @hasheddan is exploring there (e.g. an RBAC controller that watches for IPs and grants access) could apply here too.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,+you might want to see all infrastructure to be managed & represented from one+place and this has various benefits like common published APIs, cost overview,+tracking lost/forgotten resources etc. But you would want to enable application+teams to self-serve and have certain level of freedom about on a certain+infrastructure architecture they'd like to have their applications run inside.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.* You cannot interact with what you deploy directly,+    i.e. always have to use `KubernetesApplicationResource` as a proxy and that+    has its own set of challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Migration from a Helm chart that has a few `Deployment`s and a `StatefulSet`+  for DB is harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` as well as `StatefulSet` of DB with a DB+    requirement CR.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models+  provide because of that proxy and in some cases it could be functionally+  detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're more convinced that the thing+users want to manage from a single central point is infrastructure (including+policy, audit etc) but not necessarily applications.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into the+remote cluster, and they request the infrastructure from central cluster and pull+the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode as+well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster will+    be needed.++### RBAC++As we have two different Kubernetes cluster, there will be two separate security+domains and because of that, the `ServiceAccount`s in the remote cluster will+not be able to do any operation in the central cluster. Since the entity that+will execute the operations in the central cluster is the agent, we need to+define how we can deliver the necessary credentials to the agent so that it can+connect to the central cluster. Additionally, it will need some permissions to+execute operations in the remote cluster like `Secret` and+`CustomResourceDefinition` creation. We will look at how the agent will be+granted permissions to do its job in two separate domains with different+mechanisms.++#### Authenticating to The Central Cluster++In order to execute any operation, a `ServiceAccount` needs to exist in the+central cluster with appropriate permissions to read during pull and write+during push operations while synchronizing with central cluster. Since the agent+is running in the remote cluster, the credentials of this `ServiceAccount` will+be stored in a `Secret` in the remote cluster. Alongside the credentials, the+agent needs to know the namespace that it should sync to in the central cluster.++The way to have the agent find the right `Secret` to use in order to+authenticate to the central cluster for synchronization of a given requirement+will consist of two steps:++* Look for specific annotations on the requirement type to specify what `Secret`+  needs to be used:+  * `agent.crossplane.io/credentials-secret-name: sa-secret` for the full+    kubeconfig to connect to the central cluster.+  * `agent.crossplane.io/namespace: my-app-1` for the namespace it should sync

I need to think about this a little more, but my feeling is that something less flexible and more predictable would be ideal - for example requiring that each remote namespace mapped to a unique namespace within the central cluster. e.g. Remote namespace foo in cluster a would always maps to namespace a-foo in the central cluster.

Another option could be for each remote cluster to be allocated one namespace in the central cluster, with each namespace of the remote cluster corresponding to a hierarchical namespace within the remote cluster's central cluster namespace.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,+you might want to see all infrastructure to be managed & represented from one+place and this has various benefits like common published APIs, cost overview,+tracking lost/forgotten resources etc. But you would want to enable application+teams to self-serve and have certain level of freedom about on a certain+infrastructure architecture they'd like to have their applications run inside.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.* You cannot interact with what you deploy directly,+    i.e. always have to use `KubernetesApplicationResource` as a proxy and that+    has its own set of challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Migration from a Helm chart that has a few `Deployment`s and a `StatefulSet`+  for DB is harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` as well as `StatefulSet` of DB with a DB+    requirement CR.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models+  provide because of that proxy and in some cases it could be functionally+  detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're more convinced that the thing+users want to manage from a single central point is infrastructure (including+policy, audit etc) but not necessarily applications.

To my earlier point, I'm not so sure it's that users don't want to manage apps centrally so much as that we've decided to focus on helping folks manage infra centrally (for now).

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,+you might want to see all infrastructure to be managed & represented from one+place and this has various benefits like common published APIs, cost overview,+tracking lost/forgotten resources etc. But you would want to enable application+teams to self-serve and have certain level of freedom about on a certain+infrastructure architecture they'd like to have their applications run inside.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.* You cannot interact with what you deploy directly,+    i.e. always have to use `KubernetesApplicationResource` as a proxy and that+    has its own set of challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Migration from a Helm chart that has a few `Deployment`s and a `StatefulSet`+  for DB is harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` as well as `StatefulSet` of DB with a DB+    requirement CR.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models+  provide because of that proxy and in some cases it could be functionally+  detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're more convinced that the thing+users want to manage from a single central point is infrastructure (including+policy, audit etc) but not necessarily applications.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into the+remote cluster, and they request the infrastructure from central cluster and pull+the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode as+well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.+* Push+  * `*Requirement` custom resources so that infrastructure can be requested.+    Read and write permissions in a specific namespace in the central cluster will+    be needed.++### RBAC++As we have two different Kubernetes cluster, there will be two separate security

Nit:

As we have two different Kubernetes APIs, there will be two separate security
muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,+you might want to see all infrastructure to be managed & represented from one+place and this has various benefits like common published APIs, cost overview,+tracking lost/forgotten resources etc. But you would want to enable application+teams to self-serve and have certain level of freedom about on a certain+infrastructure architecture they'd like to have their applications run inside.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.* You cannot interact with what you deploy directly,+    i.e. always have to use `KubernetesApplicationResource` as a proxy and that+    has its own set of challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Migration from a Helm chart that has a few `Deployment`s and a `StatefulSet`+  for DB is harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` as well as `StatefulSet` of DB with a DB+    requirement CR.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models+  provide because of that proxy and in some cases it could be functionally+  detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're more convinced that the thing+users want to manage from a single central point is infrastructure (including+policy, audit etc) but not necessarily applications.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into the+remote cluster, and they request the infrastructure from central cluster and pull

Have we defined what "local" and "remote" cluster mean to us in this context? Doing so could be useful to folks less ingrained in Crossplane's workload terminology.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,+you might want to see all infrastructure to be managed & represented from one+place and this has various benefits like common published APIs, cost overview,+tracking lost/forgotten resources etc. But you would want to enable application+teams to self-serve and have certain level of freedom about on a certain+infrastructure architecture they'd like to have their applications run inside.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.* You cannot interact with what you deploy directly,+    i.e. always have to use `KubernetesApplicationResource` as a proxy and that+    has its own set of challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Migration from a Helm chart that has a few `Deployment`s and a `StatefulSet`+  for DB is harder.+  * You actually need to change each and every element to be in a+    `KubernetesApplicationResource` as well as `StatefulSet` of DB with a DB+    requirement CR.+* Operation experience. This is related to the first point. If you have an app+  using OAM, or some other app model, then you always have that intermediate+  proxy of workload CRs. You're losing on some value that these models+  provide because of that proxy and in some cases it could be functionally+  detrimental.++Surely, it has its own advantages as well. For example, you can manage all of+your apps from single point via `KubernetesApplication`s targeting the right+clusters. But as we see more usage patterns, we're more convinced that the thing+users want to manage from a single central point is infrastructure (including+policy, audit etc) but not necessarily applications.++## Proposal++In order to preserve the central infrastructure management ability while+alleviating the issues above, we will change our approach from push-based one to+a pull-based one where applications, and their requirements are deployed into the+remote cluster, and they request the infrastructure from central cluster and pull+the necessary credentials.++Since having this logic in the applications themselves wouldn't be a good UX, we+will have an agent that you will need to deploy into your remote cluster for+doing the heavy-lifting for you. There are several technical problems to be+solved in order to make the experience smooth. Overall, the goal is that we want+to keep the UX of local mode for application operators while keeping the power+of centralized infrastructure management for platform operators. For reference,+here is an example local mode experience we'd like to have for the remote mode as+well:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: v1+kind: Pod+metadata:+  name: wp+  namespace: default+spec:+containers:+- name: wordpress+    image: "wordpress:4.6.1-apache"+    env:+    - name: WORDPRESS_DB_PASSWORD+    valueFrom:+        secretKeyRef:+          name: sql-creds+          key: password+```++The agent will be a Kubernetes controller running in the remote cluster and+watching all `*Requirement` types. Next sections will talk about the+implementation and user experience we'd like to have.++### Synchronization++In local mode, users directly interact with what's published by the platform+team, which is `*Requirement` types and consume the infrastructure by mounting+the secret whose name they specified on the `*Requirement` custom resource. To+keep this experience, we need to have a synchronization loop for the following+resources:++* Pull+  * `CustomResourceDefinition`s of all types that we want the applications to be+    able to manage and view:+    * Requirements that are published via `InfrastructurePublication`s. The+      source of truth will be the remote cluster.+    * `InfrastructureDefinition`, `InfrastructurePublication` and `Composition`.+  * `Composition`s to discover what's available to choose. CRs of this type will+    be read-only, and the source of truth will be the central cluster.+  * `InfrastructurePublication`s to discover what's availabe as published API.+    Read-only.+  * `InfrastructureDefinition`s to discover how the secret keys are shaped.+    Read-only.+  * `Secret`s that are result of the infrastructure that is provisioned so that+    it can be mounted to `Pod`s. Read-only.

Should we pull the CRD as well as the IDs and IPs? I imagine we could pull only the IDs and IPs, and have the agent create the appropriate CRDs in the remote cluster.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,+you might want to see all infrastructure to be managed & represented from one+place and this has various benefits like common published APIs, cost overview,+tracking lost/forgotten resources etc. But you would want to enable application+teams to self-serve and have certain level of freedom about on a certain+infrastructure architecture they'd like to have their applications run inside.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.* You cannot interact with what you deploy directly,+    i.e. always have to use `KubernetesApplicationResource` as a proxy and that+    has its own set of challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.+    * Let's say you deployed a `Pod` and `spec.node` is late-initialized. You+      will not see that because we only propagate the status back, not spec+      because the template is not strong-typed and it's hard to differentiate+      between user's actual desired spec and what's only a late-inited value.+    * If you have an element in an array that is late-inited or some elements+      are added after the creation, `PATCH` command will replace the whole array+      with what you got in your template. If the type is well-constructed to+      provide its own merge mechanics, this could be avoidable but that is+      usually not the case. For example, in some cases an element of an array in+      spec is late-inited for bookkeeping the IP and removing this causes its+      controller to provision new ones each time.+* Migration from a Helm chart that has a few `Deployment`s and a `StatefulSet`+  for DB is harder.

I think this is supposed to be a general example, but it reads a little like a specific use case that doesn't work. Could we make it more obvious that this is an illustration of a class of issues?

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,+you might want to see all infrastructure to be managed & represented from one+place and this has various benefits like common published APIs, cost overview,+tracking lost/forgotten resources etc. But you would want to enable application+teams to self-serve and have certain level of freedom about on a certain+infrastructure architecture they'd like to have their applications run inside.++What we need to do is to enable a platform team to have this central+infrastructure management ability while not imposing hard restrictions on+application teams. In the end, the goal of the platform teams is to increase the+velocity of development while keeping everything manageable.++### Current Approach++Crossplane has several features built to address this use case and the main+driver is the workload API which consists of `KubernetesApplication` and+`KubernetesApplicationResource` CRDs. The gist of how it works is that users+would need to provide the Kubernetes resource YAML as template to a+`KubernetesApplication` instance along with the list of `Secret`s and tell it to+which Kubernetes cluster to schedule that YAML and to propagate the given list+of `Secret`s that will be consumed by the resource in the template. This way,+everyone would still keep their infrastructure in the central cluster but if+they wanted their workloads to run in a separate cluster, then they'd wrap them+into `KubernetesApplication` and submit to that remote cluster. For reference,+here is a short version of how `KubernetesApplicationResource` looks like:++```yaml+apiVersion: common.crossplane.io/v1alpha1+kind: MySQLInstanceRequirement+metadata:+  name: sqldb+  namespace: default+spec:+  version: "5.7"+  storageGB: 20+  writeConnectionSecretToRef:+    name: sql-creds+---+apiVersion: workload.crossplane.io/v1alpha1+kind: KubernetesApplicationResource+metadata:+  name: wp-deployment+spec:+  # Select a KubernetesTarget which points to a secret that contains kubeconfig+  # of remote cluster.+  targetSelector:+    matchLabels:+      app: wp+  # The list of secrets that should be copied from central cluster to the+  # remote cluster.+  secrets:+    - name: sql-creds+  # The template of the actual resource to be created in the remote cluster.+  template:+    apiVersion: v1+    kind: Pod+    metadata:+    ...+    spec:+    containers:+    - name: wordpress+        image: "wordpress:4.6.1-apache"+        env:+        - name: WORDPRESS_DB_PASSWORD+          valueFrom:+            secretKeyRef:+              name: wp-deployment-sql-creds+              key: password+```++This resource is created in the central cluster and Crossplane itself would+manage your workload. It'd also propagate the status of the remote resource back+into status of `KubernetesApplicationResource`. In its essence, it pushes the+resources and pulls their status. Over time, we have identified several issues+with this approach:++* You cannot interact with what you deploy directly, i.e. always have to use+  `KubernetesApplicationResource` as a proxy and that has its own set of+  challenges:+  * It's a template and gets deployed after you make the edit, so, you loose the+    admission check rejections in case something went wrong. Instead, you'll see+    them in the status, but you won't be prevented from making the change as+    opposed to directly interacting.+  * Late initialization.* You cannot interact with what you deploy directly,

Is the asterisk after "late initialization" dangling? I can't find what it's referencing.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:++* Private Networking.+  * You may want to deploy different applications into different VPCs but manage+    all of your infrastructure from one place. This isn't possible since you are+    deploying all applications into the same cluster to have them use Crossplane+    and being in the same cluster necessitates usage of the same VPC.+* Cluster Configuration.+  * Because you have to run applications in the same central cluster with+    others, you'll have to share the same Kubernetes resources like nodes and+    your needs in terms of instance types could differ greatly depending on your+    workloads, like some need GPU-powered machines and others memory-heavy ones.+* Security.+    * All applications are subject to the same user management domain, i.e. same+      api-server. This could be managed to be safe, but it's not physically+      impossible to have a `ServiceAccount` in another namespace to have access+      to resources in your namespace. So, you wouldn't really trust to have+      production in one namespace and dev in the other.++The main point around all these use cases is that users want central+infrastructure management but not a physically central place for all of their+application deployment. In other terms, as a platform team in an organization,

I like that we're focusing on where apps run here. We probably don't need to make the point explicitly, but I imagine plenty of folks would be interested in a central control plane for apps too. i.e. It's not that folks don't want that - it's that we're focused on a different part of the problem space.

muvaf

comment created time in a month

Pull request review commentcrossplane/crossplane

Crossplane Agent design doc

+# Crossplane Agent for Consumption++* Owner: Muvaffak Onuş (@muvaf)+* Reviewers: Crossplane Maintainers+* Status: Draft++## Background++Crossplane allows users to provision and manage cloud services from your+Kubernetes cluster. It has managed resources that map to the services in the+provider 1-to-1 as the lowest level resource. Then users can build & publish+their own APIs that are abstractions over these managed resources. The actual+connection and consumption of these resources by applications are handled with+namespaced types called requirements whose CRDs are created via+`InfrastructurePublication`s and have `*Requirement` suffix in their kind.++The consumption model is such that applications should create requirements to+request certain services and supply `Secret` name in the requirements which will+be used by Crossplane to populate the necessary credentials for application to+consume. As a simple example, an application bundle would have a+`MySQLInstanceRequirement` custom resource, a `Pod` and they would share the+same name for the secret so that one fills that `Secret` with credentials and+the other one mounts it for the containers to consume.++> For brevity, application will be assumed to have only one `Pod`.++This consumption model works well in cases where you deploy your application in+a cluster where Crossplane runs, i.e. local mode. However, there are many valid+use cases where this is not really feasible. Here is a subset of those use+cases:

These use cases all sound roughly like reasons you might want to run multiple Kubernetes clusters. I wonder if we should frame it more explicitly that way? I'm thinking something like briefly detailing why folks might want to run multiple Kubernetes clusters, then exploring how Crossplane could fit into these multiple clusters - i.e. you could deploy and run one Crossplane per cluster, or you could deploy and run one centralised Crossplane. The latter option gives you one API for all infrastructure, and fewer moving parts (fewer Crossplane deployments) to manage.

muvaf

comment created time in a month

push eventcrossplane/crossplane-cli

Muvaffak Onus

commit sha 6ac01ba9aa68ff627cca3c865ae59dab6f6e6d72

Make installation more streamlined by removing uncommon alternative methods. Signed-off-by: Muvaffak Onus <onus.muvaffak@gmail.com>

view details

Muvaffak Onus

commit sha 06ce3f1b98c9f25129688a2d77896704114342c1

Nicer output for installation prompts Signed-off-by: Muvaffak Onus <onus.muvaffak@gmail.com>

view details

Nic Cope

commit sha 762f8c4dd84f11e3ae32d234334492f9d61a885a

Merge pull request #64 from muvaf/installsimpler Simpler installation and some emojis in output

view details

push time in a month

PR merged crossplane/crossplane-cli

Reviewers
Simpler installation and some emojis in output

I think that's mostly what people need in 99% of the cases. Main thing removed was using the script to install from master which won't let you have trace and pack because there is no compilation. You can do make build && make install to get the master installed, similar to other repositories.

In fact, bootstrap.sh got so simple I'm wondering whether we should just put its content in the README. But I liked the emoji output :)

image

Tested on Mac OSX 10.15 with zsh and Ubuntu 20.04 with bash (in Docker container)

+9 -61

0 comment

3 changed files