profile
viewpoint
Jay Pipes jaypipes Amazon Web Services Sarasota, FL, USA http://joinfu.com

argoproj/gitops-engine 887

Democratizing GitOps

aws/aws-node-termination-handler 550

A Kubernetes Daemonset to gracefully handle EC2 instance shutdown

jaypipes/articles 14

Collection of articles written for publication on my blog or elsewhere

jaypipes/aws-api-tools 12

Toolkit for working with AWS APIs

jaypipes/cookbook-zuul 3

Chef Cookbook and Recipes for Zuul

jaypipes/alembic 0

Github mirror of Alembic

jaypipes/amazon-eks-pod-identity-webhook 0

Amazon EKS Pod Identity Webhook

jaypipes/amazon-eks-user-guide 0

The open source version of the Amazon EKS user guide. You can submit feedback & requests for changes by submitting issues in this repo or by making proposed changes & submitting a pull request.

Pull request review commentaws/aws-controllers-k8s

APIGatewayv2 e2e

+#!/usr/bin/env bash++###########################################+# API+###########################################+create_http_api_and_validate() {+## create api resource+cat <<EOF | kubectl apply -f - >/dev/null 2>&1+apiVersion: apigatewayv2.services.k8s.aws/v1alpha1+kind: API+metadata:+  name: $api_name

It's cool to clean it up in a followup PR.

vijtrip2

comment created time in 3 days

PullRequestReviewEvent

push eventaws/aws-controllers-k8s

kumargauravsharma

commit sha 606f71694430fc1ac85f173f92559f587f2d07e3

Enable finegrained diffs, update() args include desired state, latest state (#299) Issue #, if available: #298 Changed the update() method args to include 'latest' state as well. Now both desired state and latest state is available inside update() method to compute finegrained diffs to make service api calls. Also, it addresses comment received on #245 to simplify the mechanism to configure custom implementation for operation on a custom resource. Following is an example to configure CustomModifyReplicationGroup as custom_implementation operation for ModifyReplicationGroup operation in elasticache generator.yaml config file: ```yaml operations: ModifyReplicationGroup: custom_implementation: CustomModifyReplicationGroup ``` Co-authored-by: Kumar Gaurav Sharma <kumargsh@amazon.com>

view details

push time in 3 days

PR merged aws/aws-controllers-k8s

Enable finegrained diffs, update() args include desired state, latest state

Issue #, if available: #298

Description of changes: Changed the update() method args to include 'latest' state as well. Now both desired state and latest state is available inside update() method to compute finegrained diffs to make service api calls.

Also, it addresses comment received on #245 to simplify the mechanism to configure custom implementation for operation on a custom resource. Following is an example to configure CustomModifyReplicationGroup as custom_implementation operation for ModifyReplicationGroup operation in elasticache generator.yaml config file:

operations:
  ModifyReplicationGroup:
    custom_implementation: CustomModifyReplicationGroup

Due to open issue #296 not all services' controllers are auto generated in this revision of the PR. Thus, make test fails for other services as of now.

This PR includes service controllers generated for elasticache and manually updated for petstore, bookstore services to help this review. Once issue #296 is fixed, remaining service controller will be generated and this PR will include corresponding updates.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

+445 -347

5 comments

47 changed files

kumargauravsharma

pr closed time in 3 days

pull request commentaws/aws-controllers-k8s

Enable finegrained diffs, update() args include desired state, latest state

@kumargauravsharma just a quick note, I'm going to squash and merge this, but in the future, please do git commit --amend && git push origin <BRANCH> --force if you make changes to a commit based on a code review instead of pushing more commits with commit messages like "fixed after rebase" etc :)

kumargauravsharma

comment created time in 3 days

PullRequestReviewEvent

issue closedaws/aws-controllers-k8s

Test fail after generating controllers for apigatewayv2

Describe the bug After generating controllers for apigateway2, the make test fails with error.

Steps to reproduce Commit level: commit 87bd3b1338c0f2e208f5ab9062a7240b823605c3 (HEAD -> upstream-main, upstream/main)

$ make test                                              # this passes
$ make build-controller SERVICE=apigatewayv2             # this generates changes under services/apigatewayv2
$ git status                                             # shows many files
$ make test                                              # this fails with error

Error:

# github.com/aws/aws-controllers-k8s/services/apigatewayv2/pkg/resource/api
services/apigatewayv2/pkg/resource/api/sdk.go:261:6: res.SetDisableExecuteApiEndpoint undefined (type *apigatewayv2.CreateApiInput has no field or method SetDisableExecuteApiEndpoint)
services/apigatewayv2/pkg/resource/api/sdk.go:417:6: res.SetDisableExecuteApiEndpoint undefined (type *apigatewayv2.UpdateApiInput has no field or method SetDisableExecuteApiEndpoint)
# github.com/aws/aws-controllers-k8s/services/apigatewayv2/pkg/resource/domain_name
services/apigatewayv2/pkg/resource/domain_name/sdk.go:190:10: undefined: apigatewayv2.MutualTlsAuthenticationInput
services/apigatewayv2/pkg/resource/domain_name/sdk.go:197:6: res.SetMutualTlsAuthentication undefined (type *apigatewayv2.CreateDomainNameInput has no field or method SetMutualTlsAuthentication)
services/apigatewayv2/pkg/resource/domain_name/sdk.go:285:10: undefined: apigatewayv2.MutualTlsAuthenticationInput
services/apigatewayv2/pkg/resource/domain_name/sdk.go:292:6: res.SetMutualTlsAuthentication undefined (type *apigatewayv2.UpdateDomainNameInput has no field or method SetMutualTlsAuthentication)

Expected outcome A concise description of what you expected to happen. make test passes after generating controllers for apigatewayv2

Environment

  • Kubernetes version
  • Using EKS (yes/no), if so version?
  • AWS service targeted (S3, RDS, etc.): apigatewayv2

closed time in 3 days

kumargauravsharma

issue commentaws/aws-controllers-k8s

Support IAM Roles deployments

Any timeline estimates on IAM provisioning via ACK? This would be an incredibly helpful feature.

Hi! No, at this time I have not thought about a timeline for IAM provisioning via an ACK service controller. If you haven't already, please feel free to +1 this issue.

Even if we agreed to follow some of the suggestions from @mksh (which are great, btw), we would need to get an IAM service controller for ACK onto the release roadmap and then determine a likely milestone for an IAM controller to go into dev preview (and then target a GA release of the controller 3 months after we release in dev preview). Earliest I see being able to get an IAM controller for ACK into dev preview would probably be end of 2020, given the backlog of services we're currently slated to deliver these next four months.

Of course, squeaky wheel gets the grease, so no worries squeaking about this feature request AFAIC :)

max-lobur

comment created time in 3 days

Pull request review commentaws/aws-controllers-k8s

APIGatewayv2 e2e

+#!/usr/bin/env bash++###########################################+# API+###########################################+create_http_api_and_validate() {+## create api resource+cat <<EOF | kubectl apply -f - >/dev/null 2>&1+apiVersion: apigatewayv2.services.k8s.aws/v1alpha1+kind: API+metadata:+  name: $api_name+spec:+  name: $api_name+  protocolType: HTTP+EOF++sleep 10++## validate that api-id was populated in resource status+debug_msg "retrieve api-id from api/$api_name resource's status"+api_id=$(kubectl get $api_resource_name -o=json | jq -r .status.apiID)++if [[ -z "$api_id" ]];then+	echo "FAIL: $api_resource_name resource's status does not have apiID"+	exit 1+fi++## validate that api was created using apigatewayv2 get-api operation+debug_msg "apigatewayv2 get-api with api-id $api_id"+aws apigatewayv2 get-api --api-id="$api_id" > /dev/null 2>&1+assert_equal "0" "$?" "Expected success from 'apigatewayv2 get-api --api-id=$api_id' but got $?" || exit 1+}

If you wouldn't mind, could you indent the inside of this function (and the other functions in this file) :)

Makes it tough to read when there's no indentation!

vijtrip2

comment created time in 3 days

Pull request review commentaws/aws-controllers-k8s

APIGatewayv2 e2e

+#!/usr/bin/env bash++###########################################+# API+###########################################+create_http_api_and_validate() {+## create api resource+cat <<EOF | kubectl apply -f - >/dev/null 2>&1+apiVersion: apigatewayv2.services.k8s.aws/v1alpha1+kind: API+metadata:+  name: $api_name+spec:+  name: $api_name+  protocolType: HTTP+EOF++sleep 10++## validate that api-id was populated in resource status+debug_msg "retrieve api-id from api/$api_name resource's status"+api_id=$(kubectl get $api_resource_name -o=json | jq -r .status.apiID)++if [[ -z "$api_id" ]];then+	echo "FAIL: $api_resource_name resource's status does not have apiID"+	exit 1+fi++## validate that api was created using apigatewayv2 get-api operation+debug_msg "apigatewayv2 get-api with api-id $api_id"+aws apigatewayv2 get-api --api-id="$api_id" > /dev/null 2>&1+assert_equal "0" "$?" "Expected success from 'apigatewayv2 get-api --api-id=$api_id' but got $?" || exit 1+}++delete_http_api_and_validate() {+#delete api resource+debug_msg "delete api/$api_name resource"+kubectl delete $api_resource_name >/dev/null 2>&1+assert_equal "0" "$?" "Expected success from kubectl delete but got $?" || exit 1++#validate that api was deleted using apigatewayv2 get-api operation+debug_msg "get-api with api-id $api_id"+aws apigatewayv2 get-api --api-id="$api_id" > /dev/null 2>&1+assert_equal "254" "$?" "Expected not-found status code from 'apigatewayv2 get-api --api-id=$api_id' but got $?" || exit 1

keep in mind that the aws CLI version 2 and aws CLI version 1 use different return codes unfortunately for a NotFound :(

Version 1 uses 255. version 2 uses 254, if I remember correctly.

So, you'll need to either:

  1. create a new assert_in function that accepts multiple acceptable values or

  2. Use an if statement like this:

https://github.com/aws/aws-controllers-k8s/blob/main/test/e2e/ecr/smoke.sh#L68-L72

vijtrip2

comment created time in 3 days

Pull request review commentaws/aws-controllers-k8s

APIGatewayv2 e2e

+#!/usr/bin/env bash++###########################################+# API+###########################################+create_http_api_and_validate() {+## create api resource+cat <<EOF | kubectl apply -f - >/dev/null 2>&1+apiVersion: apigatewayv2.services.k8s.aws/v1alpha1+kind: API+metadata:+  name: $api_name

I strongly recommend not using environment variables (and assuming that a particular env var is set to some non-nil value). Instead, I recommend using function local-scoped variables and actually pass arguments to the Bash function. Then, make a docstring comment that outlines expected parameters.

Example:

https://github.com/aws/aws-controllers-k8s/blob/main/scripts/lib/testutil.sh#L5-L23

vijtrip2

comment created time in 3 days

PullRequestReviewEvent
PullRequestReviewEvent

issue commentaws/aws-controllers-k8s

Make easy to migrate from existing CRDs to ACK

@jaypipes sorry meant no disrespect to crossplane, and was not intending to siphone users away from crossplane. It was more meant as a possibility to support easy migration. I will remove the mention of cross plane and make the issue generic

No need to remove the mention of Crossplane, Anoop! :) It's an important question that we've been asked a few times and figured I'd expand on the question in my answer here. I will put together a document that specifically discusses Crossplane and ACK plans.

anoop2811

comment created time in 3 days

issue commentkubernetes-sigs/controller-runtime

Dependency Failing

https://github.com/gomodules/jsonpatch/issues/28

jsing157

comment created time in 3 days

issue commentaws/aws-controllers-k8s

Migrate from crossplane when ACK is GA

Hi @anoop2811! I want to make it clear that it is not a goal of ACK to siphon users away from Crossplane.

ACK's mission is to facilitate the most Kubernetes-native way for users to interact with AWS managed services via the Kubernetes API and configuration language. While Crossplane does enable Kubernetes users to create infrastructure resources using the Kubernetes API/language, Crossplane has a much broader mission of enabling cross-cloud-provider workflows and multi-provider infrastructure needs.

Crossplane and ACK contributors are actually collaborating with each other, as we view the two projects as complementary, not competitive. In fact, I've been noodling around some ideas of using the ack-generate CLI tool to output Go code that follows the Crossplane object model/interfaces -- something that would eventually allow Crossplane's AWS cloud provider code to be replaced with code generated from ACK.

Best, -jay

anoop2811

comment created time in 3 days

issue commentgomodules/jsonpatch

Not served by gomodules.xyz?

Yes, this has broken builds of any project that depend on sigs.k8s.io/controller-runtime, such as the aws/aws-controllers-k8s project that is now unable to run go mod download with the following failure:

Building 'apigatewayv2' controller docker image with tag: ack-apigatewayv2-controller:29360d6
[+] Building 59.6s (11/14)                                                                                                                                                                                         
 => [internal] load .dockerignore                                                                                                                                                                             0.0s
 => => transferring context: 2B                                                                                                                                                                               0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                          0.0s
 => => transferring dockerfile: 38B                                                                                                                                                                           0.0s
 => [internal] load metadata for docker.io/library/golang:1.14.1                                                                                                                                              0.0s
 => [internal] load metadata for docker.io/library/amazonlinux:2                                                                                                                                              0.0s
 => [builder 1/7] FROM docker.io/library/golang:1.14.1                                                                                                                                                        0.0s
 => [internal] load build context                                                                                                                                                                             0.2s
 => => transferring context: 1.37MB                                                                                                                                                                           0.2s
 => [stage-1 1/3] FROM docker.io/library/amazonlinux:2                                                                                                                                                        0.0s
 => CACHED [builder 2/7] WORKDIR /github.com/aws/aws-controllers-k8s                                                                                                                                          0.0s
 => CACHED [builder 3/7] COPY go.mod go.mod                                                                                                                                                                   0.0s
 => CACHED [builder 4/7] COPY go.sum go.sum                                                                                                                                                                   0.0s
 => ERROR [builder 5/7] RUN  go mod download                                                                                                                                                                 59.3s
------                                                                                                                                                                                                             
 > [builder 5/7] RUN  go mod download:                                                                                                                                                                             
#11 59.04 go: sigs.k8s.io/controller-runtime@v0.6.0 requires                                                                                                                                                       
#11 59.04 	gomodules.xyz/jsonpatch/v2@v2.0.1: unrecognized import path "gomodules.xyz/jsonpatch/v2": reading https://gomodules.xyz/jsonpatch/v2?go-get=1: 404 Not Found
#11 59.04 	server response: 404 page not found
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c go mod download]: runc did not terminate sucessfully
make: *** [Makefile:39: build-controller-image] Error 2
chemikadze

comment created time in 3 days

push eventjaypipes/aws-controllers-k8s

Jay Pipes

commit sha 75692029795b0867ccdae97a199d1d61019b34ac

Add e2e tests for SQS Queues Copies the SNS Topic e2e smoke test and modified it to use the SQS `aws sqs get-queue-url` call instead of `aws sqs get-queue-attributes` since the latter requires the QueueUrl and not the ARN like get-topic-attributes does... Issue #205

view details

Jay Pipes

commit sha 09923136c4e8616ed30062085d7fed82664dc2f8

NotFound when any required field missing in shape In #271, @vijtrip2 added methods that output Go code that checked if the Input shape for ReadOne operations for a resource had required fields and if so, checked that the CR's `Status.{Field}` had a non-nil value, otherwise the `resourceManager.sdkFind()` method would return `NotFound`, indicating to callers (in particular when trying to determine if a newly-created resource already existed) that the backend AWS resource had not been created yet. This patch makes that work generic in two ways: * adds support for when required fields in an Input shape are in the CR's `Spec` struct in addition to the `Status` struct. * adds support for any Input shape, not just the ReadOne operation's Input shape. For SQS, the GetQueueAttributes' Input shape has required fields (QueueUrl) that, if not present in the CR's `Status` struct, indicate the resource has not been created yet. Issue #287

view details

Jay Pipes

commit sha 29360d667ddb3d898623ccb052e6179a0cbc25c9

improve documentation for authorization (#286) This patch adds a diagram and a number of improvements to the documentation describing authorization and permissions concepts in ACK. In addition, it adds recommended IAM Policy ARNs into a `services/$SERVICE/config/iam/recommended-policy-arn` file for our three checked-in ACK service controllers.

view details

Ricardo La Rosa

commit sha 5d899c3f9c9ac966fcdf301fa07e430efc68699c

Update controller-tools to v0.4.0

view details

Ricardo La Rosa

commit sha 70c14754073055a9763aac28059212ce576ddbac

Update crd, types and auto-generated code

view details

vijtrip2

commit sha 3a3a504137fdc1874a8d54c645dcb21156cc0286

apigatewayv2 service controller and smoke test files. (#293) * apigatewayv2 service controller and smoke test files.

view details

kumargauravsharma

commit sha 28eabf6df523210024f7fdcd520e5a576b88d91b

Enable multiple update operations support on single resource (#245) Issue #210 Description of changes: The changes in this PR are to enable custom resource update scenario where Update operation on given resource does not support fields that are supported by its Create method, and there exist specialized methods to update such fields. These specialized method response provides the Resource object indicating the Update on the resource. For example: create-replication-group API arguments include: num-node-groups replicas-per-node-group however, modify-replication-group API arguments does not, instead modify-replication-group-shard-configuration and increase-replica-count, decrease-replica-count APIs corresponds to the related Update. The approach taken in this PR is to add the details of such fields and corresponding custom update methods in the generator.yaml. For example: resources: ReplicationGroup: custom_update_operations: UpdateShardConfiguration: fields: - NumNodeGroups UpdateReplicaCount: fields: - ReplicasPerNodeGroup And custom update methods' (UpdateShardConfiguration and UpdateReplicaCount) implementation is provided in a separate code file: aws-controllers-k8s/services/elasticache/pkg/resource/replication_group/custom_update_api.go These methods are supplied with DiffReporter which helps the code in this file determine the specific Update API to invoke based on the difference in desired, current resource details. Co-authored-by: Kumar Gaurav Sharma <kumargsh@amazon.com>

view details

Jay Pipes

commit sha 1b2910dcee44802f6c9fc1a9baeb2fe6598e7230

update apigwv2 service controller for Diff Recent PR changed the AWSDescriptor interface signature and this PR simply updates the API Gateway V2 service controller to abide by that new interface.

view details

Amine Hilaly

commit sha be5f830c4e69ff7eda22c3a2040273f4e3709188

`ack-generate` calls `git pull` if aws-sdk-go repository already exists

view details

Jay Pipes

commit sha aaffb681058597dd265c9117dcc2a1513fb28ab7

Merge pull request #304 from A-Hilaly/refresh-cache `ack-generate` call `git pull` if aws-sdk-go repository already exist in the cache

view details

Amine Hilaly

commit sha 21627a478beb45e815e8cbab299ab1bd41aa650f

remove petstore and bookstore services

view details

Amine Hilaly

commit sha 5b3f2e73790e5cdc5ea678aca9e7dbed02d3adb3

Add a fake Book runtime.Object implementation for `service_controller_test.go` Use `mocks` for runtime util testing

view details

Jay Pipes

commit sha 4546ae0cf06e81c2557b30c1fc7ff08422ecf544

Merge pull request #305 from A-Hilaly/cleanup Remove petstore and bookstore services

view details

Prateek gogia

commit sha a5e3cb7da40f295feca6e0f117ed11da3339ebd6

Service names arguments are accepted as case insensitive for make commands

view details

Amine Hilaly

commit sha 34ed5dae3dcd8ed01d12e0a60f3d40141d6a2429

Remove petstore and bookstore integration tests

view details

Jay Pipes

commit sha c4bc50cdd4109086e2fed41d912072e75adeab99

Merge pull request #308 from A-Hilaly/cleanup Remove petstore and bookstore integration tests

view details

Amine Hilaly

commit sha a0d2c26ed2ab72e2aa18f95cb5895f01ae3a77ff

Remove `test/integration` folder and `scripts/run-integration-tests.sh`

view details

Jay Pipes

commit sha 8fba3fd4f8e64c59abe32deed7d433cbaf584141

Merge pull request #309 from A-Hilaly/cleanup Remove `test/integration` folder and `scripts/run-integration-tests.sh`

view details

Ricardo La Rosa

commit sha 6bc0fc76a7cfa15ff889d0eb1eb4efc3d85afff7

Merge branch 'main' into update-types

view details

Ricardo La Rosa

commit sha d0ac10fb1ef75ca9c3af7f9df03a0e3f7f1bbcf7

Update crd, rbac for S3

view details

push time in 4 days

Pull request review commentaws/aws-controllers-k8s

fix issues with Get/SetAttributes Go code generation

 resources:   Topic:     unpack_attributes_map:+      set_attributes_single_attribute: true

Yeah, it's an interesting question. I did it this way because this is more of a variation on the "UnpacksAttributesMap" behaviour than it is for anything else. That said, I'm going to be using the OperationConfig.CustomOperation functionality to sidestep the weirdness in the SetTopicAttributes API call entirely for now, so the point may be moot (for the timebeing at least)

jaypipes

comment created time in 4 days

PullRequestReviewEvent

push eventjaypipes/aws-controllers-k8s

Jay Pipes

commit sha 22b1c9302630006e42bea22946a83e30abc9b842

handle SetAttributes-based update operations This patch adds to the code generator for the SDK linkage for APIs that use the SetAttributes-flavor of updating a resource. Previously, the `sdk.go` files for these APIs had empty `sdkUpdate` operations with a big TODO that ended up breaking the service controller (quite predictably). While implementing this patch, I stumbled upon yet another unexplainable inconsistency between two SetAttributes operations within the SNS service API. The SNS `SetPlatformApplicationAttributes` API call accepts a field in its Input shape called `Attributes` that is a map of key/value pairs for attributes to set on the platform application. Unfortunately, the similarly-named SNS `SetTopicAttributes` API call apparently does *NOT* work the same way. Instead, there is a single `AttributeName` and `AttributeValue` field in the Input shape and you need to call the `SetTopicAttributes` API call once for each modified attribute. :( In fact, the [official documentation][0] for the SNS SetTopicAttributes API call says that the `AttributeName` field in the Input shape is actually a "*map* of attributes with their corresponding values". But that isn't actually the case... So, the code in this patch for now just contains a TODO in the `CRD.GoCodeSetAttributesSetInput()` method for SetAttributes APIs that can only operate on a single attribute at a time. We will use the CustomOperation functionality as a temporary workaround for these APIs while we come up with a more permanent code-generated solution. Issue #296 [0]: https://docs.aws.amazon.com/sns/latest/api/API_SetTopicAttributes.html

view details

push time in 4 days

PR opened aws/aws-controllers-k8s

Reviewers
fix issues with Get/SetAttributes Go code generation

This patch adds to the code generator for the SDK linkage for APIs that use the SetAttributes-flavor of updating a resource. Previously, the sdk.go files for these APIs had empty sdkUpdate operations with a big TODO that ended up breaking the service controller (quite predictably).

While implementing this patch, I stumbled upon yet another unexplainable inconsistency between two SetAttributes operations within the SNS service API. The SNS SetPlatformApplicationAttributes API call accepts a field in its Input shape called Attributes that is a map of key/value pairs for attributes to set on the platform application. Unfortunately, the similarly-named SNS SetTopicAttributes API call apparently does NOT work the same way. Instead, there is a single AttributeName and AttributeValue field in the Input shape and you need to call the SetTopicAttributes API call once for each modified attribute. :(

In fact, the [official documentation][0] for the SNS SetTopicAttributes API call says that the AttributeName field in the Input shape is actually a "map of attributes with their corresponding values". But that isn't actually the case...

So, the code in this patch for now just contains a TODO in the CRD.GoCodeSetAttributesSetInput() method for SetAttributes APIs that can only operate on a single attribute at a time. We will use the CustomOperation functionality as a temporary workaround for these APIs while we come up with a more permanent code-generated solution.

There were a series of problems that I uncovered when investigating the cause of Issue #296:

  • The code returned from GoCodeGetAttributesForOutput was failing to set Status.ACKResourceMetadata.ARN and Status.ACKResourceMetadata.OwnerAccountID when the attribute fields corresponded to the primary resource ARN or the owner ID. This was the direct cause of Issue #296
  • The code in templates/pkg/crd_sdk.go.tpl that ran at the end of the sdkCreate() call was inadvertently overwriting any setters of Status.ACKResourceMetadata that had previously executed
  • The code returned from APIs with GetAttributes operations was always returning a nil-guard and constructor for the ACKResourceMetadata struct, even when no attribute fields actually set either ARN or OwnerAccountID. When I removed the check in the template for Status fields being required for the resp variable to be defined, this was causing "resp" variable unused compilation failures

Issue #296

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

+669 -75

0 comment

17 changed files

pr created time in 4 days

create barnchjaypipes/aws-controllers-k8s

branch : issue296

created branch time in 4 days

PullRequestReviewEvent

Pull request review commentaws/aws-controllers-k8s

CARM cache implementation

 func (r *reconciler) BindControllerManager(mgr ctrlrt.Manager) error { 	if r.rmf == nil { 		return ackerr.NilResourceManagerFactory 	}+	clusterConfig := mgr.GetConfig()+	kc, err := kubernetes.NewForConfig(clusterConfig)+	if err != nil {+		return err+	}+	r.cache = ackrtcache.New(kc, r.log) 	r.kc = mgr.GetClient()

hmmm. interesting... OK, soon as I'm done with a customer call I'll look into perhaps a cleaner way to do this.

A-Hilaly

comment created time in 4 days

Pull request review commentaws/aws-controllers-k8s

CARM cache implementation

 func (r *reconciler) BindControllerManager(mgr ctrlrt.Manager) error { 	if r.rmf == nil { 		return ackerr.NilResourceManagerFactory 	}+	clusterConfig := mgr.GetConfig()+	kc, err := kubernetes.NewForConfig(clusterConfig)+	if err != nil {+		return err+	}+	r.cache = ackrtcache.New(kc, r.log) 	r.kc = mgr.GetClient()

Sorry, you misunderstood me :) I was saying that the clientset object is already being created on line 71 with:

r.kc = mgr.GetClient()

so you don't have to create another one with the call to mgr.GetConfig() and thenkubernetes.NewForConfig(). You can just pass the reconciler'skcattribute (which is the kubernetes clientset) to yourackrtcache.New()` function.

A-Hilaly

comment created time in 4 days

PullRequestReviewEvent
PullRequestReviewEvent

push eventaws/aws-controllers-k8s

kumargauravsharma

commit sha c6dd57d61380b8d2891bdbf897f6b3e53f83b4a8

Enable custom set output operations (#307) * Enable custom set output operations Co-authored-by: Kumar Gaurav Sharma <kumargsh@amazon.com>

view details

push time in 4 days

PR merged aws/aws-controllers-k8s

Enable custom set output operations

Issue #282

Description of changes: Added support for custom set output operations using generator.yaml configuration. Example:

operations:
  DescribeReplicationGroups:
    custom_set_output_operation: CustomDescribeReplicationGroupsSetOutput

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

+116 -2

0 comment

8 changed files

kumargauravsharma

pr closed time in 4 days

Pull request review commentaws/aws-controllers-k8s

Enable custom set output operations

 type Config struct { // specify the overriding values for API operation parameters type OperationConfig struct { 	OverrideValues map[string]string `json:"override_values"`+	// SetOutputCustomMethodName provides the name of the custom method on the+	// `resourceManager` struct that will set fields on a `resource` struct+	// depending on the output of the operation.

In a future PR, we should consider documenting here the interface / function signature of the SetOutput custom method.

kumargauravsharma

comment created time in 4 days

PullRequestReviewEvent
PullRequestReviewEvent

issue commentaws/aws-controllers-k8s

service needs to be lower case ?

Thanks, @jaypipes - I'll take a gander at #199. I'm not a pro, so may pop-up with questions and can work on it only after-hours, hence may take some time (is this fine)?

Totally fine! :) We're here to help. Alternately, feel free to find me on Kubernetes Slack #provider-aws channel!

sebgoa

comment created time in 4 days

pull request commentaws/aws-controllers-k8s

Enable finegrained diffs, update() args include desired state, latest state

I'm confused why this particular patch is failing make test in the way it is (seems a mockery error).

kumargauravsharma

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentaws/aws-controllers-k8s

Enable finegrained diffs, update() args include desired state, latest state

 resources:     exceptions:       codes:         404: CacheSubnetGroupNotFoundFault-  ReplicationGroup:-    custom_update_operations:-      UpdateShardConfiguration:-        diff_paths:-          - Spec.NumNodeGroups-      UpdateReplicaCount:-        diff_paths:-          - Spec.ReplicasPerNodeGroup operations:   ModifyReplicationGroup:+    custom_implementation: CustomModifyReplicationGroup

:+1:

kumargauravsharma

comment created time in 4 days

PullRequestReviewEvent

push eventaws/aws-controllers-k8s

Ricardo La Rosa

commit sha 5d899c3f9c9ac966fcdf301fa07e430efc68699c

Update controller-tools to v0.4.0

view details

Ricardo La Rosa

commit sha 70c14754073055a9763aac28059212ce576ddbac

Update crd, types and auto-generated code

view details

Ricardo La Rosa

commit sha 6bc0fc76a7cfa15ff889d0eb1eb4efc3d85afff7

Merge branch 'main' into update-types

view details

Ricardo La Rosa

commit sha d0ac10fb1ef75ca9c3af7f9df03a0e3f7f1bbcf7

Update crd, rbac for S3

view details

Ricardo La Rosa

commit sha fca149df8cacbd67a5a98acad338c1e6c80f484b

Update types for S3BucketDestination and gen code

view details

Jay Pipes

commit sha b6f81ac7a08d333885ad5c78a5c3dd9766f38046

Merge pull request #295 from ricardo-larosa/update-types Update crd, types and auto-generated code for S3

view details

push time in 4 days

PR merged aws/aws-controllers-k8s

Update crd, types and auto-generated code for S3

Fixes https://github.com/aws/aws-controllers-k8s/issues/294

Description of changes:

  • Bump to controllers-tools v0.4.0
  • Update crd, types and auto-generated code for S3
+134 -114

2 comments

6 changed files

ricardo-larosa

pr closed time in 4 days

issue closedaws/aws-controllers-k8s

ValidationError(CustomResourceDefinition.status): missing required field "storedVersions"

Describe the bug CRD definitions are wrong

Steps to reproduce Error when installing S3 CRD

error: error validating "STDIN": error validating data: ValidationError(CustomResourceDefinition.status): missing required field "storedVersions" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinitionStatus; if you choose to ignore these errors, turn validation off with --validate=false

Expected outcome CRD Should successfully be applied.

Environment Local

  • Kubernetes version 1.16
  • Using EKS (yes/no), if so version? Yes, 1.16
  • AWS service targeted (S3, RDS, etc.) S3

closed time in 4 days

ricardo-larosa
PullRequestReviewEvent

pull request commentaws/aws-controllers-k8s

Add Github action job to build AWS service controllers

Needs a rebase, @A-Hilaly now that I merged Prateek's patch. Plus, I'm going to need to push the fix for #296 before this will pass on main branch :)

A-Hilaly

comment created time in 5 days

push eventaws/aws-controllers-k8s

Prateek gogia

commit sha a5e3cb7da40f295feca6e0f117ed11da3339ebd6

Service names arguments are accepted as case insensitive for make commands

view details

Jay Pipes

commit sha 68cf1dcb29e7f87f32f1031559d53cc3cc0e6dbb

Merge pull request #306 from prateekgogia/bug/302 Service names arguments are accepted as case insensitive for make commands

view details

push time in 5 days

PR merged aws/aws-controllers-k8s

Service names arguments are accepted as case insensitive for make commands

Issue #302

Description of changes: Convert service name to lowercase in Makefile and helper script files - build-controllers.sh and build-controller.sh

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

+8 -7

0 comment

3 changed files

prateekgogia

pr closed time in 5 days

PullRequestReviewEvent

push eventaws/aws-controllers-k8s

Amine Hilaly

commit sha a0d2c26ed2ab72e2aa18f95cb5895f01ae3a77ff

Remove `test/integration` folder and `scripts/run-integration-tests.sh`

view details

Jay Pipes

commit sha 8fba3fd4f8e64c59abe32deed7d433cbaf584141

Merge pull request #309 from A-Hilaly/cleanup Remove `test/integration` folder and `scripts/run-integration-tests.sh`

view details

push time in 5 days

PR merged aws/aws-controllers-k8s

Remove `test/integration` folder and `scripts/run-integration-tests.sh`

Follow up to #308

Description of changes:

  • Remove test/integration folder and scripts/run-integration-tests.sh

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

+0 -726

0 comment

4 changed files

A-Hilaly

pr closed time in 5 days

PullRequestReviewEvent

Pull request review commentaws/aws-controllers-k8s

CARM cache implementation

 func TestServiceController(t *testing.T) { 	sc := ackrt.NewServiceController("bookstore", "bookstore.services.k8s.aws") 	require.NotNil(sc) +	zapOptions := ctrlrtzap.Options{+		Development: true,+		Level:       zapcore.InfoLevel,+	}+	fakeLogger := ctrlrtzap.New(ctrlrtzap.UseFlagOptions(&zapOptions))

Maybe. Can do later :)

A-Hilaly

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

CARM cache implementation

+// Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.+//+// Licensed under the Apache License, Version 2.0 (the "License"). You may+// not use this file except in compliance with the License. A copy of the+// License is located at+//+//     http://aws.amazon.com/apache2.0/+//+// or in the "license" file accompanying this file. This file is distributed+// on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either+// express or implied. See the License for the specific language governing+// permissions and limitations under the License.++package cache_test++import (+	"context"+	"testing"+	"time"++	"github.com/stretchr/testify/require"+	"go.uber.org/zap/zapcore"+	corev1 "k8s.io/api/core/v1"+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"+	"k8s.io/apimachinery/pkg/watch"+	k8sfake "k8s.io/client-go/kubernetes/fake"+	k8stesting "k8s.io/client-go/testing"+	ctrlrtzap "sigs.k8s.io/controller-runtime/pkg/log/zap"++	ackrtcache "github.com/aws/aws-controllers-k8s/pkg/runtime/cache"+)++const (+	testNamespace = "ack-system"++	testAccount1    = "012345678912"+	testAccountARN1 = "arn:aws:iam::012345678912:role/S3Access"+	testAccount2    = "219876543210"+	testAccountARN2 = "arn:aws:iam::012345678912:role/root"+)++func TestAccountCache(t *testing.T) {+	accountsMap1 := map[string]string{+		testAccount1: testAccountARN1,+	}++	accountsMap2 := map[string]string{+		testAccount1: testAccountARN1,+		testAccount2: testAccountARN2,+	}++	// create a fake k8s client and a fake watcher+	k8sClient := k8sfake.NewSimpleClientset()+	watcher := watch.NewFake()+	k8sClient.PrependWatchReactor("configMaps", k8stesting.DefaultWatchReactor(watcher, nil))++	zapOptions := ctrlrtzap.Options{+		Development: true,+		Level:       zapcore.InfoLevel,+	}+	fakeLogger := ctrlrtzap.New(ctrlrtzap.UseFlagOptions(&zapOptions))++	// initlizing account cache+	accountCache := ackrtcache.NewAccountCache(k8sClient, fakeLogger)+	stopCh := make(chan struct{})++	// This is mainly used to avoid using time.Sleep+	accountCache.Run(stopCh)++	// Test create events+	k8sClient.CoreV1().ConfigMaps(testNamespace).Create(+		context.Background(),+		&corev1.ConfigMap{+			ObjectMeta: metav1.ObjectMeta{+				Name: "random-map",+			},+			Data: accountsMap1,+		},+		metav1.CreateOptions{},+	)++	time.Sleep(time.Second)

We can certainly improve this in the future and remove the sleep calls, but this is perfectly fine for now. Nice work on this, @A-Hilaly

A-Hilaly

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

CARM cache implementation

 func (r *Registry) GetResourceManagerFactories() []types.AWSResourceManagerFacto 	return res } -// RegisterManagerFactory registers a resource manager factory with the+// RegisterResourceManagerFactory registers a resource manager factory with the

thanks :)

A-Hilaly

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

CARM cache implementation

 func (r *reconciler) BindControllerManager(mgr ctrlrt.Manager) error { 	if r.rmf == nil { 		return ackerr.NilResourceManagerFactory 	}+	clusterConfig := mgr.GetConfig()+	kc, err := kubernetes.NewForConfig(clusterConfig)+	if err != nil {+		return err+	}+	r.cache = ackrtcache.New(kc, r.log) 	r.kc = mgr.GetClient()

Any reason why you did the above instead of just:

r.kc = mgr.GetClient()
r.cache = ackrtcache.New(r.kc, r.log)
A-Hilaly

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

CARM cache implementation

+// Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.+//+// Licensed under the Apache License, Version 2.0 (the "License"). You may+// not use this file except in compliance with the License. A copy of the+// License is located at+//+//     http://aws.amazon.com/apache2.0/+//+// or in the "license" file accompanying this file. This file is distributed+// on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either+// express or implied. See the License for the specific language governing+// permissions and limitations under the License.++package cache++import (+	"sync"++	"github.com/go-logr/logr"+	corev1 "k8s.io/api/core/v1"+	informersv1 "k8s.io/client-go/informers/core/v1"+	kubernetes "k8s.io/client-go/kubernetes"+	k8scache "k8s.io/client-go/tools/cache"+)++const (+	// ACKRoleAccountMap is the name of the configmap map object storing+	// all the AWS Account IDs associated with their AWS Role ARNs.+	ACKRoleAccountMap = "ack-role-account-map"+)++// AccountCache is responsible for caching the CARM configmap+// data. It is listening to all the events related to the CARM map and+// make the changes accordingly.+type AccountCache struct {+	sync.RWMutex++	log logr.Logger++	// ConfigMap informer+	informer k8scache.SharedInformer+	roleARNs map[string]string+}++// NewAccountCache makes a new AccountCache from a client.Interface+// and a logr.Logger+func NewAccountCache(clientset kubernetes.Interface, log logr.Logger) *AccountCache {+	sharedInformer := informersv1.NewConfigMapInformer(+		clientset,+		currentNamespace,+		informerResyncPeriod,+		k8scache.Indexers{},+	)+	return &AccountCache{+		informer: sharedInformer,+		log:      log.WithName("AccountCache"),+		roleARNs: make(map[string]string),+	}+}++// resourceMatchACKRoleAccountConfigMap verifies if a resource is+// the CARM configmap. It verifies the name, namespace and object type.+func resourceMatchACKRoleAccountsConfigMap(raw interface{}) bool {+	object, ok := raw.(*corev1.ConfigMap)+	return ok && object.ObjectMeta.Name == ACKRoleAccountMap+}++// Run adds the default event handler functions to the SharedInformer and+// runs the informer to begin processing items.+func (c *AccountCache) Run(stopCh <-chan struct{}) {+	c.informer.AddEventHandler(k8scache.ResourceEventHandlerFuncs{+		AddFunc: func(obj interface{}) {+			if resourceMatchACKRoleAccountsConfigMap(obj) {+				object := obj.(*corev1.ConfigMap).DeepCopy()+				c.log.V(1).Info("ack-role-account-map has been created")+				c.updateAccountRoleData(object.Data)+				c.log.V(1).Info("cached ack-role-account-map data")+			}+		},+		UpdateFunc: func(old, new interface{}) {+			if resourceMatchACKRoleAccountsConfigMap(new) {+				object := new.(*corev1.ConfigMap).DeepCopy()+				c.log.V(1).Info("ack-role-account-map has been updated")+				//TODO(a-hilaly): compare data checksum before updating the cache+				c.updateAccountRoleData(object.Data)+				c.log.V(1).Info("cached ack-role-account-map data")+			}+		},+		DeleteFunc: func(obj interface{}) {+			if resourceMatchACKRoleAccountsConfigMap(obj) {+				c.log.V(1).Info("ack-role-account-map has been deleted")+				newMap := make(map[string]string)+				c.updateAccountRoleData(newMap)+				c.log.V(1).Info("cleaned up role account map")+			}+		},+	})+	go c.informer.Run(stopCh)+}++// GetAccountRoleARN queries the AWS accountID associated Role ARN+// from the cached CARM configmap. This function is thread safe.+func (c *AccountCache) GetAccountRoleARN(accountID string) (string, bool) {+	c.RLock()+	defer c.RUnlock()+	roleARN, ok := c.roleARNs[accountID]+	return roleARN, ok

:+1:

A-Hilaly

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

CARM cache implementation

+// Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.+//+// Licensed under the Apache License, Version 2.0 (the "License"). You may+// not use this file except in compliance with the License. A copy of the+// License is located at+//+//     http://aws.amazon.com/apache2.0/+//+// or in the "license" file accompanying this file. This file is distributed+// on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either+// express or implied. See the License for the specific language governing+// permissions and limitations under the License.++package cache++import (+	"sync"++	"github.com/go-logr/logr"+	corev1 "k8s.io/api/core/v1"+	informersv1 "k8s.io/client-go/informers/core/v1"+	kubernetes "k8s.io/client-go/kubernetes"+	k8scache "k8s.io/client-go/tools/cache"+)++const (+	// ACKRoleAccountMap is the name of the configmap map object storing+	// all the AWS Account IDs associated with their AWS Role ARNs.+	ACKRoleAccountMap = "ack-role-account-map"+)++// AccountCache is responsible for caching the CARM configmap+// data. It is listening to all the events related to the CARM map and+// make the changes accordingly.+type AccountCache struct {+	sync.RWMutex++	log logr.Logger++	// ConfigMap informer+	informer k8scache.SharedInformer+	roleARNs map[string]string+}++// NewAccountCache makes a new AccountCache from a client.Interface+// and a logr.Logger+func NewAccountCache(clientset kubernetes.Interface, log logr.Logger) *AccountCache {+	sharedInformer := informersv1.NewConfigMapInformer(+		clientset,+		currentNamespace,+		informerResyncPeriod,+		k8scache.Indexers{},+	)+	return &AccountCache{+		informer: sharedInformer,+		log:      log.WithName("AccountCache"),+		roleARNs: make(map[string]string),+	}+}++// resourceMatchACKRoleAccountConfigMap verifies if a resource is+// the CARM configmap. It verifies the name, namespace and object type.+func resourceMatchACKRoleAccountsConfigMap(raw interface{}) bool {+	object, ok := raw.(*corev1.ConfigMap)+	return ok && object.ObjectMeta.Name == ACKRoleAccountMap+}++// Run adds the default event handler functions to the SharedInformer and+// runs the informer to begin processing items.+func (c *AccountCache) Run(stopCh <-chan struct{}) {+	c.informer.AddEventHandler(k8scache.ResourceEventHandlerFuncs{+		AddFunc: func(obj interface{}) {+			if resourceMatchACKRoleAccountsConfigMap(obj) {+				object := obj.(*corev1.ConfigMap).DeepCopy()+				c.log.V(1).Info("ack-role-account-map has been created")+				c.updateAccountRoleData(object.Data)+				c.log.V(1).Info("cached ack-role-account-map data")+			}+		},+		UpdateFunc: func(old, new interface{}) {

teeny nit: new is a keyword in Go and naming variables the same as keywords (something called "variable shadowing" ) is generally discouraged. Might be better to name these orig and desired?

A-Hilaly

comment created time in 5 days

PullRequestReviewEvent
PullRequestReviewEvent

pull request commentaws/aws-controllers-k8s

Remove petstore and bookstore integration tests

and the scripts/run_integration_tests.sh script :)

A-Hilaly

comment created time in 5 days

pull request commentaws/aws-controllers-k8s

Remove petstore and bookstore integration tests

You can probably now remove the entire test/integration directory :)

A-Hilaly

comment created time in 5 days

push eventaws/aws-controllers-k8s

Amine Hilaly

commit sha 34ed5dae3dcd8ed01d12e0a60f3d40141d6a2429

Remove petstore and bookstore integration tests

view details

Jay Pipes

commit sha c4bc50cdd4109086e2fed41d912072e75adeab99

Merge pull request #308 from A-Hilaly/cleanup Remove petstore and bookstore integration tests

view details

push time in 5 days

PR merged aws/aws-controllers-k8s

Remove petstore and bookstore integration tests

Follow up to #305

Description of changes:

  • Remove petstore and bookstore integration tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

+0 -68

0 comment

2 changed files

A-Hilaly

pr closed time in 5 days

PullRequestReviewEvent

Pull request review commentaws/aws-controllers-k8s

Enable custom set output operations

 func (rm *resourceManager) sdkUpdate( {{ end }}  {{ $setCode := GoCodeSetUpdateOutput .CRD "resp" "ko.Status" 1 }}+{{ $setCustomOutputOperation := .CRD.GetCustomSetOutputOperation .CRD.Ops.Update }}

ditto.

kumargauravsharma

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

Enable custom set output operations

 func (rm *resourceManager) sdkFind( 		return nil, err 	} {{ $setCode := GoCodeSetReadManyOutput .CRD "resp" "ko" 1 }}+{{ $setCustomOutputOperation := .CRD.GetCustomSetOutputOperation .CRD.Ops.ReadMany }}

any reason to place this here instead of below where it is used?

kumargauravsharma

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

Enable custom set output operations

 func (rm *resourceManager) sdkCreate( 		return nil, err 	} {{ $createCode := GoCodeSetCreateOutput .CRD "resp" "ko.Status" 1 }}+{{ $setCustomOutputOperation := .CRD.GetCustomSetOutputOperation .CRD.Ops.Create }}

ditto...

kumargauravsharma

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

Enable custom set output operations

 func (r *CRD) IsPrimaryARNField(fieldName string) bool { 		strings.EqualFold(fieldName, r.Names.Original+"arn") } +// GetCustomSetOutputOperation returns custom set output operation as string for+// given operation on custom resource, if specified in generator config+func (r *CRD) GetCustomSetOutputOperation(

You could just call this method SetOutputCustomMethodName()

Additionally, consider making the attribute a *string instead of string and returning nil here instead of "" to indicate there is no custom method name...

kumargauravsharma

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

Enable custom set output operations

 type Config struct { // specify the overriding values for API operation parameters type OperationConfig struct { 	OverrideValues map[string]string `json:"override_values"`+	// CustomSetOutputOperation provides custom operation to process operation's output shape.+	CustomSetOutputOperation string `json:"custom_set_output_operation,omitempty"`

Probably just call this SetOutputCustomMethod or SetOutputCustomMethodName and change the description to say:

// SetOutputCustomMethod provides the name of the custom method on the
// `resourceManager` struct that will set fields on a `resource` struct
// depending on the output of the operation.
kumargauravsharma

comment created time in 5 days

PullRequestReviewEvent
PullRequestReviewEvent

pull request commentaws/aws-controllers-k8s

Update crd, types and auto-generated code for S3

Hi @ricardo-larosa, thanks very much for the PR! Any chance you might be able to rebase to main?

ricardo-larosa

comment created time in 5 days

issue commentaws/aws-controllers-k8s

service needs to be lower case ?

Kind of similar problem to this one...

sebgoa

comment created time in 5 days

issue commentaws/aws-controllers-k8s

service needs to be lower case ?

Can I take a gander at this?

@ChaturvediSulabh looks like @pgogia beat you to it :)

Are you looking for a place to contribute? How about this one? https://github.com/aws/aws-controllers-k8s/issues/199

sebgoa

comment created time in 5 days

issue commentaws/aws-controllers-k8s

Handle S3 OperationAborted due to conflicting conditions

Is this the right direction with the changes to the generatorConfig? Wanted to be clear that "error codes" doesn't mean HTTP codes in this case - there are multiple 409's you can get when create a bucket and I'm assuming you wouldn't want to requeue for BucketAlreadyExists.

resources:
  Bucket:
  ...
    exceptions:
      requeue:
        - OperationAborted

Yeah, I realized way too late that S3 Bucket names are globally unique, and that OperationAborted: A conflicting condition operation... is returned when trying to Create an S3 Bucket with the same name as one currently being deleted.

This will need to be handled quite carefully; I'm still thinking about ways we can instruct the code generator about these types of weirdnesses.

jaypipes

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

Service names arguments are accepted as case insensitive for make commands

 test: | mocks	## Run code tests clean-mocks:	## Remove mocks directory 	rm -rf mocks -build-controller-image:	## Build container image for SERVICE-	./scripts/build-controller-image.sh -s $(SERVICE)+build-controller-image:	## Build container image for AWS_SERVICE+	./scripts/build-controller-image.sh -s $(AWS_SERVICE)  publish-controller-image:  ## docker push a container image for SERVICE 	@echo $(DOCKER_PASSWORD) | docker login -u $(DOCKER_USERNAME) --password-stdin-	./scripts/publish-controller-image.sh -r $(DOCKER_REPOSITORY) -s $(SERVICE)+	./scripts/publish-controller-image.sh -r $(DOCKER_REPOSITORY) -s $(AWS_SERVICE) -build-controller: build-ack-generate	## Generate controller code for SERVICE-	./scripts/build-controller.sh $(SERVICE)+build-controller: build-ack-generate	## Generate controller code for AWS_SERVICE+	./scripts/build-controller.sh $(AWS_SERVICE) -kind-test: test	## Run functional tests for SERVICE with AWS_ROLE_ARN-	./scripts/kind-build-test.sh -s $(SERVICE) -p -r $(AWS_ROLE_ARN)+kind-test: test	## Run functional tests for AWS_SERVICE with AWS_ROLE_ARN

ditto

prateekgogia

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

Service names arguments are accepted as case insensitive for make commands

 test: | mocks	## Run code tests clean-mocks:	## Remove mocks directory 	rm -rf mocks -build-controller-image:	## Build container image for SERVICE-	./scripts/build-controller-image.sh -s $(SERVICE)+build-controller-image:	## Build container image for AWS_SERVICE+	./scripts/build-controller-image.sh -s $(AWS_SERVICE)  publish-controller-image:  ## docker push a container image for SERVICE 	@echo $(DOCKER_PASSWORD) | docker login -u $(DOCKER_USERNAME) --password-stdin-	./scripts/publish-controller-image.sh -r $(DOCKER_REPOSITORY) -s $(SERVICE)+	./scripts/publish-controller-image.sh -r $(DOCKER_REPOSITORY) -s $(AWS_SERVICE) -build-controller: build-ack-generate	## Generate controller code for SERVICE-	./scripts/build-controller.sh $(SERVICE)+build-controller: build-ack-generate	## Generate controller code for AWS_SERVICE

ditto

prateekgogia

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

Service names arguments are accepted as case insensitive for make commands

 test: | mocks	## Run code tests clean-mocks:	## Remove mocks directory 	rm -rf mocks -build-controller-image:	## Build container image for SERVICE-	./scripts/build-controller-image.sh -s $(SERVICE)+build-controller-image:	## Build container image for AWS_SERVICE

Need to change this back to SERVICE since this is output for make help (and the variable to set for the user is SERVICE, not AWS_SERVICE)

prateekgogia

comment created time in 5 days

PullRequestReviewEvent
PullRequestReviewEvent

push eventaws/aws-controllers-k8s

Amine Hilaly

commit sha 21627a478beb45e815e8cbab299ab1bd41aa650f

remove petstore and bookstore services

view details

Amine Hilaly

commit sha 5b3f2e73790e5cdc5ea678aca9e7dbed02d3adb3

Add a fake Book runtime.Object implementation for `service_controller_test.go` Use `mocks` for runtime util testing

view details

Jay Pipes

commit sha 4546ae0cf06e81c2557b30c1fc7ff08422ecf544

Merge pull request #305 from A-Hilaly/cleanup Remove petstore and bookstore services

view details

push time in 5 days

PR merged aws/aws-controllers-k8s

Remove petstore and bookstore services

Issue #, if available: N/A

Description of changes:

  • Delete services/petstore and services/bookstore folders
  • Use a fake Book runtime.Object implementation in pkg/runtime/service_controller_test.go

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

+54 -5888

1 comment

37 changed files

A-Hilaly

pr closed time in 5 days

PullRequestReviewEvent

Pull request review commentaws/aws-controllers-k8s

Enable finegrained diffs, update() args include desired state, latest state

 func (rm *resourceManager) UpdateReplicaCount( 	if respErr != nil { 		return nil, respErr 	}-	return provideUpdatedResource(r, resp.ReplicationGroup)+	return provideUpdatedResource(rd, resp.ReplicationGroup) }  // newUpdate(ShardConfiguration)RequestPayload returns an SDK-specific struct for the HTTP request // payload of the Update API call for the resource func (rm *resourceManager) newUpdateShardConfigurationRequestPayload(-	r *resource,+	rd *resource, // desired+	rl *resource, // latest

actually, I was originally confused why you were passing the resourceDescriptor but then realized that you were just calling the "desired" resource parameter rd. Recommend calling these parameters desired and latest :) We use "rd" to indicate the resourceDescriptor struct

kumargauravsharma

comment created time in 5 days

Pull request review commentaws/aws-controllers-k8s

Enable finegrained diffs, update() args include desired state, latest state

 func (rm *resourceManager) newCreateRequestPayload( // observed and decides whether to call the resource manager's Update method func (rm *resourceManager) Update( 	ctx context.Context,-	res acktypes.AWSResource,+	resDesired acktypes.AWSResource, // desired+	resLatest acktypes.AWSResource, // latest

FYI, we're getting rid of the petstore and bookstore services here:

https://github.com/aws/aws-controllers-k8s/pull/305

So you won't need this after you rebase.

kumargauravsharma

comment created time in 5 days

PullRequestReviewEvent
PullRequestReviewEvent

push eventaws/aws-controllers-k8s

Amine Hilaly

commit sha be5f830c4e69ff7eda22c3a2040273f4e3709188

`ack-generate` calls `git pull` if aws-sdk-go repository already exists

view details

Jay Pipes

commit sha aaffb681058597dd265c9117dcc2a1513fb28ab7

Merge pull request #304 from A-Hilaly/refresh-cache `ack-generate` call `git pull` if aws-sdk-go repository already exist in the cache

view details

push time in 6 days

PR merged aws/aws-controllers-k8s

`ack-generate` call `git pull` if aws-sdk-go repository already exist in the cache

Issue #291

Description of changes:

ack-generate now will call git pull if aws-sdk-go repository already exist in the cache

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

+10 -0

0 comment

2 changed files

A-Hilaly

pr closed time in 6 days

PullRequestReviewEvent

issue commentaws/aws-controllers-k8s

Build for SQS not working ?

Hi @sebgoa! Yeah, I'm working on the fix for this here: #296. Should have a fix pushed within a few hours. Sorry for the delay!

sebgoa

comment created time in 6 days

PR opened aws/aws-controllers-k8s

update apigwv2 service controller for Diff

Recent PR changed the AWSDescriptor interface signature and this PR simply updates the API Gateway V2 service controller to abide by that new interface.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

+132 -60

0 comment

36 changed files

pr created time in 9 days

create barnchjaypipes/aws-controllers-k8s

branch : apigw

created branch time in 9 days

push eventjaypipes/ghw

cclerget

commit sha d3780e365eade533674c295d1ede5a7099d84f53

Determine total physical memory from sysfs for linux as primary method.

view details

Jay Pipes

commit sha 75d641ea8e0d6e71b75c8dbc655740fe9927e65e

Merge pull request #198 from cclerget/physical-mem-from-sysfs Determine total physical memory from sysfs for linux as primary method

view details

push time in 9 days

PR merged jaypipes/ghw

Determine total physical memory from sysfs for linux as primary method

This PR allows to first determine on Linux systems the total of physical memory from sysfs (/sys/devices/system/memory) as primary method and fallback to syslog file method if /sys/devices/system/memory is not present for some reasons

+79 -28

0 comment

2 changed files

cclerget

pr closed time in 9 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentjaypipes/ghw

Determine total physical memory from sysfs for linux as primary method

 func (i *Info) load() error { 	return nil } -func memTotalPhysicalBytes(paths *linuxpath.Paths) int64 {+func memTotalPhysicalBytes(paths *linuxpath.Paths) (total int64) {+	defer func() {+		// fallback to the syslog file approach in case of error+		if total < 0 {+			total = memTotalPhysicalBytesFromSyslog(paths)+		}+	}()++	// detect physical memory from /sys/devices/system/memory+	dir := paths.SysDevicesSystemMemory++	// get the memory block size in byte in hexadecimal notation+	blockSize := filepath.Join(dir, "block_size_bytes")++	d, err := ioutil.ReadFile(blockSize)+	if err != nil {+		return -1+	}+	blockSizeBytes, err := strconv.ParseUint(strings.TrimSpace(string(d)), 16, 64)+	if err != nil {+		return -1+	}++	// iterate over memory's block /sys/devices/system/memory/memory*,+	// if the memory block state is 'online' we increment the total+	// with the memory block size to determine the amount of physical+	// memory available on this system+	sysMemory, err := filepath.Glob(filepath.Join(dir, "memory*"))+	if err != nil {+		return -1+	} else if sysMemory == nil {+		return -1+	}++	for _, path := range sysMemory {+		s, err := ioutil.ReadFile(filepath.Join(path, "state"))+		if err != nil {+			return -1+		}+		if strings.TrimSpace(string(s)) != "online" {+			continue+		}+		total += int64(blockSizeBytes)+	}++	return total+}

Very, very nice. :+1:

cclerget

comment created time in 9 days

push eventaws/aws-controllers-k8s

kumargauravsharma

commit sha 28eabf6df523210024f7fdcd520e5a576b88d91b

Enable multiple update operations support on single resource (#245) Issue #210 Description of changes: The changes in this PR are to enable custom resource update scenario where Update operation on given resource does not support fields that are supported by its Create method, and there exist specialized methods to update such fields. These specialized method response provides the Resource object indicating the Update on the resource. For example: create-replication-group API arguments include: num-node-groups replicas-per-node-group however, modify-replication-group API arguments does not, instead modify-replication-group-shard-configuration and increase-replica-count, decrease-replica-count APIs corresponds to the related Update. The approach taken in this PR is to add the details of such fields and corresponding custom update methods in the generator.yaml. For example: resources: ReplicationGroup: custom_update_operations: UpdateShardConfiguration: fields: - NumNodeGroups UpdateReplicaCount: fields: - ReplicasPerNodeGroup And custom update methods' (UpdateShardConfiguration and UpdateReplicaCount) implementation is provided in a separate code file: aws-controllers-k8s/services/elasticache/pkg/resource/replication_group/custom_update_api.go These methods are supplied with DiffReporter which helps the code in this file determine the specific Update API to invoke based on the difference in desired, current resource details. Co-authored-by: Kumar Gaurav Sharma <kumargsh@amazon.com>

view details

push time in 9 days

PR merged aws/aws-controllers-k8s

Enable multiple update operations support on single resource

Issue #210

Description of changes: The changes in this PR are to enable custom resource update scenario where Update operation on given resource does not support fields that are supported by its Create method, and there exist specialized methods to update such fields. These specialized method response provides the Resource object indicating the Update on the resource.

For example: create-replication-group API arguments include:

num-node-groups
replicas-per-node-group

however, modify-replication-group API arguments does not, instead modify-replication-group-shard-configuration and increase-replica-count, decrease-replica-count APIs corresponds to the related Update.

The approach taken in this PR is to add the details of such fields and corresponding custom update methods in the generator.yaml. For example:

resources:
  ReplicationGroup:
    custom_update_operations:
      UpdateShardConfiguration:
        fields:
          - NumNodeGroups
      UpdateReplicaCount:
        fields:
          - ReplicasPerNodeGroup

And custom update methods' (UpdateShardConfiguration and UpdateReplicaCount) implementation is provided in a separate code file: aws-controllers-k8s/services/elasticache/pkg/resource/replication_group/custom_update_api.go

These methods are supplied with DiffReporter which helps the code in this file determine the specific Update API to invoke based on the difference in desired, current resource details.

Auto generating following:

make build-controller SERVICE=sns
make build-controller SERVICE=ecr
make build-controller SERVICE=s3

and after updating bookstore, petstore services files. make test passed.

Changes include autogenerated sns, ect, s3, and manually updated bookstore, petstore service files per the latest APIs.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

+718 -117

0 comment

50 changed files

kumargauravsharma

pr closed time in 9 days

PullRequestReviewEvent

push eventaws/aws-controllers-k8s

vijtrip2

commit sha 3a3a504137fdc1874a8d54c645dcb21156cc0286

apigatewayv2 service controller and smoke test files. (#293) * apigatewayv2 service controller and smoke test files.

view details

push time in 9 days

more