profile
viewpoint
Marcin Cuber marcincuber London marcincuber.github.io Lead Software/DevOps Engineer. “Any fool can write code that a computer can understand. Good programmers write code that humans can understand.” - M. Fowler

marcincuber/eks 38

AWS EKS - kubernetes project

marcincuber/modal_logic 16

Final Year Masters Project: modal logic solver tableaux

marcincuber/aws-cfm-utils 8

NPM module for AWS CloudFormation templates/stacks deployments

marcincuber/aws-scripts 5

AWS scripts used for iam rotation, s3 bucket logging etc.

marcincuber/gcloud-gke 5

Google Kubernetes Engine- terraform templates

marcincuber/k8s-flux 4

Example deployment of resources using flux

marcincuber/dotfiles 3

shell, git, etc config files, custom functions and more

jonathanmanfield/3001 2

COMP3001:Technology Management and Professional Issues - Team 5 GitHub Repository

marcincuber/aws-lambda-github-sg 2

AWS lambda function to automatically update security groups with github IPs using tags.

marcincuber/kubernetes-gitlab-runner 2

Gitlab runners configuration yamls to run in Kubernetes

issue closedmarcincuber/eks

few questions

Hi Marcin,

back again with few questions, as I am really feeling your work has been one of the very best put there for other DevOps to get their EKS work and running in a better manner.

I have three question, I would appreciate if you can give some clarification.

Question 1/ node-drainer.tf what is the file in node_drainer folder that we are zipping? There are three folders inside node_drainer, each folder has few files in it? Can you kindly give some explanation there?

I want to use terraform to zip the file on the fly instead of having zip file in the repo (for security compliance purposes). for example I want to zip the node_drainer as below:

data "archive_file" "node-draining_zip" { type = "zip" output_path = "${path.module}/node_drainer/???-${sha256(file("${path.module}/node_drainer/???.py"))}.zip" source_file = "${path.module}/node_drainer/???.py" }

then call it like this: source_code_hash = data.archive_file.node-draining_zip.output_base64sha256

Question 2/ There is a security group you are creating but seems like this hasn’t been used anywhere? https://github.com/marcincuber/eks/blob/master/terraform-aws/cluster.tf#L5 Or am I missing something?

Question 3/ for all configs provided in terraform_k8s inclusing configmap aws-auth and cluster_autoscaler, etc. is there any automated process you are using to configure these templates onto the deployed clusters and node_group?

Thank you very much.

closed time in 13 days

nashvan

push eventmarcincuber/eks

Marcin Cuber

commit sha ac81cdf41513cb27b739b3b468b4eef837b15cde

Update versions.tf

view details

push time in 13 days

push eventmarcincuber/eks

Marcin Cuber

commit sha 9b8e81dbc3a39fd87a65f1ee05892c41dca96551

Update manage-node-group.tf

view details

push time in 13 days

issue commentmarcincuber/eks

few questions

@nashvan you are absolutely right. I upgraded the module to support terraform 0.13. This means terraform-aws-eks-node-group no longer needs enabled variable and therefore it was removed.

It should be changed to (enabled is removed and terraform 0.13 required):

module "eks-node-group" {
  source = "umotif-public/eks-node-group/aws"
  version = "~> 3.0.0"
  count = var.enable_managed_workers ? 1 : 0
  ...
nashvan

comment created time in 13 days

issue commentmarcincuber/eks

few questions

pip3 install -r ./node_drainer/drainer/requirements.txt --target ./lambda_build
cp ./node_drainer/drainer/__init__.py ./node_drainer/drainer/handler.py ./node_drainer/drainer/k8s_utils.py ./lambda_build
cd lambda_build && zip -r ../node_drainer.zip ./* && cd .. && rm -rf lambda_build

These are commands that I am using locally to build the zip.

nashvan

comment created time in 13 days

issue commentmarcincuber/eks

few questions

@nashvan apologies. So the files that are zipped are in https://github.com/marcincuber/eks/tree/master/terraform-aws/node_drainer/drainer. I am zipping it into a single zip as it is not going to change anytime soon. So files handler.py and k8s_utils.py are the crucial ones.

nashvan

comment created time in 13 days

issue commentmarcincuber/eks

few questions

Hi @nashvan,

I am glad my work is helping other people.

Q1. Lambda function/code provides a means to gracefully terminate nodes in EKS cluster when managed as part of an Amazon EC2 Auto Scaling Group. For example, when you perform rolling update, nodes need to be drained before they get terminated.

The code provides an AWS Lambda function that integrates as an Amazon EC2 Auto Scaling Lifecycle Hook. When called, the Lambda function calls the Kubernetes API to cordon and evict all evictable pods from the node being terminated. It will then wait until all pods have been evicted before the Auto Scaling group continues to terminate the EC2 instance. The lambda may be killed by the function timeout before all evictions complete successfully, in which case the lifecycle hook may re-execute the lambda to try again. If the lifecycle heartbeat expires then termination of the EC2 instance will continue regardless of whether or not draining was successful. You may need to increase the function and heartbeat timeouts in template.yaml if you have very long grace periods.

Using this approach can minimise disruption to the services running in your cluster by allowing Kubernetes to reschedule the pod prior to the instance being terminated enters the TERMINATING state. It works by using Amazon EC2 Auto Scaling Lifecycle Hooks to trigger an AWS Lambda function that uses the Kubernetes API to cordon the node and evict the pods.

Q2. Just to be precise, it is a aws_security_group_rule. This one specifically allows EKS to make use of endpoints configured in https://github.com/marcincuber/eks/blob/master/terraform-aws/vpc.tf#L56. VPC endpoints allow for private connectivity with the VPC.

Q3. There are different ways to deploying yaml templates. Personally, I am making use of a tool called Flux (https://github.com/fluxcd/flux). Using flux, you are basically control templates in your git repository and flux will apply them in the cluster.

nashvan

comment created time in 13 days

issue closedmarcincuber/eks

Instance type for third node group in the AZ-C is missing

Hello Marcin. I have been going through your excellent work since yesterday, found that instance type is missing for the third AZ module "eks-node-group-c" { https://github.com/marcincuber/eks/blob/master/terraform-aws/manage-node-group.tf#L57

is that a deliberate move to let ng in the third AZ get the default instance type? or that is just been missed out?

closed time in 19 days

nashvan

issue commentmarcincuber/eks

Instance type for third node group in the AZ-C is missing

It is not missing, it is just another way to demonstrate that terraform module https://github.com/umotif-public/terraform-aws-eks-node-group is working as expected. And as you rightly mentioned it will default to instance_types = ["t3.medium"]

nashvan

comment created time in 19 days

push eventmarcincuber/eks

Marcin Cuber

commit sha ff9774912c73e7f0064e25ad6a77acf89dcb2c19

Update README.md

view details

push time in 20 days

push eventmarcincuber/eks

Marcin Cuber

commit sha 4a0f826e035be23dc1622259728e70f1d79bd01b

Update README.md

view details

push time in 20 days

push eventumotif-public/terraform-aws-eks-node-group

marcincuber

commit sha 7aaf7f7b0ccc7f8553212f06dd0b91bf9dd69909

Update changelog

view details

push time in 22 days

created tagumotif-public/terraform-aws-eks-node-group

tag3.0.0

Terraform module to provision EKS Managed Node Group

created time in 22 days

release umotif-public/terraform-aws-eks-node-group

3.0.0

released time in 22 days

delete branch umotif-public/terraform-aws-eks-node-group

delete branch : feature/launch-template

delete time in 22 days

push eventumotif-public/terraform-aws-eks-node-group

Marcin Cuber

commit sha 6790e54b6e846279b648d5c06502ef3b7983b55e

Add support for launch template and tf 0.13 (#6)

view details

push time in 22 days

PR merged umotif-public/terraform-aws-eks-node-group

Add support for launch template and tf 0.13

Description

  • Add support for launch templates
  • Fully support terraform 0.13
  • Remove enabled variable: count or for_each can be used with terraform 0.13
  • Add example with working launch template configuration
  • Upgrade minimum required version of terraform aws provider
  • Update docs

This PR forms major release 3.0.0 of this module

+219 -39

0 comment

10 changed files

marcincuber

pr closed time in 22 days

push eventumotif-public/terraform-aws-eks-node-group

marcincuber

commit sha 70d6d9019200a2671ec67a1ac2371fe3300393be

Add support for launch template and tf 0.13

view details

push time in 22 days

PR opened umotif-public/terraform-aws-eks-node-group

Add support for launch template and tf 0.13

Description

  • Add support for launch templates
  • Fully support terraform 0.13
  • Remove enabled variable: count or for_each can be used with terraform 0.13
  • Add example with working launch template configuration
  • Upgrade minimum required version of terraform aws provider
  • Update docs

This PR forms major release 3.0.0 of this module

+219 -39

0 comment

10 changed files

pr created time in 22 days

created tagumotif-public/terraform-aws-rds-aurora

tag1.2.2

Terraform module which creates AWS RDS Aurora resources

created time in a month

release umotif-public/terraform-aws-rds-aurora

1.2.2

released time in a month

PR opened umotif-public/terraform-aws-rds-aurora

Fix/cluster instance lifecycle

Description

Lifecycle rule added to aws_rds_cluster_instance allows for fluent upgrades of aurora engine versions.

+22 -9

0 comment

5 changed files

pr created time in a month

issue closedmarcincuber/eks

How can I attach spot workers to cluster?

Hello, I'm looking your settings in the latest configurations which available in terraform-aws, and I realize the deployment, I can see every was deploy. I have 3 m5 Spot instances running, and the eks cluster too

But when I run kubectl get nodes

This is the ouput.

No resources found in default namespace.

What is the right process to achieve the kubectl get nodes show me the spot instances?

Maybe, Can you help with that?, I feel a little lost and I really need your help for achieve that.

Thank you.

closed time in a month

frcho

issue commentmarcincuber/eks

How can I attach spot workers to cluster?

@frcho With my templates you already have nicely configured spot worker nodes using launch templates + auto scaling group. You can enable them by setting variable enable_spot_workers to true. Take a look at spot-worker-nodes.tf it should give you a good idea how it all works. Cloudformation template used to control spot instances can be found in cfm/worker-node-spot-stack.yaml.

I wouldn't recommend using eks module since you simply not going to have a rollingUpdate ability by using only terraform. Also, using spot fleet is not needed, much better alternative is to use launch templates + asg.

frcho

comment created time in a month

created tagumotif-public/terraform-aws-elasticache-redis

tag1.1.0

A Terraform module to create an AWS Redis ElastiCache cluster.

created time in a month

release umotif-public/terraform-aws-elasticache-redis

1.1.0

released time in a month

push eventumotif-public/terraform-aws-elasticache-redis

Marcin Cuber

commit sha 208732163ed2a602e4de36bb72fc91312b9d15a1

Update main.tf

view details

push time in a month

issue commenthashicorp/terraform

init command should support workspaces block in backend config file

I can config that JSON syntax is working as fine with terraform 0.13. Is there a plan to add support for previous HCL syntax or is this the final solution going forward?

ToruMakabe

comment created time in a month

issue openedterraform-providers/terraform-provider-aws

Managed Node Groups Launch Template Support

<!--- Please keep this note for the community --->

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

<!--- Thank you for keeping this note for the community --->

Description

Add support for Managed Node Groups Launch Template Support

References

<!--- Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests

Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor blog posts or documentation? For example:

  • https://aws.amazon.com/about-aws/whats-new/2018/04/introducing-amazon-ec2-fleet/ --->

  • See the launch blog and EKS documentation for more details

created time in a month

issue commenthashicorp/terraform

data could not be decoded from the state: unsupported attribute "request_parameters_in_json" after upgrade to 0.13

I am seeing this issue with resources and data blocks. This must not require editing manually any states, it would be enormous job at least for me to do. I feel like such a major issue should have been captured during Release Candidate testing 👎

bit4bit

comment created time in a month

delete branch marcincuber/eks

delete branch : feature/volume

delete time in 2 months

push eventmarcincuber/eks

marcincuber

commit sha aacaf3390506b42cbb0b8853bc0fb700570f6879

Add ability to modify volumes attached to worker nodes

view details

push time in 2 months

create barnchmarcincuber/eks

branch : feature/volume

created branch time in 2 months

pull request commentterraform-aws-modules/terraform-aws-vpc

feat: aws-provider version bump to >= 2.57

Can we get this merged and released asap? It is not a breaking change and it will allow to start testing it with other modules which are already using v3 aws provider.

michalschott

comment created time in 2 months

issue closedterraform-aws-modules/terraform-aws-rds-aurora

AWS Secrets Manager + Aurora RDS module

Hi,

I would like to make use of AWS Secrets Manager to rotate root credentials for the aurora db mysql. Could you advise how to do it using this module or maybe you have some useful example where this is being done?

closed time in 2 months

marcincuber

created tagumotif-public/terraform-aws-ecs-fargate-scheduled-task

tag1.0.1

Terraform module to create AWS ECS Fargate Schedule Task

created time in 2 months

created tagumotif-public/terraform-aws-ecs-fargate-task-definition

tag2.0.1

Terraform module to create AWS ECS Fargate Task Definition

created time in 2 months

created tagumotif-public/terraform-aws-ecs-service-autoscaling-cloudwatch

tag2.0.1

Terraform module to configure ECS Service autoscaling using CloudWatch metrics

created time in 2 months

created tagumotif-public/terraform-aws-eks-fargate-profile

tag1.0.1

Terraform module to configure an EKS Fargate Profile

created time in 2 months

release umotif-public/terraform-aws-eks-fargate-profile

1.0.0

released time in 2 months

created tagumotif-public/terraform-aws-eks-node-group

tag2.0.1

Terraform module to provision EKS Managed Node Group

created time in 2 months

release umotif-public/terraform-aws-eks-node-group

2.0.1

released time in 2 months

created tagumotif-public/terraform-aws-elasticache-redis

tag1.0.3

A Terraform module to create an AWS Redis ElastiCache cluster.

created time in 2 months

release umotif-public/terraform-aws-elasticache-redis

1.0.3

released time in 2 months

created tagumotif-public/terraform-aws-kms

tag1.0.1

Terraform module to configure a KMS Customer Master Key (CMK) and its alias.

created time in 2 months

release umotif-public/terraform-aws-kms

1.0.1

released time in 2 months

created tagumotif-public/terraform-aws-rds-aurora

tag1.2.1

Terraform module which creates AWS RDS Aurora resources

created time in 2 months

release umotif-public/terraform-aws-rds-aurora

1.2.1

released time in 2 months

created tagumotif-public/terraform-aws-ses-domain

tag1.0.1

Terraform module to configure a domain hosted on Route53 to work with AWS Simple Email Service (SES).

created time in 2 months

release umotif-public/terraform-aws-ses-domain

1.0.1

released time in 2 months

created tagumotif-public/terraform-aws-bastion

tag1.4.1

Terraform module to create Bastion Host in AWS VPC.

created time in 2 months

release umotif-public/terraform-aws-bastion

1.4.1

released time in 2 months

created tagumotif-public/terraform-aws-ecs-fargate

tag4.0.1

Terraform module to create AWS ECS FARGATE services

created time in 2 months

release umotif-public/terraform-aws-ecs-fargate

4.0.1

released time in 2 months

created tagumotif-public/terraform-aws-ssm-parameters

tag1.0.1

Terraform module creating encrypted and non-encrypted AWS SSM parameters

created time in 2 months

release umotif-public/terraform-aws-ssm-parameters

1.0.1

released time in 2 months

created tagumotif-public/terraform-aws-vpc-flow-logs

tag1.1.1

Terraform module for enabling AWS VPC flow logs.

created time in 2 months

release umotif-public/terraform-aws-vpc-flow-logs

1.1.1

released time in 2 months

created tagumotif-public/terraform-aws-waf-webaclv2

tag1.4.1

Terraform module to configure WAF V2 Web ACL with managed rules for Application Load Balancer

created time in 2 months

release umotif-public/terraform-aws-waf-webaclv2

1.4.1

released time in 2 months

pull request commentumotif-public/terraform-aws-waf-webaclv2

Feature/v3 provider support

It doesn't look like you made any functional changes to the module code. What is the purpose of this PR. To update documentation?

Major change is in the versions.tf and this module can now be used with v3 of aws terraform provider. Additionally, some documentation updates.

marcincuber

comment created time in 2 months

PR opened umotif-public/terraform-aws-waf-webaclv2

Feature/v3 provider support

Description

Add support for v3 AWS provider

+20 -11

0 comment

4 changed files

pr created time in 2 months

PR opened umotif-public/terraform-aws-vpc-flow-logs

Feature/v3 provider support

Description

Add support for v3 AWS provider

+10 -6

0 comment

4 changed files

pr created time in 2 months

PR opened umotif-public/terraform-aws-ssm-parameters

Feature/v3 provider support

Description

Add support for v3 AWS provider

+16 -4

0 comment

4 changed files

pr created time in 2 months

push eventumotif-public/terraform-aws-ses-domain

Marcin Cuber

commit sha bc83761004ad3ebd94490c78f05e1b06381782b7

Feature/updates (#1) * update docs and add chglog * Add CHANGELOG.md

view details

marcincuber

commit sha 02bcb07e9484cf336ebdc5b013d1653990024253

Update module versions to support v3 provider

view details

marcincuber

commit sha 90900a5371b9fa321f5499787036ca131860d5e9

update CHANGELOG

view details

push time in 2 months

PR opened umotif-public/terraform-aws-ses-domain

Feature/v3 provider support

Description

Add support for v3 AWS provider

+177 -10

0 comment

7 changed files

pr created time in 2 months

PR opened umotif-public/terraform-aws-rds-aurora

Feature/v3 provider support

Description

Add support for v3 AWS provider

+20 -11

0 comment

4 changed files

pr created time in 2 months

PR opened umotif-public/terraform-aws-kms

Feature/v3 provider support

Description

Add support for v3 AWS provider

+16 -5

0 comment

4 changed files

pr created time in 2 months

create barnchumotif-public/terraform-aws-kms

branch : feature/v3-provider-support

created branch time in 2 months

PR opened umotif-public/terraform-aws-eks-node-group

Feature/v3 provider support

Description

Add support for v3 AWS provider

+18 -7

0 comment

4 changed files

pr created time in 2 months

push eventumotif-public/terraform-aws-eks-fargate-profile

marcincuber

commit sha 3dcbfc8fead9c851d2fea41969cdde262d8796e9

make labels conditional

view details

push time in 2 months

PR opened umotif-public/terraform-aws-ecs-fargate

Feature/v3 provider support

Description

Add support for v3 AWS provider

+16 -4

0 comment

3 changed files

pr created time in 2 months

PR opened umotif-public/terraform-aws-bastion

Feature/v3 provider support

Description

Add support for v3 AWS provider

+23 -9

0 comment

4 changed files

pr created time in 2 months

create barnchumotif-public/terraform-aws-bastion

branch : feature/v3-provider-support

created branch time in 2 months

created tagumotif-public/terraform-aws-alb

tag1.2.1

A Terraform module to create an AWS Application Load Balancer (ALB).

created time in 2 months

release umotif-public/terraform-aws-alb

1.2.1

released time in 2 months

PR opened umotif-public/terraform-aws-alb

Improve module versions + add alb access logs example

Description

  • Improve version locking to allow aws provider v3
  • Add example with ALB and S3 access logs
+164 -9

0 comment

5 changed files

pr created time in 2 months

create barnchumotif-public/terraform-aws-alb

branch : feature/version

created branch time in 2 months

more