AWS EKS - kubernetes project
Final Year Masters Project: modal logic solver tableaux
NPM module for AWS CloudFormation templates/stacks deployments
AWS scripts used for iam rotation, s3 bucket logging etc.
Google Kubernetes Engine- terraform templates
Example deployment of resources using flux
shell, git, etc config files, custom functions and more
COMP3001:Technology Management and Professional Issues - Team 5 GitHub Repository
marcincuber/aws-lambda-github-sg 2
AWS lambda function to automatically update security groups with github IPs using tags.
marcincuber/kubernetes-gitlab-runner 2
Gitlab runners configuration yamls to run in Kubernetes
issue closedmarcincuber/eks
Hi Marcin,
back again with few questions, as I am really feeling your work has been one of the very best put there for other DevOps to get their EKS work and running in a better manner.
I have three question, I would appreciate if you can give some clarification.
Question 1/ node-drainer.tf what is the file in node_drainer folder that we are zipping? There are three folders inside node_drainer, each folder has few files in it? Can you kindly give some explanation there?
I want to use terraform to zip the file on the fly instead of having zip file in the repo (for security compliance purposes). for example I want to zip the node_drainer as below:
data "archive_file" "node-draining_zip" { type = "zip" output_path = "${path.module}/node_drainer/???-${sha256(file("${path.module}/node_drainer/???.py"))}.zip" source_file = "${path.module}/node_drainer/???.py" }
then call it like this: source_code_hash = data.archive_file.node-draining_zip.output_base64sha256
Question 2/ There is a security group you are creating but seems like this hasn’t been used anywhere? https://github.com/marcincuber/eks/blob/master/terraform-aws/cluster.tf#L5 Or am I missing something?
Question 3/ for all configs provided in terraform_k8s inclusing configmap aws-auth and cluster_autoscaler, etc. is there any automated process you are using to configure these templates onto the deployed clusters and node_group?
Thank you very much.
closed time in 13 days
nashvan
push eventmarcincuber/eks
commit sha ac81cdf41513cb27b739b3b468b4eef837b15cde
Update versions.tf
push time in 13 days
push eventmarcincuber/eks
commit sha 9b8e81dbc3a39fd87a65f1ee05892c41dca96551
Update manage-node-group.tf
push time in 13 days
issue commentmarcincuber/eks
@nashvan you are absolutely right. I upgraded the module to support terraform 0.13. This means terraform-aws-eks-node-group no longer needs enabled variable and therefore it was removed.
It should be changed to (enabled is removed and terraform 0.13 required):
module "eks-node-group" {
source = "umotif-public/eks-node-group/aws"
version = "~> 3.0.0"
count = var.enable_managed_workers ? 1 : 0
...
comment created time in 13 days
issue commentmarcincuber/eks
pip3 install -r ./node_drainer/drainer/requirements.txt --target ./lambda_build
cp ./node_drainer/drainer/__init__.py ./node_drainer/drainer/handler.py ./node_drainer/drainer/k8s_utils.py ./lambda_build
cd lambda_build && zip -r ../node_drainer.zip ./* && cd .. && rm -rf lambda_build
These are commands that I am using locally to build the zip.
comment created time in 13 days
issue commentmarcincuber/eks
@nashvan apologies. So the files that are zipped are in https://github.com/marcincuber/eks/tree/master/terraform-aws/node_drainer/drainer. I am zipping it into a single zip as it is not going to change anytime soon. So files handler.py and
k8s_utils.py are the crucial ones.
comment created time in 13 days
issue commentmarcincuber/eks
Hi @nashvan,
I am glad my work is helping other people.
Q1. Lambda function/code provides a means to gracefully terminate nodes in EKS cluster when managed as part of an Amazon EC2 Auto Scaling Group. For example, when you perform rolling update, nodes need to be drained before they get terminated.
The code provides an AWS Lambda function that integrates as an Amazon EC2 Auto Scaling Lifecycle Hook. When called, the Lambda function calls the Kubernetes API to cordon and evict all evictable pods from the node being terminated. It will then wait until all pods have been evicted before the Auto Scaling group continues to terminate the EC2 instance. The lambda may be killed by the function timeout before all evictions complete successfully, in which case the lifecycle hook may re-execute the lambda to try again. If the lifecycle heartbeat expires then termination of the EC2 instance will continue regardless of whether or not draining was successful. You may need to increase the function and heartbeat timeouts in template.yaml if you have very long grace periods.
Using this approach can minimise disruption to the services running in your cluster by allowing Kubernetes to reschedule the pod prior to the instance being terminated enters the TERMINATING state. It works by using Amazon EC2 Auto Scaling Lifecycle Hooks to trigger an AWS Lambda function that uses the Kubernetes API to cordon the node and evict the pods.
Q2. Just to be precise, it is a aws_security_group_rule. This one specifically allows EKS to make use of endpoints configured in https://github.com/marcincuber/eks/blob/master/terraform-aws/vpc.tf#L56. VPC endpoints allow for private connectivity with the VPC.
Q3. There are different ways to deploying yaml templates. Personally, I am making use of a tool called Flux (https://github.com/fluxcd/flux). Using flux, you are basically control templates in your git repository and flux will apply them in the cluster.
comment created time in 13 days
issue closedmarcincuber/eks
Instance type for third node group in the AZ-C is missing
Hello Marcin. I have been going through your excellent work since yesterday, found that instance type is missing for the third AZ module "eks-node-group-c" { https://github.com/marcincuber/eks/blob/master/terraform-aws/manage-node-group.tf#L57
is that a deliberate move to let ng in the third AZ get the default instance type? or that is just been missed out?
closed time in 19 days
nashvan
issue commentmarcincuber/eks
Instance type for third node group in the AZ-C is missing
It is not missing, it is just another way to demonstrate that terraform module https://github.com/umotif-public/terraform-aws-eks-node-group is working as expected. And as you rightly mentioned it will default to instance_types = ["t3.medium"]
comment created time in 19 days
push eventmarcincuber/eks
commit sha ff9774912c73e7f0064e25ad6a77acf89dcb2c19
Update README.md
push time in 20 days
push eventmarcincuber/eks
commit sha 4a0f826e035be23dc1622259728e70f1d79bd01b
Update README.md
push time in 20 days
push eventumotif-public/terraform-aws-eks-node-group
commit sha 7aaf7f7b0ccc7f8553212f06dd0b91bf9dd69909
Update changelog
push time in 22 days
created tagumotif-public/terraform-aws-eks-node-group
Terraform module to provision EKS Managed Node Group
created time in 22 days
delete branch umotif-public/terraform-aws-eks-node-group
delete branch : feature/launch-template
delete time in 22 days
push eventumotif-public/terraform-aws-eks-node-group
commit sha 6790e54b6e846279b648d5c06502ef3b7983b55e
Add support for launch template and tf 0.13 (#6)
push time in 22 days
PR merged umotif-public/terraform-aws-eks-node-group
Description
- Add support for launch templates
- Fully support terraform 0.13
- Remove enabled variable: count or for_each can be used with terraform 0.13
- Add example with working launch template configuration
- Upgrade minimum required version of terraform aws provider
- Update docs
This PR forms major release 3.0.0 of this module
pr closed time in 22 days
push eventumotif-public/terraform-aws-eks-node-group
commit sha 70d6d9019200a2671ec67a1ac2371fe3300393be
Add support for launch template and tf 0.13
push time in 22 days
PR opened umotif-public/terraform-aws-eks-node-group
Description
- Add support for launch templates
- Fully support terraform 0.13
- Remove enabled variable: count or for_each can be used with terraform 0.13
- Add example with working launch template configuration
- Upgrade minimum required version of terraform aws provider
- Update docs
This PR forms major release 3.0.0 of this module
pr created time in 22 days
create barnchumotif-public/terraform-aws-eks-node-group
branch : feature/launch-template
created branch time in 22 days
created tagumotif-public/terraform-aws-rds-aurora
Terraform module which creates AWS RDS Aurora resources
created time in a month
PR opened umotif-public/terraform-aws-rds-aurora
Description
Lifecycle rule added to aws_rds_cluster_instance allows for fluent upgrades of aurora engine versions.
pr created time in a month
create barnchumotif-public/terraform-aws-rds-aurora
branch : fix/cluster-instance-lifecycle
created branch time in a month
issue closedmarcincuber/eks
How can I attach spot workers to cluster?
Hello, I'm looking your settings in the latest configurations which available in terraform-aws, and I realize the deployment, I can see every was deploy. I have 3 m5 Spot instances running, and the eks cluster too
But when I run kubectl get nodes
This is the ouput.
No resources found in default namespace.
What is the right process to achieve the kubectl get nodes show me the spot instances?
Maybe, Can you help with that?, I feel a little lost and I really need your help for achieve that.
Thank you.
closed time in a month
frcho
issue commentmarcincuber/eks
How can I attach spot workers to cluster?
@frcho With my templates you already have nicely configured spot worker nodes using launch templates + auto scaling group. You can enable them by setting variable enable_spot_workers to true. Take a look at spot-worker-nodes.tf it should give you a good idea how it all works. Cloudformation template used to control spot instances can be found in cfm/worker-node-spot-stack.yaml.
I wouldn't recommend using eks module since you simply not going to have a rollingUpdate ability by using only terraform. Also, using spot fleet is not needed, much better alternative is to use launch templates + asg.
comment created time in a month
created tagumotif-public/terraform-aws-elasticache-redis
A Terraform module to create an AWS Redis ElastiCache cluster.
created time in a month
push eventumotif-public/terraform-aws-elasticache-redis
commit sha 208732163ed2a602e4de36bb72fc91312b9d15a1
Update main.tf
push time in a month
issue commenthashicorp/terraform
init command should support workspaces block in backend config file
I can config that JSON syntax is working as fine with terraform 0.13. Is there a plan to add support for previous HCL syntax or is this the final solution going forward?
comment created time in a month
issue openedterraform-providers/terraform-provider-aws
Managed Node Groups Launch Template Support
<!--- Please keep this note for the community --->
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
<!--- Thank you for keeping this note for the community --->
Description
Add support for Managed Node Groups Launch Template Support
References
<!--- Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests
Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor blog posts or documentation? For example:
-
https://aws.amazon.com/about-aws/whats-new/2018/04/introducing-amazon-ec2-fleet/ --->
-
See the launch blog and EKS documentation for more details
created time in a month
issue commenthashicorp/terraform
I am seeing this issue with resources and data blocks. This must not require editing manually any states, it would be enormous job at least for me to do. I feel like such a major issue should have been captured during Release Candidate testing 👎
comment created time in a month
push eventmarcincuber/eks
commit sha aacaf3390506b42cbb0b8853bc0fb700570f6879
Add ability to modify volumes attached to worker nodes
push time in 2 months
pull request commentterraform-aws-modules/terraform-aws-vpc
feat: aws-provider version bump to >= 2.57
Can we get this merged and released asap? It is not a breaking change and it will allow to start testing it with other modules which are already using v3 aws provider.
comment created time in 2 months
issue closedterraform-aws-modules/terraform-aws-rds-aurora
AWS Secrets Manager + Aurora RDS module
Hi,
I would like to make use of AWS Secrets Manager to rotate root credentials for the aurora db mysql. Could you advise how to do it using this module or maybe you have some useful example where this is being done?
closed time in 2 months
marcincuber
created tagumotif-public/terraform-aws-ecs-fargate-scheduled-task
Terraform module to create AWS ECS Fargate Schedule Task
created time in 2 months
created tagumotif-public/terraform-aws-ecs-fargate-task-definition
Terraform module to create AWS ECS Fargate Task Definition
created time in 2 months
created tagumotif-public/terraform-aws-ecs-service-autoscaling-cloudwatch
Terraform module to configure ECS Service autoscaling using CloudWatch metrics
created time in 2 months
created tagumotif-public/terraform-aws-eks-fargate-profile
Terraform module to configure an EKS Fargate Profile
created time in 2 months
created tagumotif-public/terraform-aws-eks-node-group
Terraform module to provision EKS Managed Node Group
created time in 2 months
created tagumotif-public/terraform-aws-elasticache-redis
A Terraform module to create an AWS Redis ElastiCache cluster.
created time in 2 months
created tagumotif-public/terraform-aws-kms
Terraform module to configure a KMS Customer Master Key (CMK) and its alias.
created time in 2 months
created tagumotif-public/terraform-aws-rds-aurora
Terraform module which creates AWS RDS Aurora resources
created time in 2 months
created tagumotif-public/terraform-aws-ses-domain
Terraform module to configure a domain hosted on Route53 to work with AWS Simple Email Service (SES).
created time in 2 months
created tagumotif-public/terraform-aws-bastion
Terraform module to create Bastion Host in AWS VPC.
created time in 2 months
created tagumotif-public/terraform-aws-ecs-fargate
Terraform module to create AWS ECS FARGATE services
created time in 2 months
created tagumotif-public/terraform-aws-ssm-parameters
Terraform module creating encrypted and non-encrypted AWS SSM parameters
created time in 2 months
created tagumotif-public/terraform-aws-vpc-flow-logs
Terraform module for enabling AWS VPC flow logs.
created time in 2 months
created tagumotif-public/terraform-aws-waf-webaclv2
Terraform module to configure WAF V2 Web ACL with managed rules for Application Load Balancer
created time in 2 months
pull request commentumotif-public/terraform-aws-waf-webaclv2
It doesn't look like you made any functional changes to the module code. What is the purpose of this PR. To update documentation?
Major change is in the versions.tf and this module can now be used with v3 of aws terraform provider. Additionally, some documentation updates.
comment created time in 2 months
PR opened umotif-public/terraform-aws-waf-webaclv2
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-waf-webaclv2
branch : feature/v3-provider-support
created branch time in 2 months
PR opened umotif-public/terraform-aws-vpc-flow-logs
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-vpc-flow-logs
branch : feature/v3-provider-support
created branch time in 2 months
PR opened umotif-public/terraform-aws-ssm-parameters
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-ssm-parameters
branch : feature/v3-provider-support
created branch time in 2 months
push eventumotif-public/terraform-aws-ses-domain
commit sha bc83761004ad3ebd94490c78f05e1b06381782b7
Feature/updates (#1) * update docs and add chglog * Add CHANGELOG.md
commit sha 02bcb07e9484cf336ebdc5b013d1653990024253
Update module versions to support v3 provider
commit sha 90900a5371b9fa321f5499787036ca131860d5e9
update CHANGELOG
push time in 2 months
PR opened umotif-public/terraform-aws-ses-domain
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-ses-domain
branch : feature/v3-provider-support
created branch time in 2 months
PR opened umotif-public/terraform-aws-rds-aurora
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-rds-aurora
branch : feature/v3-provider-support
created branch time in 2 months
PR opened umotif-public/terraform-aws-kms
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-kms
branch : feature/v3-provider-support
created branch time in 2 months
PR opened umotif-public/terraform-aws-elasticache-redis
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-elasticache-redis
branch : feature/v3-provider-support
created branch time in 2 months
PR opened umotif-public/terraform-aws-eks-node-group
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-eks-node-group
branch : feature/v3-provider-support
created branch time in 2 months
push eventumotif-public/terraform-aws-eks-fargate-profile
commit sha 3dcbfc8fead9c851d2fea41969cdde262d8796e9
make labels conditional
push time in 2 months
PR opened umotif-public/terraform-aws-eks-fargate-profile
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-eks-fargate-profile
branch : feature/v3-provider-support
created branch time in 2 months
PR opened umotif-public/terraform-aws-ecs-service-autoscaling-cloudwatch
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-ecs-service-autoscaling-cloudwatch
branch : feature/v3-provider-support
created branch time in 2 months
PR opened umotif-public/terraform-aws-ecs-fargate-task-definition
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-ecs-fargate-task-definition
branch : feature/v3-provider-support
created branch time in 2 months
PR opened umotif-public/terraform-aws-ecs-fargate-scheduled-task
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-ecs-fargate-scheduled-task
branch : feature/v3-provider-support
created branch time in 2 months
PR opened umotif-public/terraform-aws-ecs-fargate
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-ecs-fargate
branch : feature/v3-provider-support
created branch time in 2 months
PR opened umotif-public/terraform-aws-bastion
Description
Add support for v3 AWS provider
pr created time in 2 months
create barnchumotif-public/terraform-aws-bastion
branch : feature/v3-provider-support
created branch time in 2 months
created tagumotif-public/terraform-aws-alb
A Terraform module to create an AWS Application Load Balancer (ALB).
created time in 2 months
PR opened umotif-public/terraform-aws-alb
Description
- Improve version locking to allow aws provider v3
- Add example with ALB and S3 access logs
pr created time in 2 months
create barnchumotif-public/terraform-aws-alb
created branch time in 2 months