profile
viewpoint
Bryan™ brycahta QuaranTina Belcher

brycahta/amazon-ec2-instance-qualifier 0

A CLI tool that automates benchmarking on a range of EC2 instance types.

brycahta/amazon-ec2-instance-selector 0

A CLI tool and go library which recommends instance types based on resource criteria like vcpus and memory

brycahta/amazon-ec2-metadata-mock 0

A tool to simulate Amazon EC2 instance metadata

brycahta/aws-node-termination-handler 0

A Kubernetes Daemonset to gracefully handle EC2 instance shutdown

brycahta/container-web-scraper-example 0

An example of a web scraper in a Docker container running on AWS

brycahta/eks-charts 0

Amazon EKS Helm chart repository

Pull request review commentawslabs/amazon-ec2-instance-qualifier

integrate cloudwatch agent

+{

these changes will be addressed in separate PR

brycahta

comment created time in 2 days

PullRequestReviewEvent

push eventbrycahta/amazon-ec2-instance-qualifier

Bryan™️

commit sha ee3297140c74f2b2e4a348c1feb5fa82ecffb0ce

more robust cw agent installation

view details

push time in 2 days

Pull request review commentawslabs/amazon-ec2-instance-qualifier

integrate cloudwatch agent

+{

discussed offline:

  • by offering this flexibility we put onus on users to provide valid/existing metrics
  • pass in metrics as another JSON object in config
  • users can either provide metrics they want XOR use the default metrics in this file
brycahta

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentawslabs/amazon-ec2-instance-qualifier

integrate cloudwatch agent

+{+	"agent": {+		"metrics_collection_interval": 30,+		"run_as_user": "root"+	},+	"metrics": {+		"append_dimensions": {+			"AutoScalingGroupName": "${aws:AutoScalingGroupName}",+			"InstanceId": "${aws:InstanceId}",+			"InstanceType": "${aws:InstanceType}"

Agreed, but from the docs:

append_dimensions – Optional. Adds Amazon EC2 metric dimensions to all metrics collected by the agent. The only supported key-value pairs are shown in the following list. Any other key-value pairs are ignored.

"ImageID":"${aws:ImageId}" sets the instance's AMI ID as the value of the ImageID dimension.

"InstanceId":"${aws:InstanceId}" sets the instance's instance ID as the value of the InstanceID dimension.

"InstanceType":"${aws:InstanceType}" sets the instance's instance type as the value of the InstanceType dimension.

"AutoScalingGroupName":"${aws:AutoScalingGroupName}" sets the instance's Auto Scaling group name as the value of the AutoScalingGroupName dimension.

brycahta

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentawslabs/amazon-ec2-instance-qualifier

integrate cloudwatch agent

+{

Yeah, I think that's a good user experience, but the main concern here is communicating to IQ which metrics to benchmark (i.e. use as target-utilization) which is one reason why I wanted the metrics to be known/valid when IQ starts up.

With that said, it's also possible to create another arg/config in JSON that encapsulates metrics and it could be a string of custom-named metrics that the user knows should be emitted by CWA, but the downside being more opportunity for failure (fat finger a name, forgot to configure it, etc.).

Who knew finding a balance between flexibility, UX, and functionality would be difficult

brycahta

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentawslabs/amazon-ec2-instance-qualifier

integrate cloudwatch agent

 for file in *; do 		chmod u+x "$file" 	fi done++wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm

good catch! reading docs is hard

brycahta

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentawslabs/amazon-ec2-instance-qualifier

integrate cloudwatch agent

 for file in *; do 		chmod u+x "$file" 	fi done++wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm

This section is more or less hard-coded; wanted to get this working and get some input/feedback before making it more robust. The following can be improved:

  • use regional download link
  • the architecture in the download link also supports arm64-- so need to introduce logic to change this value based on $ARCHITECTURE
    • no arm testing right now; should add that too
  • config file cwagent-config.json could be dynamic, but I think I'll let IQ enforce the name here
brycahta

comment created time in 3 days

Pull request review commentawslabs/amazon-ec2-instance-qualifier

integrate cloudwatch agent

+{

So this file should drive all the metrics being uploaded to CW via the agent, so any custom metrics or metrics captured via StatsD or collectd would also be defined here.

Looking for ideas on how to make this dynamic/configurable by the user:

  • have users provide their own via another CLI arg
  • interactive mode within IQ that'll basically invoke the CW Agent Configuration Wizard
  • leave this file as-is (collecting Basic metrics) and introduce a different mechanism for subscribing to other metrics
brycahta

comment created time in 3 days

PullRequestReviewEvent
PullRequestReviewEvent

PR opened awslabs/amazon-ec2-instance-qualifier

integrate cloudwatch agent

Issue #, if available:

  • These changes make emitting metrics via CW Agent possible

Description of changes:

  • Wanted to break up changes into multiple PRs, so this first one is basically just enabling the agent, but the real benchmark data/comparisons is still coming from Instance-Qualifier scripts/agent
  • Added cloud-watch-agent-config.json configured to capture Basic metrics every 30 secs
    • I'm treating this JSON as another 'script' in the TestSuite because that was the easiest ingress point to get the file onto the instance-- REFACTOR WILL BE SOON^tm
  • Added CloudWatch Agent installation to user data
  • Added CloudWatchAgentServerPolicy to IAM Role
  • Updated tests

Testing:

  • make clean && make test locally

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

+138 -22

0 comment

10 changed files

pr created time in 3 days

create barnchbrycahta/amazon-ec2-instance-qualifier

branch : cloudwatch-integr8

created branch time in 3 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentaws/aws-node-termination-handler

Add AEMM mock interruption documentation

+# AWS Node Termination Handler & Amazon EC2 Metadata Mock++We have open sourced a tool called the [amazon-ec2-metadata-mock](https://github.com/aws/amazon-ec2-metadata-mock) (AEMM)+that simulates spot interruption notices and more by starting a real webserver that serves data similar to EC2 Instance+Metadata Service. The tool is easily deployed to kubernetes with a Helm chart.++Below is a short guide on how to set AEMM up with your Node Termination Handler cluster in case you'd like to verify the+behavior yourself.++## Triggering AWS Node Termination Handler with Amazon EC2 Metadata Mock++Start by installing AEMM on your cluster. For full and up to date installation instructions reference the AEMM repository.+Here's just one way to do it.++Download the latest tar ball from the releases page, at the time of writing this that was v1.6.0. Then install it using+Helm:+```+helm install amazon-ec2-metadata-mock amazon-ec2-metadata-mock-1.6.0.tgz \+  --namespace default+```++Once AEMM is installed, you need to change the instance metadata url of Node Termination Handler to point+to the location AEMM is serving from. If you use the default values of AEMM, the installation will look similar to this:+```+helm upgrade --install aws-node-termination-handler \+  --namespace kube-system \+  --set instanceMetadataURL="http://amazon-ec2-metadata-mock-service.default.svc.cluster.local:1338" \+  eks/aws-node-termination-handler+```++That's it! Instead of polling the real IMDS service endpoint, AWS Node Termination Handler will instead poll AEMM.+If you open the logs of an AWS Node Termination Handler pod you should see that it receives (mock) interruption+events from AEMM and that the nodes are cordoned and drained. Keep in mind that these nodes won't actually get terminated,+so you might need to manually uncordon the nodes if you want to reset your test cluster.++### AEMM Advanced Configuration+If you run the example above you might notice that the logs are heavily populated. Here's an example output:+```+2020/09/15 21:13:41 Sending interruption event to the interruption channel+2020/09/15 21:13:41 Got interruption event from channel {InstanceID:i-1234567890abcdef0 InstanceType:m4.xlarge PublicHostname:ec2-192-0-2-54.compute-1.amazonaws.com PublicIP:192.0.2.54 LocalHostname:ip-172-16-34-43.ec2.internal LocalIP:172.16.34.43 AvailabilityZone:us-east-1a} {EventID:spot-itn-47ddfb5e39791606bec3e91fea4cdfa86f86a60ddaf014c8b4af8e008f134b19 Kind:SPOT_ITN Description:Spot ITN received. Instance will be interrupted at 2020-09-15T21:15:41Z+ State: NodeName:ip-192-168-123-456.us-east-1.compute.internal StartTime:2020-09-15 21:15:41 +0000 UTC EndTime:0001-01-01 00:00:00 +0000 UTC Drained:false PreDrainTask:0x113c8a0 PostDrainTask:<nil>}+WARNING: ignoring DaemonSet-managed Pods: default/amazon-ec2-metadata-mock-pszj2, kube-system/aws-node-bl2bj, kube-system/aws-node-termination-handler-2pvjr, kube-system/kube-proxy-fct9f+evicting pod "coredns-67bfd975c5-rgkh7"+evicting pod "coredns-67bfd975c5-6g88n"+2020/09/15 21:13:42 Node "ip-192-168-123-456.us-east-1.compute.internal" successfully cordoned and drained.+2020/09/15 21:13:43 Sending interruption event to the interruption channel+2020/09/15 21:13:43 Got interruption event from channel {InstanceID:i-1234567890abcdef0 InstanceType:m4.xlarge PublicHostname:ec2-192-0-2-54.compute-1.amazonaws.com PublicIP:192.0.2.54 LocalHostname:ip-172-16-34-43.ec2.internal LocalIP:172.16.34.43 AvailabilityZone:us-east-1a} {EventID:spot-itn-97be476b6246aba6401ba36e54437719bfdf987773e9c83fe30336eb7fea9704 Kind:SPOT_ITN Description:Spot ITN received. Instance will be interrupted at 2020-09-15T21:15:43Z+ State: NodeName:ip-192-168-123-456.us-east-1.compute.internal StartTime:2020-09-15 21:15:43 +0000 UTC EndTime:0001-01-01 00:00:00 +0000 UTC Drained:false PreDrainTask:0x113c8a0 PostDrainTask:<nil>}+WARNING: ignoring DaemonSet-managed Pods: default/amazon-ec2-metadata-mock-pszj2, kube-system/aws-node-bl2bj, kube-system/aws-node-termination-handler-2pvjr, kube-system/kube-proxy-fct9f+2020/09/15 21:13:44 Node "ip-192-168-123-456.us-east-1.compute.internal" successfully cordoned and drained.+2020/09/15 21:13:45 Sending interruption event to the interruption channel+2020/09/15 21:13:45 Got interruption event from channel...+```++This isn't a mistake, by default AEMM will respond to any request for metadata with a spot interruption occurring 2 minutes+later than the request time.\* AWS Node Termination Handler polls for events every 2 seconds by default, so the effect is+that new interruption events are found and processed every 2 seconds. ++In reality there will only be a single interruption event, and you can mock this by setting the `spot.time` parameter of+AEMM when installing it. +```+helm install amazon-ec2-metadata-mock amazon-ec2-metadata-mock-1.6.0.tgz \+  --set aemm.spot.time="2020-09-09T22:40:47Z" \+  --namespace default+```++Now when you check the logs you should only see a single event get processed. ++For more ways of configuring AEMM check out the [Helm configuration page](https://github.com/aws/amazon-ec2-metadata-mock/tree/master/helm/amazon-ec2-metadata-mock).++## Node Termination Handler E2E Tests++AEMM started out as a test server for aws-node-termination-handler's end-to-end tests in this repo. We use AEMM throughout+our end to end tests to create interruption notices.++The e2e tests install aws-node-termination-handler using Helm and set the metadata url [here](https://github.com/aws/aws-node-termination-handler/blob/master/test/e2e/spot-interruption-test#L36).+This becomes where aws-node-termination-handler looks for metadata; other applications on the node still look at the real+EC2 metadata service.++We set the metadata url environment variable [here](https://github.com/aws/aws-node-termination-handler/blob/master/test/k8s-local-cluster-test/run-test#L18)+for the local tests that use a kind cluster, and [here](https://github.com/aws/aws-node-termination-handler/blob/master/test/eks-cluster-test/run-test#L117)+for the eks-cluster e2e tests.++Check out the [ReadMe](https://github.com/aws/aws-node-termination-handler/tree/master/test) in our test folder for more+info on the e2e tests. ++---++\* Only the first two unique IPs to request data from AEMM receive spot itn information in the response. This was introduced+in AEMM v1.6.0. This can be overridden with a configuration paramter. For previous versions there is no unique IP restriction. 

was about to ask how spellcheck didn't catch it

haugenj

comment created time in 4 days

Pull request review commentaws/aws-node-termination-handler

Add AEMM mock interruption documentation

+# AWS Node Termination Handler & Amazon EC2 Metadata Mock++We have open sourced a tool called the [amazon-ec2-metadata-mock](https://github.com/aws/amazon-ec2-metadata-mock) (AEMM)+that simulates spot interruption notices and more by starting a real webserver that serves data similar to EC2 Instance+Metadata Service. The tool is easily deployed to kubernetes with a Helm chart.++Below is a short guide on how to set AEMM up with your Node Termination Handler cluster in case you'd like to verify the+behavior yourself.++## Triggering AWS Node Termination Handler with Amazon EC2 Metadata Mock++Start by installing AEMM on your cluster. For full and up to date installation instructions reference the AEMM repository.+Here's just one way to do it.++Download the latest tar ball from the releases page, at the time of writing this that was v1.6.0. Then install it using+Helm:+```+helm install amazon-ec2-metadata-mock amazon-ec2-metadata-mock-1.6.0.tgz \+  --namespace default+```++Once AEMM is installed, you need to change the instance metadata url of Node Termination Handler to point+to the location AEMM is serving from. If you use the default values of AEMM, the installation will look similar to this:+```+helm upgrade --install aws-node-termination-handler \+  --namespace kube-system \+  --set instanceMetadataURL="http://amazon-ec2-metadata-mock-service.default.svc.cluster.local:1338" \+  eks/aws-node-termination-handler+```++That's it! Instead of polling the real IMDS service endpoint, AWS Node Termination Handler will instead poll AEMM.+If you open the logs of an AWS Node Termination Handler pod you should see that it receives (mock) interruption+events from AEMM and that the nodes are cordoned and drained. Keep in mind that these nodes won't actually get terminated,+so you might need to manually uncordon the nodes if you want to reset your test cluster.++### AEMM Advanced Configuration+If you run the example above you might notice that the logs are heavily populated. Here's an example output:+```+2020/09/15 21:13:41 Sending interruption event to the interruption channel+2020/09/15 21:13:41 Got interruption event from channel {InstanceID:i-1234567890abcdef0 InstanceType:m4.xlarge PublicHostname:ec2-192-0-2-54.compute-1.amazonaws.com PublicIP:192.0.2.54 LocalHostname:ip-172-16-34-43.ec2.internal LocalIP:172.16.34.43 AvailabilityZone:us-east-1a} {EventID:spot-itn-47ddfb5e39791606bec3e91fea4cdfa86f86a60ddaf014c8b4af8e008f134b19 Kind:SPOT_ITN Description:Spot ITN received. Instance will be interrupted at 2020-09-15T21:15:41Z+ State: NodeName:ip-192-168-123-456.us-east-1.compute.internal StartTime:2020-09-15 21:15:41 +0000 UTC EndTime:0001-01-01 00:00:00 +0000 UTC Drained:false PreDrainTask:0x113c8a0 PostDrainTask:<nil>}+WARNING: ignoring DaemonSet-managed Pods: default/amazon-ec2-metadata-mock-pszj2, kube-system/aws-node-bl2bj, kube-system/aws-node-termination-handler-2pvjr, kube-system/kube-proxy-fct9f+evicting pod "coredns-67bfd975c5-rgkh7"+evicting pod "coredns-67bfd975c5-6g88n"+2020/09/15 21:13:42 Node "ip-192-168-123-456.us-east-1.compute.internal" successfully cordoned and drained.+2020/09/15 21:13:43 Sending interruption event to the interruption channel+2020/09/15 21:13:43 Got interruption event from channel {InstanceID:i-1234567890abcdef0 InstanceType:m4.xlarge PublicHostname:ec2-192-0-2-54.compute-1.amazonaws.com PublicIP:192.0.2.54 LocalHostname:ip-172-16-34-43.ec2.internal LocalIP:172.16.34.43 AvailabilityZone:us-east-1a} {EventID:spot-itn-97be476b6246aba6401ba36e54437719bfdf987773e9c83fe30336eb7fea9704 Kind:SPOT_ITN Description:Spot ITN received. Instance will be interrupted at 2020-09-15T21:15:43Z+ State: NodeName:ip-192-168-123-456.us-east-1.compute.internal StartTime:2020-09-15 21:15:43 +0000 UTC EndTime:0001-01-01 00:00:00 +0000 UTC Drained:false PreDrainTask:0x113c8a0 PostDrainTask:<nil>}+WARNING: ignoring DaemonSet-managed Pods: default/amazon-ec2-metadata-mock-pszj2, kube-system/aws-node-bl2bj, kube-system/aws-node-termination-handler-2pvjr, kube-system/kube-proxy-fct9f+2020/09/15 21:13:44 Node "ip-192-168-123-456.us-east-1.compute.internal" successfully cordoned and drained.+2020/09/15 21:13:45 Sending interruption event to the interruption channel+2020/09/15 21:13:45 Got interruption event from channel...+```++This isn't a mistake, by default AEMM will respond to any request for metadata with a spot interruption occurring 2 minutes+later than the request time.\* AWS Node Termination Handler polls for events every 2 seconds by default, so the effect is+that new interruption events are found and processed every 2 seconds. ++In reality there will only be a single interruption event, and you can mock this by setting the `spot.time` parameter of+AEMM when installing it. +```+helm install amazon-ec2-metadata-mock amazon-ec2-metadata-mock-1.6.0.tgz \+  --set aemm.spot.time="2020-09-09T22:40:47Z" \+  --namespace default+```++Now when you check the logs you should only see a single event get processed. ++For more ways of configuring AEMM check out the [Helm configuration page](https://github.com/aws/amazon-ec2-metadata-mock/tree/master/helm/amazon-ec2-metadata-mock).++## Node Termination Handler E2E Tests++AEMM started out as a test server for aws-node-termination-handler's end-to-end tests in this repo. We use AEMM throughout+our end to end tests to create interruption notices.++The e2e tests install aws-node-termination-handler using Helm and set the metadata url [here](https://github.com/aws/aws-node-termination-handler/blob/master/test/e2e/spot-interruption-test#L36).+This becomes where aws-node-termination-handler looks for metadata; other applications on the node still look at the real+EC2 metadata service.++We set the metadata url environment variable [here](https://github.com/aws/aws-node-termination-handler/blob/master/test/k8s-local-cluster-test/run-test#L18)+for the local tests that use a kind cluster, and [here](https://github.com/aws/aws-node-termination-handler/blob/master/test/eks-cluster-test/run-test#L117)+for the eks-cluster e2e tests.++Check out the [ReadMe](https://github.com/aws/aws-node-termination-handler/tree/master/test) in our test folder for more+info on the e2e tests. ++---++\* Only the first two unique IPs to request data from AEMM receive spot itn information in the response. This was introduced+in AEMM v1.6.0. This can be overridden with a configuration paramter. For previous versions there is no unique IP restriction. 

paramter --> parameter

haugenj

comment created time in 4 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentaws/aws-node-termination-handler

Reformat logs to support proper JSON logging

 func addTaint(node *corev1.Node, nth Node, taintKey string, taintValue string, e 		}  		if err != nil {-			log.Log().Msgf("Error while adding %v taint on node %v: %v", taintKey, node.Name, err)+			log.Log().

Got it, I just meant comparing what you currently have in code now vs. the same line but wrapping the message with Err()

haugenj

comment created time in 5 days

PullRequestReviewEvent
PullRequestReviewEvent

pull request commentaws/aws-node-termination-handler

Reformat logs to support proper JSON logging

I'm learning toward making all the keys we control snake_case.

+ 1

haugenj

comment created time in 5 days

Pull request review commentaws/aws-node-termination-handler

Reformat logs to support proper JSON logging

 func removeTaint(node *corev1.Node, client kubernetes.Interface, taintKey string 		}  		if err != nil {-			log.Log().Msgf("Error while releasing %v taint on node %v: %v", taintKey, node.Name, err)+			log.Log().+				Str("TaintKey", taintKey).+				Str("NodeName", node.Name).+				Msg("Error while releasing taint on node")

I like this format in code, makes it easier to read/understand 👍🏼

haugenj

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentaws/aws-node-termination-handler

Reformat logs to support proper JSON logging

 func addTaint(node *corev1.Node, nth Node, taintKey string, taintValue string, e 		}  		if err != nil {-			log.Log().Msgf("Error while adding %v taint on node %v: %v", taintKey, node.Name, err)+			log.Log().

out of curiosity, what does this log look like compared to the one you have posted? I'm assuming you didn't wrap in Err for formatting reasons

log.Log().Err("Error while adding taint on node").Str("TaintKey", taintKey). ...

haugenj

comment created time in 5 days

PullRequestReviewEvent

push eventbrycahta/amazon-ec2-instance-qualifier

Bryan™

commit sha 0913e07ab997cb812a74963e68a7c793c20eb3af

pass args via config file (#8) * pass userConfig as JSON * update e2e & docs * fixing config precedence

view details

push time in 5 days

delete branch brycahta/amazon-ec2-instance-qualifier

delete branch : config-file

delete time in 5 days

push eventawslabs/amazon-ec2-instance-qualifier

Bryan™

commit sha 0913e07ab997cb812a74963e68a7c793c20eb3af

pass args via config file (#8) * pass userConfig as JSON * update e2e & docs * fixing config precedence

view details

push time in 5 days

PR merged awslabs/amazon-ec2-instance-qualifier

pass args via config file

Issue #, if available: N/A

Description of changes:

  • args can be passed as JSON via --config-file flag
    • config file will take precedence over args provided via CLI or as Env Vars
  • Updated userConfig accessibility from private to public to leverage JSON marshal + unmarshal
  • Updated tests and documentation
  • Fix bucket creation as e2e tests sometimes stall and fail due to 'bucket already created' error

Testing:

  • make clean && make test locally

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

+264 -278

1 comment

10 changed files

brycahta

pr closed time in 5 days

Pull request review commentawslabs/amazon-ec2-instance-qualifier

pass args via config file

 func (t TestFixture) CfnTemplateFilename() string { 	return t.cfnTemplateFilename } -// AmiId returns amiId.+// AmiId returns AmiId. func (t TestFixture) AmiId() string { 	return t.amiId }++// SetUserConfig sets empty fields of UserConfig to reqConfig+func (UserConfig) SetUserConfig(reqConfig UserConfig) {+	if userConfig.InstanceTypes == "" {

Had a couple more if checks in here at first such as:

if userConfig.InstanceTypes == "" && reqConfig.InstanceTypes != "" , but go-report-card (mainly gocyclo) complained the whole function was too complex so had to limit to 15 if statements.

I think the right long-term move would be using Viper to handle the precedence for us, but I think this works for now

brycahta

comment created time in 6 days

PullRequestReviewEvent

push eventbrycahta/amazon-ec2-instance-qualifier

Bryan™️

commit sha 52ee8ad5b4ddee212c90a4c09121e3fb06034d88

fixing config precedence

view details

push time in 6 days

PR opened awslabs/amazon-ec2-instance-qualifier

pass args via config file

Issue #, if available: N/A

Description of changes:

  • args can be passed as JSON via --config-file flag
    • config file will take precedence over args provided via CLI or as Env Vars
  • Updated userConfig accessibility from private to public to leverage JSON marshal + unmarshal
  • Updated tests and documentation
  • Fix bucket creation as e2e tests sometimes stall and fail due to 'bucket already created' error

Testing:

  • make clean && make test locally

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

+211 -274

0 comment

10 changed files

pr created time in 6 days

push eventbrycahta/amazon-ec2-instance-qualifier

Bryan™️

commit sha aa7b1b0aa56c9eab8b7a8b6e6ca9623deb09b5f2

update e2e & docs

view details

push time in 6 days

create barnchbrycahta/amazon-ec2-instance-qualifier

branch : config-file

created branch time in 6 days

push eventbrycahta/amazon-ec2-instance-qualifier

Bryan™

commit sha 017f4cf09a6383a92ab061c0dcfdec0cd63e4ea0

update session-creation logic (#6) * update session-creation logic * update region selection * readme updates * go mod tidy * update licenses and attributions

view details

push time in 6 days

fork brycahta/amazon-ec2-instance-selector

A CLI tool and go library which recommends instance types based on resource criteria like vcpus and memory

fork in 9 days

delete branch brycahta/amazon-ec2-instance-qualifier

delete branch : session-refactor

delete time in 9 days

push eventawslabs/amazon-ec2-instance-qualifier

Bryan™

commit sha 017f4cf09a6383a92ab061c0dcfdec0cd63e4ea0

update session-creation logic (#6) * update session-creation logic * update region selection * readme updates * go mod tidy * update licenses and attributions

view details

push time in 9 days

PR merged awslabs/amazon-ec2-instance-qualifier

update session-creation logic

Issue #, if available: N/A

Description of changes:

  • Update aws session creation logic to align with Instance-Selector
  • Pass region from CLI to agent
  • If users do not provide a region, then do not assign in userConfig

Testing:

  • make clean && make test locally

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

+282 -69

0 comment

15 changed files

brycahta

pr closed time in 9 days

push eventbrycahta/amazon-ec2-instance-qualifier

Bryan™️

commit sha 1a3edbaed9835cf378886954178d6101287fc5dc

go mod tidy

view details

Bryan™️

commit sha 6c27ef295037d784929b85811c3be01c56e57abb

update licenses and attributions

view details

push time in 9 days

Pull request review commentawslabs/amazon-ec2-instance-qualifier

update session-creation logic

 go 1.14  require ( 	github.com/aws/aws-sdk-go v1.31.5+	github.com/mitchellh/go-homedir v1.1.0

good catch!

brycahta

comment created time in 9 days

PullRequestReviewEvent

Pull request review commentawslabs/amazon-ec2-instance-qualifier

update session-creation logic

 func TestPopulateTestFixtureForResumedRun(t *testing.T) {  func TestParseCliArgsEnvSuccess(t *testing.T) { 	resetFlagsForTest()-	os.Setenv("AWS_DEFAULT_REGION", "AWS_DEFAULT_REGION")+	os.Setenv("AWS_REGION", "us-weast-1")

not a typo

brycahta

comment created time in 10 days

PullRequestReviewEvent

push eventbrycahta/amazon-ec2-instance-qualifier

Bryan™

commit sha 967944ecd0824c6297ae3e302c5d8cc8718a6696

allow users to provide script in userData (#7)

view details

Bryan™️

commit sha 7353c1324dc99493e1b35ad5a0ce73e41a21816f

update session-creation logic

view details

Bryan™️

commit sha 90103be145d66cf85c891b821d36d775ad39aa68

update region selection

view details

Bryan™️

commit sha cd44a4223dd1b59fdb236fbdff8517231a7f4fed

readme updates

view details

push time in 10 days

push eventbrycahta/amazon-ec2-instance-qualifier

Bryan™

commit sha 967944ecd0824c6297ae3e302c5d8cc8718a6696

allow users to provide script in userData (#7)

view details

push time in 10 days

delete branch brycahta/amazon-ec2-instance-qualifier

delete branch : user-bash

delete time in 10 days

push eventawslabs/amazon-ec2-instance-qualifier

Bryan™

commit sha 967944ecd0824c6297ae3e302c5d8cc8718a6696

allow users to provide script in userData (#7)

view details

push time in 10 days

PR merged awslabs/amazon-ec2-instance-qualifier

allow users to provide script in userData

Issue #, if available: N/A

Description of changes:

  • add --custom-scripts flag
    • users provide path to a script which will execute before agent and before setup

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

+70 -39

0 comment

13 changed files

brycahta

pr closed time in 10 days

PR opened awslabs/amazon-ec2-instance-qualifier

allow users to provide script in userData

Issue #, if available: N/A

Description of changes:

  • add --custom-scripts flag
    • users provide path to a script which will execute before agent and before setup

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

+70 -39

0 comment

13 changed files

pr created time in 10 days

create barnchbrycahta/amazon-ec2-instance-qualifier

branch : user-bash

created branch time in 10 days

PR opened awslabs/amazon-ec2-instance-qualifier

update session-creation logic

Issue #, if available: N/A

Description of changes:

  • Update aws session creation logic to align with Instance-Selector
  • Pass region from CLI to agent
  • If users do not provide a region, then do not assign in userConfig

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

+122 -54

0 comment

11 changed files

pr created time in 11 days

create barnchbrycahta/amazon-ec2-instance-qualifier

branch : session-refactor

created branch time in 11 days

delete branch brycahta/aws-node-termination-handler

delete branch : aemm-upgrade

delete time in 12 days

push eventaws/aws-node-termination-handler

Bryan™

commit sha ac6f0e79c3493c7ac621dc1aac6e7f642f9b4af7

upgrade aemm to 1.6 (#251)

view details

push time in 12 days

PR merged aws/aws-node-termination-handler

upgrade aemm to 1.6

Issue #, if available: N/A

Description of changes:

  • upgrade aemm to 1.6

Testing:

  • make e2e-test locally

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

+5 -5

1 comment

3 changed files

brycahta

pr closed time in 12 days

PR opened aws/aws-node-termination-handler

upgrade aemm to 1.6

Issue #, if available: N/A

Description of changes:

  • upgrade aemm to 1.6

Testing:

  • make e2e-test locally

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

+5 -5

0 comment

3 changed files

pr created time in 12 days

create barnchbrycahta/aws-node-termination-handler

branch : aemm-upgrade

created branch time in 12 days

push eventbrycahta/aws-node-termination-handler

Bryan™

commit sha e68df3460c92a4db705c37319108989585b07e62

add readme for e2e tests (#218)

view details

Brandon Wagner

commit sha bad975d59e330b8af966b087f23cbf8c5692ecbb

only deploy to worker node on e2e tests (#217) * only deploy to worker node on e2e tests * fix webhook tests without using host networking * fix webhook port consistency * print gomod download logs

view details

Brandon Wagner

commit sha 91b481abceb22816019b3953ec970ae16f31877a

fix webhook test stability (#219)

view details

Jerad C

commit sha a91b0e788596ceb43b8a1893573ae9c899250fdb

add missing helm chart properties (#220) (#222)

view details

Thomas O'Neill

commit sha 95fdc025df8f92b725f988869854917d961cc14b

Fix Helm chart comment on enableSpotInterruptionDraining default behavior (#221) * default enableSpotInterruptionDraining to true * enableSpotInterruptionDraining default to "true" * Update README for helm chart, and roll back value change Co-authored-by: Thomas O'Neill <toneill@new-innov.com>

view details

Prathibha Datta Kumar

commit sha 1943f8bce4f9a1ef73a52c8973712aac1e1c3462

fix missing new line for updateStrategy (#227)

view details

Prathibha Datta Kumar

commit sha 292e4af3844abae869fb95bcd8f37255dfbf5286

updating default updateStrategy and bumping version for helm chart release (#228)

view details

Bryan™

commit sha ca45b8f8274c25be8c5180e51efdea4b52e7e847

fix manifest updating in push-docker-images (#231)

view details

Brandon Wagner

commit sha 3e355f41607f887a3ebc7e39d38137cfb193c6eb

Add Amazon EC2 Spot Instances Integration Roadmap to readme (#232)

view details

Brandon Wagner

commit sha e48c1f8b965f7e608688fb59a555f87203bd1326

upgrade to go 1.15 (#233) * upgrade to go 1.15 * remove duplicate linker flags * remove -s linker flag from windows

view details

Paulo Martins

commit sha c685bbaf7a614305bed5e33f4aed867e15a041b3

fix identation on PodMonitor (#235)

view details

Brandon Wagner

commit sha 021b71f48a2500e54def0d04dbd59046cc086bd0

Helm Template Gen Test (#236) * add helm test to check template generation * sync enableSpotInterruption usage statement

view details

Bryan™

commit sha 97f42ef9f6c656f84ccfda982be4da4fa3529e3b

Update .travis.yml go1.15 (#238)

view details

Brandon Wagner

commit sha 713db7c6b117e638500984cbe6ab26b74783b6a2

up helm chart to use v1.7.0 image (#234)

view details

Brandon Wagner

commit sha 3fa790071c86c17ad6fbb35910f982dea3072dcf

retry aemm installs (#237)

view details

Brandon Wagner

commit sha f92ed02c5a6eb7681582b8858574c9a5853d9e90

fix release script upload to github (#240)

view details

Brandon Wagner

commit sha ef0eec0e9f4cd69f873cf1fc21744ea20e799552

only build one binary to check licenses (#239)

view details

Brandon Wagner

commit sha e5705ffd6fed89b277990bf349327f09eaadc790

clear up shellcheck warnings (#242) * clear up shellcheck warnings * changes based on PR feedback * missed some spots

view details

Brandon Wagner

commit sha 24ff89c46173c56e14ed37eb26b9525bb9194e6c

fix helm test (#243)

view details

David Pait

commit sha 92fb0c2fab2dc43c335766e3288b9b1604d198b4

Add retries when reponse from IMDSv2 retruns a 401 (#244) * Add retries when reponse from IMDSv2 retruns a 401 * Revert "Add retries when reponse from IMDSv2 retruns a 401" This reverts commit c1e34774098384db6b33c0774ec7dcf7a5c9bce7. * move IMDSv2 401 retries to Request function. move Panic to main function. add tests for 401 retries. * add missing equals to duplicateErrorThershold check in main function * fix spelling of NTH * fix report card

view details

push time in 12 days

push eventbrycahta/amazon-ec2-metadata-mock

Bryan™

commit sha c93b63d4c668acecb11e956216ea491023740862

🥑🤖 v1.6.0 release prep [Skip Helm E2E Tests] 🤖🥑 (#96)

view details

push time in 12 days

created tagaws/amazon-ec2-metadata-mock

tagv1.6.0

A tool to simulate Amazon EC2 instance metadata

created time in 12 days

release aws/amazon-ec2-metadata-mock

v1.6.0

released time in 12 days

delete branch brycahta/amazon-ec2-metadata-mock

delete branch : pr/v1.6.0-release

delete time in 12 days

push eventaws/amazon-ec2-metadata-mock

Bryan™

commit sha c93b63d4c668acecb11e956216ea491023740862

🥑🤖 v1.6.0 release prep [Skip Helm E2E Tests] 🤖🥑 (#96)

view details

push time in 12 days

PR merged aws/amazon-ec2-metadata-mock

🥑🤖 v1.6.0 release prep release-prep 🤖 auto-generated🤖

🥑🤖 Auto-generated PR for v1.6.0 release. Updating release versions in repo.

+10 -10

0 comment

4 changed files

brycahta

pr closed time in 12 days

PR opened aws/amazon-ec2-metadata-mock

🥑🤖 v1.6.0 release prep

🥑🤖 Auto-generated PR for v1.6.0 release. Updating release versions in repo.

+10 -10

0 comment

4 changed files

pr created time in 12 days

create barnchbrycahta/amazon-ec2-metadata-mock

branch : pr/v1.6.0-release

created branch time in 12 days

delete branch brycahta/amazon-ec2-metadata-mock

delete branch : update-paths

delete time in 12 days

push eventbrycahta/amazon-ec2-metadata-mock

Bryan™

commit sha 715555a712a48820c9d83c3a210486cc04a4ae2c

Update README.md (#93)

view details

Bryan™

commit sha 3ecba41c7c57661bd8cee442cface2edf9fc2948

adding placement paths (#95)

view details

Bryan™

commit sha 38f2e3befb1e3fc00a1370f8ec7c314fce5b99e5

Adding mock-ip-count feature (#94) * convert from daemonset to deployment * adding termination-nodes feature * update travis and documentation * rename to mock-ip-count * address pr feedback * add feature to scheduled events for parity * add e2e tests for cmds * fix ReadMe description

view details

push time in 12 days

delete branch brycahta/amazon-ec2-metadata-mock

delete branch : helm_deployment

delete time in 12 days

push eventaws/amazon-ec2-metadata-mock

Bryan™

commit sha 38f2e3befb1e3fc00a1370f8ec7c314fce5b99e5

Adding mock-ip-count feature (#94) * convert from daemonset to deployment * adding termination-nodes feature * update travis and documentation * rename to mock-ip-count * address pr feedback * add feature to scheduled events for parity * add e2e tests for cmds * fix ReadMe description

view details

push time in 12 days

PR merged aws/amazon-ec2-metadata-mock

Adding mock-ip-count feature

Issue #, if available:

  • AEMM as a daemonset wasn't appropriate since it's exposed as a k8s service
  • Prior to these changes, there was no way to limit the amount of nodes that could get a Spot Interrupt; therefore, if someone had a cluster of 5 nodes and AEMM pods running on each one and sent a request to spot path, then all nodes in cluster could be affected

Description of changes:

  • Change AEMM from a daemonset to a deployment w/replicas defaulted to 1
  • Add --mock-ip-count=<int> flag to set the number of nodes that will get a valid Spot INT-- all other nodes will get a 404
  • Add an e2e cluster test
  • Updated Makefile, Travis, documentation

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

+381 -46

1 comment

20 changed files

brycahta

pr closed time in 12 days

push eventaws/amazon-ec2-metadata-mock

Bryan™

commit sha 3ecba41c7c57661bd8cee442cface2edf9fc2948

adding placement paths (#95)

view details

push time in 12 days

PR merged aws/amazon-ec2-metadata-mock

adding placement paths

Issue #, if available: N/A

Description of changes:

Testing:

  • make test locally
  • queried newly added paths successfully

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

+62 -4

0 comment

8 changed files

brycahta

pr closed time in 12 days

push eventbrycahta/amazon-ec2-metadata-mock

Bryan™️

commit sha 71e856af010e67e492b1a674087f07578a447efb

fix ReadMe description

view details

push time in 12 days

Pull request review commentaws/amazon-ec2-metadata-mock

Adding mock-ip-count feature

 Parameter | Description | Default in Helm | Default AEMM configuration `aemm.server.hostname` | hostname to run AEMM on | `""`, in order to listen on all available interfaces e.g. ClusterIP | `0.0.0.0` `aemm.mockDelaySec` | mock delay in seconds, relative to the start time of AEMM | `0` | `0` `aemm.mockTriggerTime` | mock trigger time in RFC3339 format | `""` | `""`+`aemm.mockIPCount` | number of IPs that can receive spot interrupts and/or scheduled events; subsequent requests will return 404 | `""` | `99`

yup

brycahta

comment created time in 12 days

PullRequestReviewEvent

Pull request review commentaws/amazon-ec2-metadata-mock

Adding mock-ip-count feature

 var ( 		gf.ConfigFileFlag:       cfg.GetDefaultCfgFileName(), 		gf.MockDelayInSecFlag:   0, 		gf.MockTriggerTimeFlag:  "",+		gf.MockIPCountFlag:      99,

good catch!

brycahta

comment created time in 12 days

PullRequestReviewEvent

PR opened aws/amazon-ec2-metadata-mock

adding placement paths

Issue #, if available: N/A

Description of changes:

Testing:

  • make test locally
  • queried newly added paths successfully

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

+62 -4

0 comment

8 changed files

pr created time in 12 days

create barnchbrycahta/amazon-ec2-metadata-mock

branch : update-paths

created branch time in 12 days

push eventbrycahta/amazon-ec2-metadata-mock

Bryan™️

commit sha 27659b96a28d4afa482966ceb17ef240f4b59108

fix ReadMe description

view details

push time in 12 days

Pull request review commentaws/amazon-ec2-metadata-mock

Adding mock-ip-count feature

 Flags:   -t, --time string               time specifies the approximate time when the spot instance will receive the shutdown signal in RFC3339 format to execute instance action E.g. 2020-01-07T01:03:47Z (default: request time + 2 minutes in UTC)  Global Flags:-  -c, --config-file string    config file for cli input parameters in json format (default: $HOME/aemm-config.json)-  -n, --hostname string       the HTTP hostname for the mock url (default: localhost)-  -I, --imdsv2                whether to enable IMDSv2 only, requiring a session token when submitting requests (default: false, meaning both IMDS v1 and v2 are enabled)-  -d, --mock-delay-sec int    mock delay in seconds, relative to the application start time (default: 0 seconds)-  -p, --port string           the HTTP port where the mock runs (default: 1338)-  -s, --save-config-to-file   whether to save processed config from all input sources in .ec2-metadata-mock/.aemm-config-used.json in $HOME or working dir, if homedir is not found (default: false)+  -c, --config-file string         config file for cli input parameters in json format (default: $HOME/aemm-config.json)

need to fix this. i used root --help instead of spot --help

brycahta

comment created time in 12 days

PullRequestReviewEvent

push eventbrycahta/amazon-ec2-metadata-mock

Bryan™️

commit sha 49fcae03895717ac5842628af2c4862ca493b280

rename to mock-ip-count

view details

Bryan™️

commit sha 2f76530e99d80cc947ebf5666e858d9c38791bd2

address pr feedback

view details

Bryan™️

commit sha 94a2d845a08d89fa17a7dac33dbb55559638c143

add feature to scheduled events for parity

view details

Bryan™️

commit sha 68b9ef61eb73a9bf69c9906e92a89965c4aa349c

add e2e tests for cmds

view details

push time in 16 days

Pull request review commentaws/amazon-ec2-metadata-mock

Define number of nodes to receive Spot interrupts

 get_chart_test_config() {     echo "$config" } +test_termination_nodes() {+    if [[ $REUSE_ENV == false ]]; then+        mkdir -p $TMP_DIR+        install_kind+        create_kind_cluster+    fi++    build_and_load_image+    install_helm

:( yes

brycahta

comment created time in 16 days

PullRequestReviewEvent

Pull request review commentaws/amazon-ec2-metadata-mock

Define number of nodes to receive Spot interrupts

 var ( 		gf.ConfigFileFlag:       cfg.GetDefaultCfgFileName(), 		gf.MockDelayInSecFlag:   0, 		gf.MockTriggerTimeFlag:  "",+		gf.TerminationNodesFlag: 99,
  • +1 to the -1 recommendation
  • I'm good with defaulting to 2
    • do you think I should default this in the helm values.yaml file as well even though it'll be defaulted in app code? Not sure the etiquette on that
    • since this will technically break backwards compat should we major release it?
brycahta

comment created time in 16 days

PullRequestReviewEvent

Pull request review commentaws/amazon-ec2-metadata-mock

Define number of nodes to receive Spot interrupts

 Flags:       --mock-trigger-time string   mock trigger time in RFC3339 format. This takes priority over mock-delay-sec (default: none)   -p, --port string                the HTTP port where the mock runs (default: 1338)   -s, --save-config-to-file        whether to save processed config from all input sources in .ec2-metadata-mock/.aemm-config-used.json in $HOME or working dir, if homedir is not found (default: false)+  -x, --termination-nodes int      number of nodes in a cluster that can receive Spot interrupt notice (default: 99, meaning unlimited nodes) (default 99)

Yeah, it definitely should. Not that I'm hyped about --termination-nodes as a name, but could apply to both if i update the description to be non-spot specific.

Thoughts on keeping -x as shorthand?

brycahta

comment created time in 16 days

PullRequestReviewEvent

Pull request review commentaws/amazon-ec2-metadata-mock

Define number of nodes to receive Spot interrupts

 image:   tag: "v1.5.0"   pullPolicy: "IfNotPresent" +replicaCount: 1+ # nameOverride overrides the name of the helm chart nameOverride: "" # fullnameOverride overrides the name of the application fullnameOverride: "" -# Create node OS specific daemonset(s). (e.g. "linux", "windows", "linux windows")+# Create node OS specific deployment(s). (e.g. "linux", "windows", "linux windows") targetNodeOs: "linux"

win..what?

brycahta

comment created time in 16 days

push eventbrycahta/amazon-ec2-metadata-mock

Bryan™️

commit sha 5eb3eb22e26803c3ccce00888e02bb9be8e31329

update travis and documentation

view details

push time in 17 days

more