profile
viewpoint
Miles Wilson mileswilson @10xbanking London

mileswilson/feathers-rethinky 11

A FeathersJS Service Adaptor for thinkyJS.

mileswilson/repl.history 1

persist a node repl's history

mileswilson/aws-check 0

Contains fake AWS credentials to test security scanners

mileswilson/ecs-refarch-cloudformation 0

A reference architecture for deploying containerized microservices with Amazon ECS and AWS CloudFormation (YAML)

mileswilson/ergo 0

The management of multiple local apps running over different ports made easy

mileswilson/feathers-authentication 0

Feathers REST and Websocket authentication using JSON Web Tokens (JWT) with PassportJS.

mileswilson/feathers-rethinkdb 0

Create an RethinkDB service for FeathersJS.

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 0829e74aca3c2e502f4c4ee60b2bc0c11dc61a4f

Restoring to known good cluster to verify if cockroach is getting slower by filling up

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 5abae21c7affe97893271e11460519f683cb97a6

Finish previous day run

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha ba1dba817157dd2030c8b83a037287f42b814a98

Job runs really slow now. Increase to n1-standard-2

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 541bb0f3b016667e5819c7424cc16ebf9fa8d34c

Must have at least two workers; try 2 * n1-standard-1

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 168ed5b47b4fa012231a90f4368e450abe2427c0

Change pool to have only one small node.

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 72f9f0947d775875af6fd8859931d56408c93cf3

Change date for job

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 859deac7bef118c00f8e8e3c3767bafd1a65c813

Change date for job

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha ed1cc2647c2c4fe5042cec83f7da01d8e977a2b9

Change to 16 n1-standard-2 instances

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha f56f5d33ec4d7c260b3e5b85e3a65b1cf4f8b311

Set sparkPartitionOnSubScriptionKeyRanges option

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 0c85dc5a5f72e6ec5cfe6d0dd7f8460b0b196530

Still hitting quota error message. Try with 4 worker nodes

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 888fd4d9ada98a4bf7d1519e2698ccc5814089fd

Update DAG with new instance config + product key

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha f7f26a0a18a1392a39ebcff50b72eb0f68b5ccc5

Restore username

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 9c5d75e4393244131d2605e10e22cee002521af5

Connection works but airflow user is not allowed to select. See if we can use the ledgermanager user (probably need its certs tho)

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 4b3a0d1b4597d9325465295925c55c45f8fba5c5

We can't verify the hostname so going from verify-full to verify-ca until we can change the cockroach cert

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha f5602154212370be2964a6a1a1336f14486688f0

Use sslmode=verify-full just like we do on ledger

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 7e22982ba3dc01480980c2138527f8ea578f35e0

Change initialization_actions to list

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 545d074eafca27de78ffbdaede144c4a5e54e989

Configure node startup script and postgres URL

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 5658520e4944acfd63e20877430808131c0364d1

Disable pyspark job

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 8fd327cba4ff62bba8383bdf9a5a2219e21e0274

Update GCS URI

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 87305ea142820005f16c0c9ec9914361a7b4dc80

Update project

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 2665672d9b84e08770fb1e493cba7e8ba4ce7baf

Fix postgres driver name

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha fece2535a4162c3edfac85fc7b8a691356ef295e

Take out --sparkPartitionOnSubscriptionKeyRanges

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 3d35cd20b9275b33007d72e50161200d0ffb3a0f

Take out --master option. Might be an AWS thing?

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 40718b832470cb3fff87496b3e65f5ca7619e815

Add Spark job to DAG

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 0acb730c2ad05eb49bea5df673c3d5b4a84c56e3

Add Spark job

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha c6d1a9737cdc5af22d50d535d398b19e9df7eefd

Try to make request to ledger manager. Doesn't really prove anything because of external load balancers, though

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha a6337e289ddf533e5052ea256eb6357ac1ec13b6

Wrong type. Not missing python.

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 30a080e108ce4c1ba0fed683983d246138245096

Add allow-airflow network tag for new firewall rule

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha 31515a9b58fe230452f256c808dda827d8f74cb4

Python fix

view details

push time in 3 months

push event10x-gcp/airflow-dags

Bart Spaans

commit sha d48ad94999b07793bb576f11e3456858e3890a1e

Move subnetwork config into CLUSTER_CONFIG structure

view details

push time in 3 months

more