profile
viewpoint
Alex Lembiewski alembiewski Minsk, Belarus

alembiewski/auto-theme-switcher 0

Set of scripts to customize the appearance of a desktop

alembiewski/cake-builder 0

Cake Docker Builder

alembiewski/kudo-spark-operator 0

KUDO Spark Operator

alembiewski/operators 0

Collection of Kubernetes Operators built with KUDO.

alembiewski/spark-on-k8s-operator 0

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

push eventkudobuilder/kuttl

Alex Lembiewski

commit sha f51d23e0871b336fb5a84b783a8a810c6bf5d73b

Reports: write elapsed time in seconds (#157) Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

push time in 2 days

delete branch kudobuilder/kuttl

delete branch : alembiewski/fix-time-format-in-reports

delete time in 2 days

PR merged kudobuilder/kuttl

Reviewers
Reports: write elapsed time in seconds

Signed-off-by: Alex Lembiewski alembiyeuski.c@d2iq.com

<!-- Thanks for sending a pull request! Here are some tips for you:

  1. If this is your first time, please read our contributor guidelines: https://github.com/kudobuilder/kuttl/blob/main/CONTRIBUTING.md
  2. Make sure you have added and ran the tests before submitting your PR
  3. If the PR is unfinished, start it as a Draft PR: https://github.blog/2019-02-14-introducing-draft-pull-requests/ -->

What this PR does / why we need it:

This PR changes the output format for all time attributes to adhere to XSD schema.

The current format cannot be parsed properly by Teamcity (test durations are reported as < 1 ms):

Screen Shot 2020-07-07 at 09 41 40

<!-- *Automatically closes linked issue when PR is merged. Usage: Fixes #<issue number>, or Fixes (paste link of issue). -->

+3 -3

0 comment

1 changed file

alembiewski

pr closed time in 2 days

PR opened kudobuilder/kuttl

Reviewers
Reports: write elapsed time in seconds

Signed-off-by: Alex Lembiewski alembiyeuski.c@d2iq.com

<!-- Thanks for sending a pull request! Here are some tips for you:

  1. If this is your first time, please read our contributor guidelines: https://github.com/kudobuilder/kuttl/blob/main/CONTRIBUTING.md
  2. Make sure you have added and ran the tests before submitting your PR
  3. If the PR is unfinished, start it as a Draft PR: https://github.blog/2019-02-14-introducing-draft-pull-requests/ -->

What this PR does / why we need it:

This PR changes the output format for all time attributes to adhere to XSD specification.

The current format cannot be parsed properly by Teamcity (test durations are reported as < 1 ms):

Screen Shot 2020-07-07 at 09 41 40

<!-- *Automatically closes linked issue when PR is merged. Usage: Fixes #<issue number>, or Fixes (paste link of issue). -->

+3 -3

0 comment

1 changed file

pr created time in 3 days

create barnchkudobuilder/kuttl

branch : alembiewski/fix-time-format-in-reports

created branch time in 3 days

push eventmesosphere/kudo-spark-operator

Alex Lembiewski

commit sha f0834d860343f7e3ae79fe51e318f526c055414c

Automate release process with Github Actions (#107) * Add release workflow * Update the readme Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

push time in 4 days

delete branch mesosphere/kudo-spark-operator

delete branch : release-automation

delete time in 4 days

PR merged mesosphere/kudo-spark-operator

Automate release process with Github Actions

What changes were proposed in this pull request?

This PR adds release automation workflow based on GitHub Actions.

Why are the changes needed?

  • automate the release process
  • to create an operator package automatically and attach it to the release

How were the changes tested?

The workflow was developed and tested in a personal fork, results can be seen here:

  • Run: https://github.com/alembiewski/kudo-spark-operator/runs/830062644
  • Release example: https://github.com/alembiewski/kudo-spark-operator/releases
+67 -0

0 comment

2 changed files

alembiewski

pr closed time in 4 days

push eventmesosphere/kudo-spark-operator

Alex Lembiewski

commit sha bd28c3c5e06eed3f331e820fb70fa178dadb4046

Update the readme Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

Alex Lembiewski

commit sha c9c3bd91d11c19e6ce59c19d12075b337e82b975

Clean-up Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

push time in 8 days

PR opened mesosphere/kudo-spark-operator

Reviewers
Automate release process with Github Actions

What changes were proposed in this pull request?

This PR adds release automation workflow based on GitHub Actions.

Why are the changes needed?

  • automate the release process
  • to create an operator package automatically and attach it to the release

How were the changes tested?

The workflow was developed and tested in a personal fork, results can be seen here:

  • Run: https://github.com/alembiewski/kudo-spark-operator/runs/830062644
  • Release example: https://github.com/alembiewski/kudo-spark-operator/releases
+54 -0

0 comment

1 changed file

pr created time in 8 days

create barnchmesosphere/kudo-spark-operator

branch : release-automation

created branch time in 8 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.2

KUDO Spark Operator

created time in 8 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha a00573b0c3dc260f8113bab8b9d3c347d03c9966

Add release workflow

view details

push time in 8 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha c8d5a8f37832ac5351b6a41b558f7a17d82651e1

Update release.yml

view details

push time in 8 days

created tagalembiewski/kudo-spark-operator

tagv1.1.1

KUDO Spark Operator

created time in 8 days

create barnchalembiewski/kudo-spark-operator

branch : new-branch

created branch time in 8 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha be2b3632b91e4e91e46d7463cb69ee544376b4c5

Update release.yml

view details

push time in 8 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha cfae9777bc8c99355f5835bea89e3f90910c0835

Update release.yml

view details

push time in 8 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.10

KUDO Spark Operator

created time in 8 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha dd8d5c43f431d77261b6f5bcf188a0d8e7fd4189

Update release.yml

view details

push time in 8 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.9

KUDO Spark Operator

created time in 8 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 750d258fd5f0964fce0726f6d2a1d089683b67a3

Update release.yml

view details

push time in 8 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.8

KUDO Spark Operator

created time in 8 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha b1b76273771364d9505401c4f3d4ceef0e626011

Update release.yml

view details

push time in 8 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.7

KUDO Spark Operator

created time in 8 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 9685b203292e585e9aa59069524d2c32984a8dda

Update release.yml

view details

push time in 8 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.6

KUDO Spark Operator

created time in 8 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 40fcac4f0a47f86cd2d226ec7c44282fec130fa2

Update release.yml

view details

push time in 8 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.5

KUDO Spark Operator

created time in 8 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha f575a0000a0a2c69ba612d33c0dff9c3cc9be393

Update release.yml

view details

push time in 8 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.4

KUDO Spark Operator

created time in 8 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha d2ef35c98bb504c89a0694df4fc9f462b0bd70a1

Update release.yml

view details

push time in 8 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.3

KUDO Spark Operator

created time in 8 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 56c7e472f8ce2c78ce8da3888dde4e3b9b177121

Update release.yml

view details

push time in 8 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.2

KUDO Spark Operator

created time in 9 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha f2d76b9273c60491e6df0f17c4112ba1c5c5fe32

Add release workflow

view details

push time in 9 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.7

KUDO Spark Operator

created time in 9 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha bac988f26bf213210350d6f342afd1491f0b0830

Update release workflow

view details

push time in 9 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.6

KUDO Spark Operator

created time in 9 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha d1a120e08a3a31aff2b036708cfa93d5bc08c5cf

Update release workflow

view details

push time in 9 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.5

KUDO Spark Operator

created time in 9 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.4

KUDO Spark Operator

created time in 9 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 2112113133f64ebfe751409c3f8cc392b6768d5b

Update release workflow

view details

push time in 9 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 610bf4ae3dfad40c67d7d387862a03ed8a4a47f5

Extract version from tag

view details

push time in 9 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.3

KUDO Spark Operator

created time in 9 days

created tagalembiewski/kudo-spark-operator

tagv2.4.5-1.0.2

KUDO Spark Operator

created time in 9 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha e37d070dd1cec5d700e8e58fe3ec2430c8fe757d

Update release workflow

view details

push time in 9 days

delete tag alembiewski/kudo-spark-operator

delete tag : refs/heads/master

delete time in 9 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 8d679e45f25fdd859b55e555eb82a796ae50c133

Update main.yml

view details

push time in 9 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 5707dc8482c4e75ea6b8b2bb2c69eb4e4c288ffa

Update main.yml

view details

push time in 9 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 64b5cc1e40886f3b261a9c1e193cc0e8987151ff

Update main.yml

view details

push time in 9 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 29a73d0aa3e9d1e09acc1cd92d28b4a49828ba90

Update main.yml

view details

push time in 10 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 09710bb79fbb1c8f91ea12a255271e8bbeede75b

Update main.yml

view details

push time in 10 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 630dc64da437cf5cbd1885037245ed3581d5cbd2

Update main.yml

view details

push time in 10 days

created tagalembiewski/kudo-spark-operator

tagv1

KUDO Spark Operator

created time in 10 days

push eventalembiewski/kudo-spark-operator

Alex Lembiewski

commit sha 1dff33d38559fa66ef2a3f67e391bbf468bd20f5

Create main.yml

view details

push time in 10 days

PR opened mesosphere/kudo-spark-operator

Reviewers
Post-MWT fixes

This PR contains a few fixes in scripts as well as dashboard updates, which were made during the MWT run.

Signed-off-by: Alex Lembiewski alembiyeuski.c@d2iq.com

+4 -5

0 comment

3 changed files

pr created time in 23 days

create barnchmesosphere/kudo-spark-operator

branch : scale-test-scripts-updates

created branch time in 23 days

created tagmesosphere/kudo-spark-operator

tagv2.4.5-1.0.1

KUDO Spark Operator

created time in 24 days

release mesosphere/kudo-spark-operator

v2.4.5-1.0.1

released time in 24 days

push eventmesosphere/kudo-spark-operator

Alex Lembiewski

commit sha f4c2dff9c88215337d08b367d0df17ebf31b88bb

KUDO Spark Operator 2.4.5-1.0.1 release (#102) Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

push time in 24 days

delete branch mesosphere/kudo-spark-operator

delete branch : release-spark-operator-1.0.1

delete time in 24 days

PR merged mesosphere/kudo-spark-operator

KUDO Spark Operator 2.4.5-1.0.1 release

What changes were proposed in this pull request?

This PR concludes the KUDO Spark Operator 2.4.5-1.0.1 release

How were the changes tested?

tests from this repo

+7 -7

0 comment

5 changed files

alembiewski

pr closed time in 24 days

push eventmesosphere/kudo-spark-operator

Marcin Owsiany

commit sha 0017daa266624cdaae00447d89434a3a39dc7c4b

Bump the shared submodule to current master. (#103) This includes the following commits: Submodule shared f5116d5..361ff73: > Introduce a way of disabling $IMAGE_DISAMBIGUATION_SUFFIX. (#64) > Fix retry counter. (#62) > Bump to a recent konvoy version with better diagnostics. (#63)

view details

Marcin Owsiany

commit sha 7eeb2ce17e0cad9c5d18b49e2247b6db6640c2be

Add empty Dispatch file to silence dispatch complaints. (#104)

view details

Marcin Owsiany

commit sha dc37496eb98608d7cf0a17d4a920d082553cb0df

Bump the shared submodule to current master. (#105) This includes the following commits: Submodule shared 361ff73..ecb5d79: > added must have tags to cluster.yaml (#65)

view details

Alex Lembiewski

commit sha 3bb7bd2deef92d33f661d9a7f833262fbd8ec6f9

Merge branch 'master' into release-spark-operator-1.0.1

view details

push time in 24 days

PR opened mesosphere/kudo-spark-operator

Reviewers
KUDO Spark Operator 2.4.5-1.0.1 release

What changes were proposed in this pull request?

This PR concludes the KUDO Spark Operator 2.4.5-1.0.1 release

How were the changes tested?

tests from this repo

+7 -7

0 comment

5 changed files

pr created time in a month

create barnchmesosphere/kudo-spark-operator

branch : release-spark-operator-1.0.1

created branch time in a month

delete branch mesosphere/kudo-spark-operator

delete branch : chore-enable-metrics-for-scale-tests

delete time in a month

push eventmesosphere/kudo-spark-operator

Anton Kirillov

commit sha 0e90a51f9d36d46fd8647eae05d64f29723e59b6

chore: fix scale test install script (#98) Co-authored-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

push time in a month

PR merged mesosphere/kudo-spark-operator

chore: fix scale test install script

What changes were proposed in this pull request?

This PR adds minor fixes to install script used for scale tests

Why are the changes needed?

To enable metrics reporting

How were the changes tested?

Tested during scale tests

+3 -1

0 comment

1 changed file

akirillov

pr closed time in a month

push eventmesosphere/kudo-spark-operator

Alex Lembiewski

commit sha 64cd8e9fb43ef6c441924f30fa07d8fd8b4bb8c7

Spark Operator updated to v1beta2-1.1.1-2.4.5, bump KUDO version to 0.13.0 (#100) * Bump KUDO version to 0.13.0 * Update operators submodule * Switch to using Golang build feature for parsing tests * Use 'kudo uninstall' command for removing operator instances * Setting 'sparkJobNamespace' in tenancy tests, update scale test scripts * Fix multi-tenancy tests - commented-out test for CRD deletion - set 'sparkJobNamespace' to enable job isolation between multiple operators * Bump query timeout and retry delay * Use a randomly generated name for operator instance * Update Operator submodule to upstream tag v1beta2-1.1.1-2.4.5 Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

Alex Lembiewski

commit sha 8d0a17ce94390e8ed630fc1c65be29bda5f7d09d

Merge branch 'master' into chore-enable-metrics-for-scale-tests

view details

push time in a month

PR merged kudobuilder/operators

Spark Operator 1.0.1 Release

Signed-off-by: Alex Lembiewski alembiyeuski.c@d2iq.com

+4417 -13

0 comment

27 changed files

alembiewski

pr closed time in a month

push eventkudobuilder/operators

Alex Lembiewski

commit sha d4cbfc347069d1e1f2795525a2c3e04e90b3d8a3

Spark Operator 1.0.1 Release (#265) Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

push time in a month

delete branch mesosphere/kudo-spark-operator

delete branch : bump-kudo-version

delete time in a month

push eventmesosphere/kudo-spark-operator

Alex Lembiewski

commit sha 64cd8e9fb43ef6c441924f30fa07d8fd8b4bb8c7

Spark Operator updated to v1beta2-1.1.1-2.4.5, bump KUDO version to 0.13.0 (#100) * Bump KUDO version to 0.13.0 * Update operators submodule * Switch to using Golang build feature for parsing tests * Use 'kudo uninstall' command for removing operator instances * Setting 'sparkJobNamespace' in tenancy tests, update scale test scripts * Fix multi-tenancy tests - commented-out test for CRD deletion - set 'sparkJobNamespace' to enable job isolation between multiple operators * Bump query timeout and retry delay * Use a randomly generated name for operator instance * Update Operator submodule to upstream tag v1beta2-1.1.1-2.4.5 Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

push time in a month

PR merged mesosphere/kudo-spark-operator

Spark Operator updated to v1beta2-1.1.1-2.4.5, bump KUDO version to 0.13.0

What changes were proposed in this pull request?

This PR updates upstream operator to v1beta2-1.1.1-2.4.5 and also bumps KUDO to the latest version in tests (and in the CI). Additional changes/fixes:

  • TestCRDsDeletion test is disabled due to current KUDO limitations
  • tests refactorings;
  • switch to kudo uninstall for removing instances;
  • scale test scripts updates;
  • switch to using the new Golang build feature for parsing Go tests, which has become available after the recent TC upgrade.

Why are the changes needed?

  • to verify Spark Operator is running on the latest KUDO

How were the changes tested?

tests from this repo

+78 -150

0 comment

13 changed files

alembiewski

pr closed time in a month

push eventkudobuilder/operators

Alex Lembiewski

commit sha 330605461e98a605844d8dfb77b9aa03096ba4f3

Update docs Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

push time in a month

push eventmesosphere/kudo-spark-operator

Alex Lembiewski

commit sha 37d82106a86cccaa63c7e64c6623dd56751e4742

Update Operator submodule to upstream tag v1beta2-1.1.1-2.4.5

view details

push time in a month

push eventmesosphere/spark-on-k8s-operator

Chaoran Yu

commit sha 4d5fcaff3d2acb8567e3985b651d35bb826ca4a6

Go and dependency upgrades (#824)

view details

Jim Kleckner

commit sha d968a5a287d55fe7e84beba57134b06c0608128a

Make spark-local-dir-1 be spark-local-dir-2 (#829) This would make it match the location.

view details

Yinan Li

commit sha 60676326be905c16bae5c2007cb287114dc82a00

Update user-guide.md (#830) Expand the description of when to use volumes instead of `/tmp` for scratch space.

view details

Yinan Li

commit sha 18572fa33bb0b99ed2f5ce24fa8be8f364c727f4

Added terminationGracePeriodSeconds and pod/container lifecycle hook to driver pods (#811) * feat: delete driver pods with a grace period * feat: adding lifecycle pod spec for driver pods * adding tests for grace period and lifecycle * fix: adding user guide for termination grace period and container hooks

view details

roitvt

commit sha bcf75c9b254591691eb48748575f29e1503c325b

add Nielsen Identity Engine to who using (#836)

view details

isan_rivkin

commit sha dd3df7a86e52aa1654b32b273fe3ae68fcaabffa

Update links (#837) Updated a broken link and reference on the page.

view details

Yinan Li

commit sha aae36546e51e714db8280a639cc0c88e2ecdab3d

Fix for #826 and some refactoring (#832)

view details

push time in a month

push eventalembiewski/spark-on-k8s-operator

Hu Sheng

commit sha a2403c2c3969d9daf7c4ab8e77cb7d6d2cf04bb6

Add batchSchedulerOptions support (#606) * Add batchSchedulerOptions support * Add batchSchedulerOptions support

view details

Yinan Li

commit sha 409ac4c4439098201f5d9c5baa1ae53dcf675ddd

Run controller-gen to update CRD definitions (#623)

view details

Yinan Li

commit sha bae9d14be3825b023453417048bf8679df8bb9ea

Derive driver pod name if it's not found in status (#624)

view details

Coetzee van Staden

commit sha c16a2f8249502f98b5cb66d80da8b7a88c732adf

Enumerate string spec on Python version (#629)

view details

Yinan Li

commit sha af4ab7d08e42453cf2d99e5641aaff71ca4c0598

Fixed kubebuilder annotation for pythonVersion (#630)

view details

Yinan Li

commit sha 86ee076aab8d1bf64bd25dd6d7f400e416e7a8d4

Upgraded default Spark version from 2.4.0 to 2.4.4 (#625)

view details

Hu Sheng

commit sha 001644602c668ac53caaacf78c1fae08f7de832b

Fix integration issue with volcano (#632) See above

view details

Yinan Li

commit sha 74bd887581bd6467662371b81fd5bc03c58bd313

Replaced 2.4.0 with 2.4.4 (#634)

view details

kevin hogeland

commit sha 1e716f80aca106c481bed79c33943433e50a3814

Fix .gitignore typo (#636)

view details

kevin hogeland

commit sha d2f1d75884ddeedfa2c58ea24d2358c001387765

Fix namespace selector when webhook failure disabled (#635)

view details

Yinan Li

commit sha 5d9e8e924ff23a30fe3c64363f84865b0148a807

Use ExponentialBackoff when updating SparkApplication status (#638)

view details

Yinan Li

commit sha 3e6fabb0d4423c6379ee709e3f7e26be44a5748a

Updated the Prerequisite section of README.md (#646)

view details

akhurana001

commit sha 61ce189fc87e38563a0f415edd977216545d00f7

Fix v1beta2: Use UpdateStatus to update subresource (#645) * Use UpdateStatus for subresource update * Remove terminationTime check * Remove status validation * Use UpdateStatus for ScheduledSparkApplications

view details

Yinan Li

commit sha 22e51f265e4bdd2100eb5391a33d97afcfc8fc45

Fixed RBAC permissions after enabling of subresources (#647)

view details

Yinan Li

commit sha f78361119976beb7a147df9cd64e1fdd317b9311

Fixed a typo in the validation tag for Prometheus port (#649)

view details

Pasalietis

commit sha d615901d19b3f11fc99759892bc08d44a041fb08

Adding exacaster (#659)

view details

Hen Ben Hemo

commit sha 2b260cb6af675abf3710313a60dc7e3b99bcff73

Add Riskified to who-is-using.md (#663)

view details

akhurana001

commit sha 85180bc59327a9216f4d36de1a1635f1d3ec45f3

Update Ingress details in quick-start-guide.md (#667) * Update quick-start-guide.md * Update Ingress Setup docs

view details

Vaishnavi Giridaran

commit sha c40596e9b1c15c4b62a17de06df5e53b91438b1a

Fix typo in the k8s code gen in the developer guide. (#668)

view details

XsWack

commit sha 4990c026d087c2fa2fe570af06621a46bd9c41f1

if ApplicationState is SUCCEEDING and we could found the executor pod, assume the executor pod has been completed (#672)

view details

push time in a month

PR closed mesosphere/kudo-spark-operator

Spark: change `sparkJobNamespace` parameter propagation logic

Signed-off-by: Alex Lembiewski alembiyeuski.c@d2iq.com

What changes were proposed in this pull request?

Resolves ticket ID or # (issue) (Please include a summary of the change and which issue is fixed)

Why are the changes needed?

(Please clarify why the changes are needed. For instance, please also include relevant motivation and context)

How were the changes tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If tests are complex, please provide steps to test locally)

+1 -1

0 comment

1 changed file

alembiewski

pr closed time in a month

PR opened mesosphere/kudo-spark-operator

Spark: change `sparkJobNamespace` parameter propagation logic

Signed-off-by: Alex Lembiewski alembiyeuski.c@d2iq.com

What changes were proposed in this pull request?

Resolves ticket ID or # (issue) (Please include a summary of the change and which issue is fixed)

Why are the changes needed?

(Please clarify why the changes are needed. For instance, please also include relevant motivation and context)

How were the changes tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If tests are complex, please provide steps to test locally)

+1 -1

0 comment

1 changed file

pr created time in a month

PR opened kudobuilder/operators

Reviewers
Spark Operator 1.0.1 Release

Signed-off-by: Alex Lembiewski alembiyeuski.c@d2iq.com

+4413 -11

0 comment

26 changed files

pr created time in a month

create barnchkudobuilder/operators

branch : alex/spark-release

created branch time in a month

push eventmesosphere/kudo-spark-operator

Alex Lembiewski

commit sha c9f99f029e0d2b5ceb07bbbb13a538f5d8257de8

Fix multi-tenancy tests - commented-out test for CRD deletion - set 'sparkJobNamespace' to enable job isolation between multiple operators Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

Alex Lembiewski

commit sha e34434a3286517faceecfb07ef01215a74392460

Bump query timeout and retry delay Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

Alex Lembiewski

commit sha 9f9a23af3dd49e48d30c377b817a5f8cf1c302af

Use randomly generated name for operator instance Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

push time in a month

issue closedkudobuilder/kudo

Cluster-scoped resources are not being deleted after operator instance is removed

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security sensitive, please disclose it privately to a security contact: https://github.com/kudobuilder/kudo/blob/master/SECURITY_CONTACTS -->

What happened: Spark Operator tests started failing after we bumped the KUDO version in CI (0.10.1 -> 0.13.0). After a quick investigation, we found that some tests are failing because of cluster-scoped resources are not being cleaned-up when the operator instance is deleted. Is it some sort of regression or is it expected behavior of the latest version?

What you expected to happen: there are no cluster-scoped resources remain in the cluster after the operator is removed.

How to reproduce it (as minimally and precisely as possible): Create a local cluster using kind and init kudo:

kind create cluster && kubectl kudo init

Install Spark operator from the registry:

kubectl kudo install spark

Then uninstall the instance:

kubectl kudo uninstall --instance spark-instance

Check CRDs or cluster-roles:

kubectl get crd | grep spark
scheduledsparkapplications.sparkoperator.k8s.io   2020-05-25T14:54:55Z
sparkapplications.sparkoperator.k8s.io            2020-05-25T14:54:55Z

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): v1.17.0
  • Kudo version (use kubectl kudo version): 0.13.0
  • Operator: KUDO Spark
  • operatorVersion: spark-1.0.0
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release): macOS 10.15.4
  • Kernel (e.g. uname -a): uname -a Darwin MacBook-Pro.local 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar 4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64
  • Install tools:
  • Others:

closed time in a month

alembiewski

issue commentkudobuilder/kudo

Cluster-scoped resources are not being deleted after operator instance is removed

Got it. Thanks, @porridge!

alembiewski

comment created time in a month

issue openedkudobuilder/kudo

Cluster-scoped resources are not being deleted after operator instance is removed

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security sensitive, please disclose it privately to a security contact: https://github.com/kudobuilder/kudo/blob/master/SECURITY_CONTACTS -->

What happened: Spark Operator tests started failing after we bumped the KUDO version in CI (0.10.1 -> 0.13.0). After a quick investigation, we found that some tests are failing because of cluster-scoped resources are not being cleaned-up when the operator instance is deleted. Is it some sort of regression or is it expected behavior of the latest version?

What you expected to happen: there are no cluster-scoped resources remain in the cluster after the operator is removed.

How to reproduce it (as minimally and precisely as possible): Create a local cluster using kind and init kudo:

kind create cluster && kubectl kudo init

Install Spark operator from the registry:

kubectl kudo install spark

Then uninstall the instance:

kubectl kudo uninstall --instance spark-instance

Check CRDs or cluster-roles:

kubectl get crd | grep spark
scheduledsparkapplications.sparkoperator.k8s.io   2020-05-25T14:54:55Z
sparkapplications.sparkoperator.k8s.io            2020-05-25T14:54:55Z

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): v1.17.0
  • Kudo version (use kubectl kudo version): 0.13.0
  • Operator: KUDO Spark
  • operatorVersion: spark-1.0.0
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release): macOS 10.15.4
  • Kernel (e.g. uname -a): uname -a Darwin MacBook-Pro.local 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar 4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64
  • Install tools:
  • Others:

created time in a month

push eventmesosphere/kudo-spark-operator

Alex Lembiewski

commit sha 5e8c33d5015ffb2f4f79e0e46a7dd30ffb6b39a8

Use 'kudo uninstall' command for removing operator instances Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

Alex Lembiewski

commit sha 559b2a0f5cf6f6476c2eaa86b4a39bafab29835f

Setting 'sparkJobNamespace' in tenancy tests, update scale test scripts Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

push time in a month

PR opened mesosphere/kudo-spark-operator

Reviewers
Bump KUDO version to 0.13.0

What changes were proposed in this pull request?

This PR updates KUDO to the latest version for tests (and in the CI). Also, it switches to using the new Golang build feature for parsing Go tests, which has become available after the recent TC upgrade.

Why are the changes needed?

  • to verify Spark Operator is running on the latest KUDO

How were the changes tested?

tests from this repo

+6 -5

0 comment

5 changed files

pr created time in a month

create barnchmesosphere/kudo-spark-operator

branch : bump-kudo-version

created branch time in a month

push eventkudobuilder/operators

Alex Lembiewski

commit sha 1db171b3597ba8de91637df6e94edadea4d97d52

Spark: change `sparkJobNamespace` parameter propagation logic (#264) Signed-off-by: Alex Lembiewski <alembiyeuski.c@d2iq.com>

view details

push time in a month

PR merged kudobuilder/operators

Spark: change `sparkJobNamespace` parameter propagation logic

This PR changes the logic around sparkJobNamespace parameter handling to prevent switching to operator namespace by default and pass the original parameter value to operator arguments. Also, it updates the documentation to reflect the change.

Signed-off-by: Alex Lembiewski alembiyeuski.c@d2iq.com

+5 -8

0 comment

3 changed files

alembiewski

pr closed time in a month

Pull request review commentkudobuilder/operators

Spark: change `sparkJobNamespace` parameter propagation logic

 spec:         {{ end }}         args:         - -v={{ .Params.logLevel }}-        {{- if (ne .Params.sparkJobNamespace "") }}

Yes, I’ve checked and can confirm empty value works, so we can pass the parameter value directly to the operator

alembiewski

comment created time in 2 months

create barnchmesosphere/kudo-spark-operator

branch : change-spark-job-namespace

created branch time in 2 months

PR opened kudobuilder/operators

Reviewers
Spark: change `sparkJobNamespace` parameter propagation logic

This PR changes the logic around sparkJobNamespace parameter handling to prevent switching to operator namespace by default and pass the original parameter value to operator arguments. Also, it updates the documentation to reflect the change.

Signed-off-by: Alex Lembiewski alembiyeuski.c@d2iq.com

+5 -8

0 comment

3 changed files

pr created time in 2 months

create barnchkudobuilder/operators

branch : change-spark-job-namespace

created branch time in 2 months

issue commentkudobuilder/kuttl

Allow partial comparison of slices in asserts

Here is the example: let's say we have a custom resource TrainingJob, which describes some machine-learning training job, and it has a status.conditions[] field (slice). The job goes through several conditions before it reaches the final state. In my test case, I want to check the job is succeeded. Currently, to work-around the slice length mismatch error, my assert file looks like this:

kind: TrainingJob
metadata:
  name: ml-job
status:
  conditions:
  - reason: JobCreated      # I have to provide all the intermediate conditions here
  - reason: JobRunning      # to make kuttl happy
  - reason: JobSucceeded   <-- this is what I actually want to test
    status: "True"
    type: Succeeded

This is how I would like my assert to be:

kind: MLJob
metadata:
  name: ml-job
status:
  conditions:
  - reason: JobSucceeded
    status: "True"
    type: Succeeded

So the problem is that I have to specify all the conditions (but this could be env variables, volumes - any property with array type).

alembiewski

comment created time in 2 months

more