profile
viewpoint

Bert-R/quartz-repro 1

Quartz example triggering one job twice

Bert-R/b3-propagation 0

Repository that describes and sometimes implements B3 propagation

Bert-R/citus_docs 0

Citus Documentation

Bert-R/eventuate-client-java 0

Eventuate Java client

Bert-R/fhir 0

FHIR Protocol Buffers

Bert-R/FhirPathTester 0

Simple WPF HL7 Fluentpath Tester tool

Bert-R/ghprb-plugin 0

github pull requests builder plugin for Jenkins

Bert-R/gradle-docker 0

A Gradle plugin to build Docker images from the build script.

Bert-R/gs-accessing-data-jpa 0

Accessing Data with JPA :: Learn how to work with JPA data persistence using Spring Data JPA.

Bert-R/Hangfire.Documentation 0

Sphinx-based documentation for Hangfire

pull request commentyonadev/yona-server

YD-683 Don't fail pin reset when user is removed

With the last update all tests work and the change in entity loading is reasonable. I consider this PR ready for merging.

Bert-R

comment created time in 4 hours

push eventyonadev/yona-server

Bert Roos

commit sha 94222968daf9786956f85b55e9a3ba86ce83fcce

YD-687 Optimized entity loading Regained pretty much all of the previous loss by using the cached user anonymized entity. Along with this corrected a slight mistake in UserAddService (accidentally used a static method on an instance).

view details

push time in 4 hours

push eventyonadev/yona-server

Bert Roos

commit sha e7ea610c8d566cd95ad15af1f9aec973045f6012

YD-687 Correct goal update scenarios The previous change broke scenarios where the goal with activities was updated. The issue is that originally, we initialized the GoalIdMapping using the DTOs but then we started using the entities. The DTOs have a collection goals which actually contains goalsIncludingHistoryItems. Because of that, the tests that use activities on updated goals now fail, as the history items are not available in the mapping. This is corrected now. * The goal ID mapping is now initialized including the history items * Several more goal collections that actually contain "goals including history items" are now renamed * The user entity is locked prior to making goal changes * The behavior for calculating the set of goals including history items (which was already based on entities) is now moved to the UserAnonymized entity * The entity loading statistics were recalculated: the situation worsened considerably. Before this PR is merged, I want to see whether that can be recovered.

view details

push time in 5 hours

push eventyonadev/yona-server

Bert Roos

commit sha 519bfb9ee40c600da579612c4a2f532c35eb9e63

YD-665 Improve entity loading (#592) * YD-665 Improve entity loading Initially, the test run loaded 1466 entities through 308 statements and 146 queries. The following improvements were made: * Eagerly load DeviceAnonymized along with every Activity. Reduced loading 30 entities, with the same number of statements and queries * Added fetch = FetchType.LAZY to devicesAnonymized on UserAnonymized. Reduced loading 5 entities at the cost of one prepared statement * Instead of loading the UserAnonymized when needed, take it from the UserAnymizedService, which caches it. Reduced loading 198 entities, preparing 12 statements, executing 65 queries. If we ignore the first time loading of the user, entity loading improves by 201 and prepared statements by 22. In total, this leads to the following new numbers: the test run loaded 1248 (vs 1466) entities through 259 (vs 308) statements and 97 (vs 146) queries Along with this: * Improved BasicBuddyTest.groovy to make it independent of the order of the goals. * Corrected the week date format in YonaServer.toIsoWeekDateString * YD-665 Be explicit about availability of user anonymized Instead of 'get' on the optional userAnomymizedDto, we now use 'orElseThrow' with a clear message: Should have user anonymized when buddy relationship is established * YD-665 Use cached user anonymized for device info The anonymized user entity is cached. It includes the anonymized device info, which is often needed. This is now reused, so the get....First calls are occasionally more inefficient, but the get...Second calls are a lot more efficient. This leads to the following new numbers: the test run loaded 1094 (vs 1248) entities through 252 (vs 259) statements and 86 (vs 97) queries * YD-665 Do not load UserAnon in subsequent requests The UserAnonymized entity was still being loaded in nonfirst user load requests. This is resolved now. This leads to the following new numbers: the test run loaded 1076 (vs 1094) entities through 208 (vs 252) statements and 86 (vs 86) queries * YD-665 Take buddy anonymized from cache The BuddyAnonymized entity was still being loaded in nonfirst user load requests. This is resolved now. This leads to the following new numbers: the test run loaded 1013 (vs 1076) entities through 2192 (vs 208) statements and 86 (vs 70) queries * YD-665 Don't load goal for activity overview Introduced an interface above goal DTO and entity, to enable interchangeably use either one for certain things. This leads to the following new numbers: the test run loaded 997 (vs 1013) entities through 177 (vs 192) statements and 70 (vs 86) queries * YD-665 Apply more lazy loading Applied for: * Activity.deviceAnonymized * IntervalActivity.userAnonymized This leads to the following new numbers: the test run loaded 974 (vs 997) entities through 161 (vs 177) statements and 70 (vs 70) queries * YD-665 Eclipsified everything The last changes were done in IntelliJ. Given our current standard is Eclipse, I've reformatted all changed files using Eclipse and also organized the imports the Eclipse-way. Next to that updated the copyright years. * YD-665 Processed SonarQube feedback * Removed unnecessary public modifier * Removed unnecessary parameter * YD-665 Processed review comments * Renamed BuddyUserPrivateDataDto.getDeviceAnonymized into getDeviceAnonymizedIfExisting and moved its implementation into UserPrivate Along with this: * Renamed Buddy.getUserIfExists into getUserIfExisting, for alignment with other method names, also in an open PR. * YD-665 Processed review comment Spelling correction in method name: getIdWIthoutLoadingEntity --> getIdWithoutLoadingEntity

view details

Bert Roos

commit sha 0645f4127c082cd8b83eda733c40e3150c3ad2f1

YD-684 Log startup failures and shutdown (#594) * YD-684 Log startup failures and shutdown * YD-684 Implemented app close logging for all services * YD-684 Removed unused imports

view details

Bert Roos

commit sha 48e5a59c9bb56bbaf5a93c7ea18834c23b519eab

YD-685 Gracefully handle activity on orphaned user anonymized (#599) A device might continue to use its VPN after the user overwrote their account. In that case, new goal conflict messages might be created for a user user anonymized that is orphaned and where the buddies already wiped the related messages and buddy entity. Such a new goal conflict message would lead to an inconsistency where the belonging buddy entity cannot be found. To fix this, BuddyService.getBuddyUsersAnonymized now does not return buddy entities for which the corresponding buddy entity at the other side does not exist. This has a minor side effect: earlier, if two users became buddies, the goal conflicts of the accepting user would be reported to the requesting user immediately after the accepting user accepted the request. Now this is delayed a little: goal conflicts of the accepting user are now reported to the requesting user after the requesting user processed the buddy connect response message of the accepting user.

view details

Bert Roos

commit sha 1c9ad8af005cd7ce317f2e0e7ef74bb2f449c1dd

YD-685 Again support info change right after accept It used to be possible to receive a buddy info change right after that buddy accepted the buddy relationship. The change for YD-685 broke this inadvertently. For some reason, the broken test case went unnoticed. This is corrected now. Along with this: * Renamed some parameters in BuddyService to indicate we're dealing with a user anonymized rather than a regular user * Changed the order of assertions in UpdateBuddyUserInfoTest.'Richard can receive Bob's buddy info change message along with his acceptance' to give a more clear indication of a failure in the key behavior.

view details

Bert Roos

commit sha 3797f22ab402f4f0e8bb4918cfcea58fb5ce0494

Merge branch 'master' into yd-683-user-del-pin-reset

view details

push time in 8 days

push eventyonadev/yona-server

Bert Roos

commit sha 1c9ad8af005cd7ce317f2e0e7ef74bb2f449c1dd

YD-685 Again support info change right after accept It used to be possible to receive a buddy info change right after that buddy accepted the buddy relationship. The change for YD-685 broke this inadvertently. For some reason, the broken test case went unnoticed. This is corrected now. Along with this: * Renamed some parameters in BuddyService to indicate we're dealing with a user anonymized rather than a regular user * Changed the order of assertions in UpdateBuddyUserInfoTest.'Richard can receive Bob's buddy info change message along with his acceptance' to give a more clear indication of a failure in the key behavior.

view details

push time in 8 days

push eventyonadev/yona-server

Nicole Macfarlane

commit sha c14eb460a1843a84e6c6b6a9c4f98373f8ec20ea

Upgrade helm charts for new API versions

view details

Nicole Macfarlane

commit sha 1554467b1bb5cf17b9f553643aabf335fd330883

Unify on mariadb.db settings

view details

kti-nicole

commit sha fec3c5fb9dc434fab4bb255b9e601befca6a6465

Merge pull request #591 from yonadev/kti-upgrade-helm-api Upgrade helm charts for new API versions

view details

Nicole Macfarlane

commit sha cb718002f10e90a9ca8d7870efadfc42daf01812

Remove depreciated kubectl -a optioon

view details

Bert Roos

commit sha 8e0a7ae323a7cf6babebf61c36ad2d2b0707a131

YD-636 Temporary fix for flipping tests (#597) * YD-636 Temporary fix for flipping tests See the analysis on the ticket. The cause of the issue seems to be that updates to goals do not immediately reach the analysis engine service. Due to that, the next request might reach the analysis engine service before the cache update reaches it, causing the test to fail Added a 0.5 second sleep after the goal creation, marked with a // TODO. If this indeed resolves it, we need to make up our mind on a definitive fix. * Verifying new pull request build configuration This is a test commit that'll be reverted * Verifying new pull request build configuration This is a test commit that'll be reverted * Reverted test commit

view details

Bert Roos

commit sha bed6276020f065c5f12bdda5bac2075656cf692f

YD-690 Added extra logging

view details

Bert Roos

commit sha ceecca69a16a84fedc474f48a492a50061cf83f3

Merge branch 'master' of https://github.com/yonadev/yona-server

view details

Bert Roos

commit sha 5a1687ed972f2ac3a0b9c76a7cef663a55eee981

YD-690 Added SMTP connection timeouts (#601) For both SMTP and SMTPS, all set to three minutes. From infinite to 3 minutes is a significant change. If need be, we reduce the values in a next round.

view details

Bert Roos

commit sha f39b5e811d554a8d7069a167ce591b8134dda108

Start to use GitHub app credentials

view details

Bert Roos

commit sha 9d17d474b1b3ade5d904980180f495af3b5f7247

Removed excess {

view details

Bert Roos

commit sha 31f61780f915382309addc4aaba7b3d3bda31928

Use GitHub JWT And don't use the YonaBuild account anymore

view details

Nicole Macfarlane

commit sha 81c3cc9b1fd470214ef08f42b6f6cbb900ccc27b

Remove preceding slash for gitlab API call on beta/loadtest

view details

kti-nicole

commit sha 2d815d1c51f130be3745fe9cffd3ac311f0f3578

Merge pull request #602 from yonadev/CP-379844 Remove preceding slash for gitlab API call on beta/loadtest

view details

Nicole Macfarlane

commit sha 72b1671aee68c8f613cc578f8885e4cf8e8c7b41

Yaml upgrade for secret/configmap names

view details

kti-nicole

commit sha 4688f3d71420f9d57eaa91ebba3c333b4a470539

Merge pull request #603 from yonadev/CP-379852 Yaml upgrade for secret/configmap names

view details

Bert Roos

commit sha 00c6afced764f0dd59e6c9eb8b968b39109d97cc

YD-691 Use user time zone to determine creation date (#604) We used UTC to determine the user creation date, which caused the test to fail.

view details

Nicole Macfarlane

commit sha ffa118fa3c8192784e26b121d71369247a36f7c5

Add revision to Job, so helm upgrades will always trigger it

view details

kti-nicole

commit sha f30ba92383a02bc6aa54879b5c096a53709cf670

Merge pull request #605 from yonadev/CP-385463 Add helm revision to Job name, so helm upgrades will always trigger it

view details

Nicole Macfarlane

commit sha 89da4b70ba7151252c889b4c2138eca2debf66c8

Update Jenkins helm delete

view details

Bert Roos

commit sha fa571910d3ba7afe9e0820b861e09e77841caacf

Merge branch 'master' into yd-683-user-del-pin-reset

view details

push time in 14 days

pull request commentyonadev/yona-server

YD-683 Don't fail pin reset when user is removed

This one I take forward once #592 is merged. The issue is that originally, we initialized the GoalIdMapping using the DTOs and now using the entities. The DTOs have a collection goals which actually contains goalsIncludingHistoryItems. Because of that, the tests that use activities on updated goals now fail, as the history items are not available in the mapping. Once the mentioned PR is merged, I'm going to see what's the best way to fix this.

Bert-R

comment created time in 14 days

delete branch yonadev/yona-server

delete branch : yd-691-activity-test-fails-at-night

delete time in 14 days

delete branch yonadev/yona-server

delete branch : yd-690-configure-smtp-timeout

delete time in 14 days

delete branch yonadev/yona-server

delete branch : yd-636-flipping-tests

delete time in 14 days

delete branch yonadev/yona-server

delete branch : yd-682-really-clear-fb-instance-id

delete time in 14 days

delete branch yonadev/yona-server

delete branch : yd-681-spring-forward-headers

delete time in 14 days

delete branch yonadev/yona-server

delete branch : yd-678-polish-firebase

delete time in 14 days

delete branch yonadev/yona-server

delete branch : yd-679-500-on-multipart-ex

delete time in 14 days

Pull request review commentyonadev/yona-server

YD-684 Log startup failures and shutdown

 	public static void main(String[] args) 	{ 		PropertyInitializer.initializePropertiesFromEnvironment();-		SpringApplication.run(AdminServiceApplication.class, args);+		ConfigurableApplicationContext context = SpringApplication.run(AdminServiceApplication.class, args);+		ApplicationStatusLogger.addLoggerForContextClosedEvent(context);

The logger now only logs when 'our' application context is logged. Otherwise, multiple "application closed" messages would be logged for a single close, which would be confusing. I quickly tried whether e.g. the display name would be distinguishing, but it's not: Application org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@6f6745d6 closed

Bert-R

comment created time in 15 days

PullRequestReviewEvent

pull request commentyonadev/yona-server

YD-683 Don't fail pin reset when user is removed

There's one more issue with this PR, causing the ActivityTest integration test to fail. So don't merge this PR yet.

Bert-R

comment created time in 15 days

push eventyonadev/yona-server

Bert Roos

commit sha 7e91e1fae0b61026b17b7687d41019a807b34e4b

YD-683 Processed review comment The change in an earlier commit was accidentally removed. This is restored now.

view details

push time in 15 days

Pull request review commentyonadev/yona-server

YD-683 Don't fail pin reset when user is removed

 public void requestPinReset(UUID userId) 	@Transactional 	public void sendPinResetConfirmationCode(UUID userId) 	{-		User user = userService.getUserEntityById(userId);+		User user = userService.lockUserForUpdate(userId);

See my other comment.

Bert-R

comment created time in 15 days

PullRequestReviewEvent

Pull request review commentyonadev/yona-server

YD-683 Don't fail pin reset when user is removed

 public void assertValidEmailAddress(String emailAddress) 	{ 		UserAssertionService.assertValidEmailAddress(emailAddress); 	}++	/**+	 * Locks the user entity with the given ID for update, thus preventing concurrent updates on that user. To prevent from using+	 * stale data that was retrieved before any previous update completed, the Hibernate session is cleared. This operation fails+	 * if the session is dirty. Callers should obtain the lock on the user entity prior to any checks or updates. Note that this+	 * operation is idempotent: if the user was already locked during this session, that user will be returned. The session won't+	 * be cleared.+	 *+	 * @param userId The ID of the user to lock+	 * @return The locked user entity+	 */+	@Transactional+	public User lockUserForUpdate(UUID userId)

Indeed. Once the transaction completes (whether successfully or in error), the lock is released.

Bert-R

comment created time in 15 days

PullRequestReviewEvent

pull request commentyonadev/yona-server

YD-683 Don't fail pin reset when user is removed

Reviewed, because it is so large I was unable to look at all the details. In many places there is a change from DTO to entity in the code, which I think is not bad, but the reason is not entirely clear to me. I also see that locking of the user entity for update is added in multiple places. Is that only for the pin reset or is it a general improvement on race conditions?

Moreover, getting to the reason of the PR, I see no changes in sendPinResetConfirmationCode except that it locks the user for update. So that will throw if the user is removed, while the previous getUserEntityById would not throw?

Note that after the initial commit and PR creation, fixes for several related issues were pushed to this PR. The first issue, related to a PIN reset for a removed user is resolved in the first commit (7ac06bc69ab2236d609e9d4a8ddf3a2af6fb11fb) at line 69 of PinResetRequestService.java

Bert-R

comment created time in 15 days

pull request commentyonadev/yona-server

YD-665 Improve entity loading

I understand your difficulty reviewing this big PR. The issue is that the various changes depend on each other. To minimize the problem of comprehensibility, I committed each improvement separately, with an explanatory comment. The alternative would have been to do a separate PR and review consecutively for all changes. Given the changes are done in 7 iterations, the improvement process would have been extremely long.

I've pushed an update with the rename and responded to your other comments.

Bert-R

comment created time in 15 days

push eventyonadev/yona-server

Bert Roos

commit sha 4c741ab797fbd2bfa49309c73bee3ffd5750385b

YD-665 Processed review comment Spelling correction in method name: getIdWIthoutLoadingEntity --> getIdWithoutLoadingEntity

view details

push time in 15 days

Pull request review commentyonadev/yona-server

YD-665 Improve entity loading

+/*+ * Copyright (c) 2020 Stichting Yona Foundation This Source Code Form is subject to the terms of the Mozilla Public License,+ * v.2.0. If a copy of the MPL was not distributed with this file, You can obtain one at https://mozilla.org/MPL/2.0/.+ */++package nu.yona.server.goals.entities;++import java.time.LocalDateTime;+import java.time.ZonedDateTime;+import java.time.temporal.TemporalUnit;+import java.util.Optional;++import nu.yona.server.analysis.entities.DayActivity;+import nu.yona.server.util.TimeUtil;++public interface IGoal

This was introduced through this commit: 45a7f320ce78f4e11fd748b1b7e621b131ab8db5. Your comment precisely touches the issue. The optimization was to use the cached goal rather than fetching the goal entities from the database again. Given we don't want to use the DTOs in the entity layer, I introduced an interface to decouple the layers.

To reduce the entity loading as drastically as I did, it is necessary to use the cache where we can. This is one of the mechanisms. I feel it's an appropriate way to manage the dependencies between the design layers. The logic is part of the entity layer, as all shared logic is implemented as default methods on the interfaces.

Bert-R

comment created time in 15 days

PullRequestReviewEvent

Pull request review commentyonadev/yona-server

YD-665 Improve entity loading

 class YonaServer 
 	static String toIsoWeekDateString(ZonedDateTime dateTime)
 	{
-		DateTimeFormatter.ofPattern("YYYY-'W'w").format(dateTime)

Yes, it caused a failure in the HibernateStatsTest during single-digit weeks. It is part of this commit: ad4503eb30c330481412b0e053f6acd2d10b33be

Bert-R

comment created time in 15 days

PullRequestReviewEvent

Pull request review commentyonadev/yona-server

YD-665 Improve entity loading

 protected EntityWithUuid(UUID id) 		this.id = id;
 	}
 
+	public static UUID getIdWIthoutLoadingEntity(EntityWithUuid entity)

Well spotted!

Bert-R

comment created time in 15 days

PullRequestReviewEvent

fork Bert-R/kafka-service-broker-boshrelease

This BOSH release and deployment manifest deploy a cluster of kafka/kafka manager/zookeeper, plus a Cloud Foundry service broker

fork in a month

push eventyonadev/yona-server

Bert Roos

commit sha 00c6afced764f0dd59e6c9eb8b968b39109d97cc

YD-691 Use user time zone to determine creation date (#604) We used UTC to determine the user creation date, which caused the test to fail.

view details

push time in 2 months

PR merged yonadev/yona-server

YD-691 Use user time zone to determine creation date

We used UTC to determine the user creation date, which caused the test to fail.

+10 -1

0 comment

1 changed file

Bert-R

pr closed time in 2 months

PR opened yonadev/yona-server

YD-691 Use user time zone to determine creation date

We used UTC to determine the user creation date, which caused the test to fail.

+10 -1

0 comment

1 changed file

pr created time in 2 months

create barnchyonadev/yona-server

branch : yd-691-activity-test-fails-at-night

created branch time in 2 months

more