profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/MrLotU/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Jari (LotU) MrLotU @wehkamp The Netherlands https://jarict.com Tech Lead @wehkamp | Swift Developer | WWDC 2017 & 2019 Scholarship winner | Server Side Swift & Video team @raywenderlich

autimatisering/DeclarativeVapor 1

Declarative Programming in Vapor APIs

MrLotU/LearningJournalTH 1

Project 5 of the Treehouse Python Techdegree

MrLotU/AdventOfCode2018 0

Advent Of Code 2018 submissions

MrLotU/AdventOfCode2020 0

https://adventofcode.com 2020

MrLotU/articles 0

Articles for NSHipster.com

MrLotU/auth-template 0

A starting point for Vapor applications using the auth provider.

issue openedapple/swift-metrics-extras

Initial release

I just realized this when trying to use this in one of my projects, but this package never got it's first release published. Could we get a 1.0 released anytime soon? 😄

created time in 5 days

issue commentMrLotU/SwiftPrometheus

Release 1.0

This was the initial plan, indeed. However more recently I've been thinking about those 3 stories specifically and realised I'm not a huge fan of steering the library the way it'd require.

I'm more and more considering just "wont-fix"-ing them, and releasing a 1.0 in it's current state (Alpha 15).

Open to suggestions & comments though 😄

ktoso

comment created time in 18 days

release MrLotU/SwiftPrometheus

1.0.0-alpha.15

released time in 18 days

created tagMrLotU/SwiftPrometheus

tag1.0.0-alpha.15

Client side Prometheus library in Swift

created time in 18 days

push eventMrLotU/SwiftPrometheus

Anton

commit sha ba329daafa03027d36cda605df6c0d9b065b088e

move labels equality check out of critical section in summary/histogram (#61) * move labels equality check out of critical section in summary/histogram * preconditions, addressing review comments

view details

push time in 18 days

PR merged MrLotU/SwiftPrometheus

move labels equality check out of critical section in summary/histogram

<!-- 🚀 Thank you for contributing! --->

This PR moves Labels equality check outside of Lock. The rationale behind it is following: App's metrics list is usually bound and don't grow a lot after some time of the app being operational. The code already has this constraint implicitly by the fact that none of Prometheus.metrics, Histogram.subHistograms, Summary.subSummaries, Counter.metrics are bounded, and will eventually cause an OOM if grow indefinitely. From the domain standpoint, unique label names and values are discouraged and don't provide a lot of insight to app's behaviour. With the above constraint we know that after some time of app being operational, all or most of label key-value pairs are already stored in corresponding Histogram.subHistograms / Summary.subSummaries. This makes getting a subSummary/subHistogram from the map a target for optimisations: getOrCreate flow becomes highly skewed towards get part. Suggested change improves performance of get part of getOrCreate and increases the cost less frequently used create part. On my local machine this results in more than 2x throughput boost for the following test setups:


    func testConcurrent() throws {
        let prom = PrometheusClient()
        let histogram = prom.createHistogram(forType: Double.self, 
                                             named: "my_histogram",
                                             helpText: "Histogram for testing",
                                             buckets: Buckets.exponential(start: 1, factor: 2, count: 63),
                                             labels: DimensionHistogramLabels.self)
        let elg = MultiThreadedEventLoopGroup(numberOfThreads: 8)
        let semaphore = DispatchSemaphore(value: 2)
        let time = DispatchTime.now()
        elg.next().submit {
            while DispatchTime.now().uptimeNanoseconds - time.uptimeNanoseconds < 10_000_000_000 {
                let labels = DimensionHistogramLabels([("myValue", "1")])
                histogram.observe(1.0, labels)
            }
            semaphore.signal()
        }
        elg.next().submit {
            while DispatchTime.now().uptimeNanoseconds - time.uptimeNanoseconds < 10_000_000_000 {
                let labels = DimensionHistogramLabels([("myValue", "2")])
                histogram.observe(1.0, labels)
            }
            semaphore.signal()
        }
        semaphore.wait()
        try elg.syncShutdownGracefully()
        print(histogram.collect())
        print(DispatchTime.now().uptimeNanoseconds - time.uptimeNanoseconds)
    }

Checklist

  • [ + ] The provided tests still run.
  • [ + ] I've created new tests where needed.
  • [ N/A ] I've updated the documentation if necessary.

Motivation and Context

This PR optimises metric emission, which results in higher throughput and lower impact on running system.

Description

scope of locking is now reduced to only obtain/modify current value of PromHistogram.subHistograms / PromSummary.subSummaries If subHistogram/subSummaries for given labels is not yet added to the corresponding PromHistogram / PromSummary, the lock will be held twice: to get current value and to re-check + store new value.

  • Created tests to access/update Summary/Histogram from multiple threads
  • Ran TSAN against the change to check for concurrency issues.
+168 -54

0 comment

4 changed files

avolokhov

pr closed time in 18 days

PullRequestReviewEvent

Pull request review commentMrLotU/SwiftPrometheus

Move to GH actions

 final class GaugeTests: XCTestCase {         """)     } +    #if os(Linux)

If I understand correctly this would turn this test from sleeping for 0.05 seconds, to sleeping for 50. Which would than require to also update the asserts based on the time dilation, correct? In that case, would that not only make the problem worse?

In this case it's not a timeout that's messing things up, it's measuring the time a Thread.sleep takes, and on slow CI machines it takes longer than it's supposed to, resulting in "wrong" metrics.

MrLotU

comment created time in 25 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentMrLotU/SwiftPrometheus

Move to GH actions

 final class GaugeTests: XCTestCase {         """)     } +    #if os(Linux)

Any input appreciated. CC @ktoso

MrLotU

comment created time in 25 days

Pull request review commentMrLotU/SwiftPrometheus

Move to GH actions

 final class GaugeTests: XCTestCase {         """)     } +    #if os(Linux)

Not super happy with this solution, but from some checking the MacOS runners make this sleep last somewhere between 0.05 and 0.1 seconds, which makes these tests incredibly flaky. Locally they run just fine.

MrLotU

comment created time in 25 days

PullRequestReviewEvent

push eventMrLotU/SwiftPrometheus

MrLotU

commit sha 89d36a7b126b748379c2e4049744ce2bed68a680

Don't run timing based tests on MacOS

view details

push time in 25 days

push eventMrLotU/SwiftPrometheus

MrLotU

commit sha e139f9e827e9ee6cb250033f9624a27a3bd9aa46

Add debug output to failing tests

view details

push time in 25 days

push eventMrLotU/SwiftPrometheus

MrLotU

commit sha 475dc0dfe369f225c6b697d4f136ddbd622b0859

Add floating point counter

view details

Mikhail Akopov

commit sha e85499c759b3f0bd2ad6188623c8b24b47a70e7d

Introduce PromSummary.capacity to improve performance (#57) * Introduce PromSummary.capacity to improve performance * Add a test for summary with custom capacity * Use Deque instead of Array to store summary values * Revert "Use Deque instead of Array to store summary values" This reverts commit 1dd8c32f1aebed776f520496f60dc6dd1a076613. * Use CircularBuffer instead of Array to store summary values * Slightly improve wording in comments

view details

MrLotU

commit sha 354fa7e939461dc973a9bed072dc072f9c0a3d6e

Merge branch 'master' into feature/floating-point-counter

view details

MrLotU

commit sha 41247dc2f21204232d2b3f75a16fee403f8e3866

swift-metrics 2.2.0

view details

Konrad `ktoso` Malawski

commit sha 7e0cf8f913ed4353462bd4ad46f1db1b34c28a82

Merge pull request #55 from MrLotU/feature/floating-point-counter Add floating point counter

view details

Anton

commit sha b31a6e6d0969e5b20c380db15d2992d870c88163

Finer concurrency in 'MetricType.observe' / 'MetricType.collect' (#59) * finer concurrency when collecting metrics * better concurrency for 'PromHistogram.observe' * make subHistograms a map

view details

Anton

commit sha 196573e2f9cdbbccfb14a409270c725f7b436edd

make metrics collection independed from metrics creation (#60) Co-authored-by: Jari (LotU) <j.koopman@jarict.nl>

view details

MrLotU

commit sha 731ba3ee38b1ae4003dfc9010c7e6519aa1fb1ee

Merge branch 'master' into gh_actions

view details

push time in 25 days

push eventMrLotU/SwiftPrometheus

MrLotU

commit sha db0dad805eb48a6ebf2e143a8235a07a817a162b

Make time based tests less flakey

view details

push time in 25 days

push eventMrLotU/SwiftPrometheus

Anton

commit sha 196573e2f9cdbbccfb14a409270c725f7b436edd

make metrics collection independed from metrics creation (#60) Co-authored-by: Jari (LotU) <j.koopman@jarict.nl>

view details

push time in 25 days

PR merged MrLotU/SwiftPrometheus

make metrics collection independed from metrics creation

This PR allows metrics collection to be concurrent to metrics creation. SwiftPrometheus' metrics types are already thread-safe, there's no need in holding a global lock while calling .collect on every single metric.

Checklist

  • [ + ] The provided tests still run.
  • [ N/A ] I've created new tests where needed.
  • [ N/A ] I've updated the documentation if necessary.

Motivation and Context

Currently metrics collection is done under a global lock and it's stopping new metrics from being created in parallel. Allowing this to happen in parallel will reduce the impact metrics framework is having on the running system.

Description

PrometheusClient.collect now only holds global lock to get a list on metrics in a thread safe manner. metric.collect on individual metric is called without a global lock and relying on corresponding metric type to be thread safe.

+8 -10

0 comment

1 changed file

avolokhov

pr closed time in 25 days

PullRequestReviewEvent

push eventavolokhov/SwiftPrometheus

Anton

commit sha b31a6e6d0969e5b20c380db15d2992d870c88163

Finer concurrency in 'MetricType.observe' / 'MetricType.collect' (#59) * finer concurrency when collecting metrics * better concurrency for 'PromHistogram.observe' * make subHistograms a map

view details

Jari (LotU)

commit sha 7706772b90b78a8d7921e84d08e142a074520b8f

Merge branch 'master' into prom_concurrency

view details

push time in 25 days

push eventMrLotU/SwiftPrometheus

Anton

commit sha b31a6e6d0969e5b20c380db15d2992d870c88163

Finer concurrency in 'MetricType.observe' / 'MetricType.collect' (#59) * finer concurrency when collecting metrics * better concurrency for 'PromHistogram.observe' * make subHistograms a map

view details

push time in 25 days

PR merged MrLotU/SwiftPrometheus

Finer concurrency in 'MetricType.observe' / 'MetricType.collect'

Every metric type has a lock that's taken for the whole duration of consuming a metric value, and also for the whole duration of observe. This PR puts as much computations out of critical section as we can afford, which results in a significant performance boost. With these changes, Lock.withLock takes only 25% of PromHistogram.observe, of which half is locking/unlocking and the other half is collection.filter. Reducing the scope of locking helps bringing lock.withLock to 50% of PromHistogram.observe. Changing subHistograms from array to map, reduces it further to 25%

Checklist

  • [ + ] The provided tests still run.
  • [ N/A ] I've created new tests where needed.
  • [ N/A ] I've updated the documentation if necessary.

Motivation and Context

This PR brings concurrency optimisations that result in better emission latencies.

Description

Every MetricType's lock is now only held during access to corresponding state in a thread safe manner, all computations are moved outside critical section. subHistograms is now a Map instead of array.

+139 -135

0 comment

5 changed files

avolokhov

pr closed time in 25 days

PullRequestReviewEvent

pull request commentapple/swift-metrics

Introduce FloatingPointCounter

Native SwiftPrometheus support added in https://github.com/MrLotU/SwiftPrometheus/releases/tag/1.0.0-alpha.14

rauhul

comment created time in a month

pull request commentMrLotU/SwiftPrometheus

Add floating point counter

Released in 1.0.0-alpha.14

MrLotU

comment created time in a month

created tagMrLotU/SwiftPrometheus

tag1.0.0-alpha.14

Client side Prometheus library in Swift

created time in a month

release MrLotU/SwiftPrometheus

1.0.0-alpha.14

released time in a month

delete branch MrLotU/SwiftPrometheus

delete branch : feature/floating-point-counter

delete time in a month

push eventMrLotU/SwiftPrometheus

MrLotU

commit sha daf102f0c9234011e013d174b82861ef94decfe8

Fix label equality issue

view details

Mikhail Akopov

commit sha e85499c759b3f0bd2ad6188623c8b24b47a70e7d

Introduce PromSummary.capacity to improve performance (#57) * Introduce PromSummary.capacity to improve performance * Add a test for summary with custom capacity * Use Deque instead of Array to store summary values * Revert "Use Deque instead of Array to store summary values" This reverts commit 1dd8c32f1aebed776f520496f60dc6dd1a076613. * Use CircularBuffer instead of Array to store summary values * Slightly improve wording in comments

view details

MrLotU

commit sha 354fa7e939461dc973a9bed072dc072f9c0a3d6e

Merge branch 'master' into feature/floating-point-counter

view details

MrLotU

commit sha 41247dc2f21204232d2b3f75a16fee403f8e3866

swift-metrics 2.2.0

view details

push time in a month