autimatisering/DeclarativeVapor 1
Declarative Programming in Vapor APIs
Project 5 of the Treehouse Python Techdegree
Advent Of Code 2018 submissions
https://adventofcode.com 2020
Articles for NSHipster.com
A starting point for Vapor applications using the auth provider.
issue openedapple/swift-metrics-extras
I just realized this when trying to use this in one of my projects, but this package never got it's first release published. Could we get a 1.0 released anytime soon? 😄
created time in 5 days
issue commentMrLotU/SwiftPrometheus
This was the initial plan, indeed. However more recently I've been thinking about those 3 stories specifically and realised I'm not a huge fan of steering the library the way it'd require.
I'm more and more considering just "wont-fix"-ing them, and releasing a 1.0 in it's current state (Alpha 15).
Open to suggestions & comments though 😄
comment created time in 18 days
push eventMrLotU/SwiftPrometheus
commit sha ba329daafa03027d36cda605df6c0d9b065b088e
move labels equality check out of critical section in summary/histogram (#61) * move labels equality check out of critical section in summary/histogram * preconditions, addressing review comments
push time in 18 days
PR merged MrLotU/SwiftPrometheus
<!-- 🚀 Thank you for contributing! --->
This PR moves Labels equality check outside of Lock.
The rationale behind it is following:
App's metrics list is usually bound and don't grow a lot after some time of the app being operational. The code already has this constraint implicitly by the fact that none of Prometheus.metrics, Histogram.subHistograms, Summary.subSummaries, Counter.metrics are bounded, and will eventually cause an OOM if grow indefinitely. From the domain standpoint, unique label names and values are discouraged and don't provide a lot of insight to app's behaviour.
With the above constraint we know that after some time of app being operational, all or most of label key-value pairs are already stored in corresponding Histogram.subHistograms / Summary.subSummaries. This makes getting a subSummary/subHistogram from the map a target for optimisations: getOrCreate flow becomes highly skewed towards get part. Suggested change improves performance of get part of getOrCreate and increases the cost less frequently used create part. On my local machine this results in more than 2x throughput boost for the following test setups:
func testConcurrent() throws {
let prom = PrometheusClient()
let histogram = prom.createHistogram(forType: Double.self,
named: "my_histogram",
helpText: "Histogram for testing",
buckets: Buckets.exponential(start: 1, factor: 2, count: 63),
labels: DimensionHistogramLabels.self)
let elg = MultiThreadedEventLoopGroup(numberOfThreads: 8)
let semaphore = DispatchSemaphore(value: 2)
let time = DispatchTime.now()
elg.next().submit {
while DispatchTime.now().uptimeNanoseconds - time.uptimeNanoseconds < 10_000_000_000 {
let labels = DimensionHistogramLabels([("myValue", "1")])
histogram.observe(1.0, labels)
}
semaphore.signal()
}
elg.next().submit {
while DispatchTime.now().uptimeNanoseconds - time.uptimeNanoseconds < 10_000_000_000 {
let labels = DimensionHistogramLabels([("myValue", "2")])
histogram.observe(1.0, labels)
}
semaphore.signal()
}
semaphore.wait()
try elg.syncShutdownGracefully()
print(histogram.collect())
print(DispatchTime.now().uptimeNanoseconds - time.uptimeNanoseconds)
}
Checklist
- [ + ] The provided tests still run.
- [ + ] I've created new tests where needed.
- [ N/A ] I've updated the documentation if necessary.
Motivation and Context
This PR optimises metric emission, which results in higher throughput and lower impact on running system.
Description
scope of locking is now reduced to only obtain/modify current value of PromHistogram.subHistograms / PromSummary.subSummaries
If subHistogram/subSummaries for given labels is not yet added to the corresponding PromHistogram / PromSummary, the lock will be held twice: to get current value and to re-check + store new value.
- Created tests to access/update Summary/Histogram from multiple threads
- Ran TSAN against the change to check for concurrency issues.
pr closed time in 18 days
Pull request review commentMrLotU/SwiftPrometheus
final class GaugeTests: XCTestCase { """) } + #if os(Linux)
If I understand correctly this would turn this test from sleeping for 0.05 seconds, to sleeping for 50. Which would than require to also update the asserts based on the time dilation, correct? In that case, would that not only make the problem worse?
In this case it's not a timeout that's messing things up, it's measuring the time a Thread.sleep takes, and on slow CI machines it takes longer than it's supposed to, resulting in "wrong" metrics.
comment created time in 25 days
Pull request review commentMrLotU/SwiftPrometheus
final class GaugeTests: XCTestCase { """) } + #if os(Linux)
Any input appreciated. CC @ktoso
comment created time in 25 days
Pull request review commentMrLotU/SwiftPrometheus
final class GaugeTests: XCTestCase { """) } + #if os(Linux)
Not super happy with this solution, but from some checking the MacOS runners make this sleep last somewhere between 0.05 and 0.1 seconds, which makes these tests incredibly flaky. Locally they run just fine.
comment created time in 25 days
push eventMrLotU/SwiftPrometheus
commit sha 89d36a7b126b748379c2e4049744ce2bed68a680
Don't run timing based tests on MacOS
push time in 25 days
push eventMrLotU/SwiftPrometheus
commit sha e139f9e827e9ee6cb250033f9624a27a3bd9aa46
Add debug output to failing tests
push time in 25 days
push eventMrLotU/SwiftPrometheus
commit sha 475dc0dfe369f225c6b697d4f136ddbd622b0859
Add floating point counter
commit sha e85499c759b3f0bd2ad6188623c8b24b47a70e7d
Introduce PromSummary.capacity to improve performance (#57) * Introduce PromSummary.capacity to improve performance * Add a test for summary with custom capacity * Use Deque instead of Array to store summary values * Revert "Use Deque instead of Array to store summary values" This reverts commit 1dd8c32f1aebed776f520496f60dc6dd1a076613. * Use CircularBuffer instead of Array to store summary values * Slightly improve wording in comments
commit sha 354fa7e939461dc973a9bed072dc072f9c0a3d6e
Merge branch 'master' into feature/floating-point-counter
commit sha 41247dc2f21204232d2b3f75a16fee403f8e3866
swift-metrics 2.2.0
commit sha 7e0cf8f913ed4353462bd4ad46f1db1b34c28a82
Merge pull request #55 from MrLotU/feature/floating-point-counter Add floating point counter
commit sha b31a6e6d0969e5b20c380db15d2992d870c88163
Finer concurrency in 'MetricType.observe' / 'MetricType.collect' (#59) * finer concurrency when collecting metrics * better concurrency for 'PromHistogram.observe' * make subHistograms a map
commit sha 196573e2f9cdbbccfb14a409270c725f7b436edd
make metrics collection independed from metrics creation (#60) Co-authored-by: Jari (LotU) <j.koopman@jarict.nl>
commit sha 731ba3ee38b1ae4003dfc9010c7e6519aa1fb1ee
Merge branch 'master' into gh_actions
push time in 25 days
push eventMrLotU/SwiftPrometheus
commit sha db0dad805eb48a6ebf2e143a8235a07a817a162b
Make time based tests less flakey
push time in 25 days
push eventMrLotU/SwiftPrometheus
commit sha 196573e2f9cdbbccfb14a409270c725f7b436edd
make metrics collection independed from metrics creation (#60) Co-authored-by: Jari (LotU) <j.koopman@jarict.nl>
push time in 25 days
PR merged MrLotU/SwiftPrometheus
This PR allows metrics collection to be concurrent to metrics creation.
SwiftPrometheus' metrics types are already thread-safe, there's no need in holding a global lock while calling .collect on every single metric.
Checklist
- [ + ] The provided tests still run.
- [ N/A ] I've created new tests where needed.
- [ N/A ] I've updated the documentation if necessary.
Motivation and Context
Currently metrics collection is done under a global lock and it's stopping new metrics from being created in parallel. Allowing this to happen in parallel will reduce the impact metrics framework is having on the running system.
Description
PrometheusClient.collect now only holds global lock to get a list on metrics in a thread safe manner. metric.collect on individual metric is called without a global lock and relying on corresponding metric type to be thread safe.
pr closed time in 25 days
push eventavolokhov/SwiftPrometheus
commit sha b31a6e6d0969e5b20c380db15d2992d870c88163
Finer concurrency in 'MetricType.observe' / 'MetricType.collect' (#59) * finer concurrency when collecting metrics * better concurrency for 'PromHistogram.observe' * make subHistograms a map
commit sha 7706772b90b78a8d7921e84d08e142a074520b8f
Merge branch 'master' into prom_concurrency
push time in 25 days
push eventMrLotU/SwiftPrometheus
commit sha b31a6e6d0969e5b20c380db15d2992d870c88163
Finer concurrency in 'MetricType.observe' / 'MetricType.collect' (#59) * finer concurrency when collecting metrics * better concurrency for 'PromHistogram.observe' * make subHistograms a map
push time in 25 days
PR merged MrLotU/SwiftPrometheus
Every metric type has a lock that's taken for the whole duration of consuming a metric value, and also for the whole duration of observe. This PR puts as much computations out of critical section as we can afford, which results in a significant performance boost.
With these changes, Lock.withLock takes only 25% of PromHistogram.observe, of which half is locking/unlocking and the other half is collection.filter.
Reducing the scope of locking helps bringing lock.withLock to 50% of PromHistogram.observe. Changing subHistograms from array to map, reduces it further to 25%
Checklist
- [ + ] The provided tests still run.
- [ N/A ] I've created new tests where needed.
- [ N/A ] I've updated the documentation if necessary.
Motivation and Context
This PR brings concurrency optimisations that result in better emission latencies.
Description
Every MetricType's lock is now only held during access to corresponding state in a thread safe manner, all computations are moved outside critical section.
subHistograms is now a Map instead of array.
pr closed time in 25 days
pull request commentapple/swift-metrics
Introduce FloatingPointCounter
Native SwiftPrometheus support added in https://github.com/MrLotU/SwiftPrometheus/releases/tag/1.0.0-alpha.14
comment created time in a month
pull request commentMrLotU/SwiftPrometheus
Released in 1.0.0-alpha.14
comment created time in a month
delete branch MrLotU/SwiftPrometheus
delete branch : feature/floating-point-counter
delete time in a month
push eventMrLotU/SwiftPrometheus
commit sha daf102f0c9234011e013d174b82861ef94decfe8
Fix label equality issue
commit sha e85499c759b3f0bd2ad6188623c8b24b47a70e7d
Introduce PromSummary.capacity to improve performance (#57) * Introduce PromSummary.capacity to improve performance * Add a test for summary with custom capacity * Use Deque instead of Array to store summary values * Revert "Use Deque instead of Array to store summary values" This reverts commit 1dd8c32f1aebed776f520496f60dc6dd1a076613. * Use CircularBuffer instead of Array to store summary values * Slightly improve wording in comments
commit sha 354fa7e939461dc973a9bed072dc072f9c0a3d6e
Merge branch 'master' into feature/floating-point-counter
commit sha 41247dc2f21204232d2b3f75a16fee403f8e3866
swift-metrics 2.2.0
push time in a month