Skip to content

move labels equality check out of critical section in summary/histogram #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 31, 2021

Conversation

avolokhov
Copy link
Contributor

This PR moves Labels equality check outside of Lock.
The rationale behind it is following:
App's metrics list is usually bound and don't grow a lot after some time of the app being operational. The code already has this constraint implicitly by the fact that none of Prometheus.metrics, Histogram.subHistograms, Summary.subSummaries, Counter.metrics are bounded, and will eventually cause an OOM if grow indefinitely. From the domain standpoint, unique label names and values are discouraged and don't provide a lot of insight to app's behaviour.
With the above constraint we know that after some time of app being operational, all or most of label key-value pairs are already stored in corresponding Histogram.subHistograms / Summary.subSummaries. This makes getting a subSummary/subHistogram from the map a target for optimisations: getOrCreate flow becomes highly skewed towards get part. Suggested change improves performance of get part of getOrCreate and increases the cost less frequently used create part. On my local machine this results in more than 2x throughput boost for the following test setups:

    func testConcurrent() throws {
        let prom = PrometheusClient()
        let histogram = prom.createHistogram(forType: Double.self, 
                                             named: "my_histogram",
                                             helpText: "Histogram for testing",
                                             buckets: Buckets.exponential(start: 1, factor: 2, count: 63),
                                             labels: DimensionHistogramLabels.self)
        let elg = MultiThreadedEventLoopGroup(numberOfThreads: 8)
        let semaphore = DispatchSemaphore(value: 2)
        let time = DispatchTime.now()
        elg.next().submit {
            while DispatchTime.now().uptimeNanoseconds - time.uptimeNanoseconds < 10_000_000_000 {
                let labels = DimensionHistogramLabels([("myValue", "1")])
                histogram.observe(1.0, labels)
            }
            semaphore.signal()
        }
        elg.next().submit {
            while DispatchTime.now().uptimeNanoseconds - time.uptimeNanoseconds < 10_000_000_000 {
                let labels = DimensionHistogramLabels([("myValue", "2")])
                histogram.observe(1.0, labels)
            }
            semaphore.signal()
        }
        semaphore.wait()
        try elg.syncShutdownGracefully()
        print(histogram.collect())
        print(DispatchTime.now().uptimeNanoseconds - time.uptimeNanoseconds)
    }

Checklist

  • [ + ] The provided tests still run.
  • [ + ] I've created new tests where needed.
  • [ N/A ] I've updated the documentation if necessary.

Motivation and Context

This PR optimises metric emission, which results in higher throughput and lower impact on running system.

Description

scope of locking is now reduced to only obtain/modify current value of PromHistogram.subHistograms / PromSummary.subSummaries
If subHistogram/subSummaries for given labels is not yet added to the corresponding PromHistogram / PromSummary, the lock will be held twice: to get current value and to re-check + store new value.

  • Created tests to access/update Summary/Histogram from multiple threads
  • Ran TSAN against the change to check for concurrency issues.

return histogram
let subHistograms = lock.withLock { self.subHistograms }
if let histogram = subHistograms[labels] {
if histogram.name == self.name, histogram.help == self.help {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't believe this can ever fail should we assert/precondition instead?

self.subHistograms[labels] = newHistogram
return newHistogram
return lock.withLock {
if let histogram = subHistograms[labels], histogram.name == self.name, histogram.help == self.help {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above we fatalError if name and help don't match up. Do we want to do the same here?

fileprivate func getOrCreateSummary(withLabels labels: Labels) -> PromSummary<NumType, Labels> {
let subSummaries = self.lock.withLock { self.subSummaries }
if let summary = subSummaries[labels] {
if summary.name == self.name, summary.help == self.help {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be an assert/precondition as above?

@avolokhov avolokhov force-pushed the optimise_existing_labels_check branch from ddeb19b to 969f235 Compare August 25, 2021 17:27
@MrLotU MrLotU added this to the V 1.0.0 Alpha 15 milestone Aug 31, 2021
Copy link
Collaborator

@MrLotU MrLotU left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for the PR 😄

@MrLotU MrLotU merged commit ba329da into swift-server:master Aug 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants