Skip to content

AWS X-Ray Remote Sampler Part 3 - rate limiter logic and get sampling targets #55

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Feb 16, 2024
Merged

AWS X-Ray Remote Sampler Part 3 - rate limiter logic and get sampling targets #55

merged 8 commits into from
Feb 16, 2024

Conversation

jj22ee
Copy link
Contributor

@jj22ee jj22ee commented Feb 14, 2024 •
edited
Loading

Issue #, if available:
This is the 3rd and final part of the X-Ray Remote Sampler implementation for Python
See Part 2

Description of changes:

  • Added logic to fetch sampling targets for each sampling rule applier.
    • The sampling targets are periodically fetched every 10 seconds by making the GetSamplingTargets API call to X-Ray.
    • The targets determine the reservoir quota and the rate at which a sampling rule applier will sample the requests.
    • Each rule applier keeps and updates a sampling statistics document which is required in GetSamplingTargets call to determine the next target
  • Added the rate limiting and fixed rate samplers to be used in each rule applier.
    • Together these sampler determine how many requests to sample every second and what percentage of additional requests to sample in that second.
    • The FallbackSampler is a combination of above samplers to sample 1 req/sec and 5% of additional requests in that second.

Testing:
Unit Tests and Remote Sampling Testbed

Testbed:

  1. Have XRay Daemon running or OTel collector with XRay Proxy Client setup running. Ensure AWS credentials used has only default rule with 5% sampling and 1 req/s

  2. Checkout this PR Branch, and install pip3 install aws-opentelemetry-distro/

  3. Download this Python Sample App: https://github.com/jj22ee/aws-otel-community/tree/python-sample/centralized-sampling-tests/sample-apps/python-flask
    Install python3 requirements.txt
    Replace the following:

###
### Set sampler HERE
###

with:

from amazon.opentelemetry.distro.sampler.aws_xray_remote_sampler import AwsXRayRemoteSampler
xray_sampler = AwsXRayRemoteSampler(resource, polling_interval=10)
trace.set_tracer_provider(TracerProvider(sampler=xray_sampler))

Run Python Sample app with python3 app.py

  1. Download this repository: https://github.com/aws-observability/aws-otel-community/
    Within directory centralized-sampling-tests/ run: ./gradlew :integration-tests:run
    Check that the tests have passed.

TODO: wire in the remote sampler to the ADOT Python customizer in this PR or a new PR

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@jj22ee jj22ee requested review from AsakerMohd, srprash and a team February 14, 2024 02:50
@srprash srprash changed the title rate limiter logic and get sampling targets AWS X-Ray Remote Sampler Part 2 - rate limiter logic and get sampling targets Feb 14, 2024
@srprash srprash changed the title AWS X-Ray Remote Sampler Part 2 - rate limiter logic and get sampling targets AWS X-Ray Remote Sampler Part 3 - rate limiter logic and get sampling targets Feb 14, 2024
srprash
srprash reviewed Feb 14, 2024
View reviewed changes
@jj22ee jj22ee mentioned this pull request Feb 15, 2024
Merged
srprash
srprash approved these changes Feb 16, 2024
View reviewed changes
Copy link
Contributor

@srprash srprash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@jj22ee jj22ee merged commit 453a3d5 into main Feb 16, 2024
@jj22ee jj22ee deleted the remote-sampler-3 branch February 16, 2024 17:37
jj22ee added a commit that referenced this pull request Feb 16, 2024
@jj22ee
Temporarily Default to XRay Remote Sampler is Sampler is not Specified (
5ce4162 
#64)

*Issue #, if available:*
Use X-Ray default sampler as default sampler if user has not specified
one
Requires
#55
to be merged first before this PR can work.

*Description of changes:*
Use X-Ray default sampler as default sampler if user has not specified
one

*Testing:*
Assume
#55
is merged

1. Enabled span_metrics_processor and added debug statement to print out
Sampled status of a span in `on_start()`
```
# aws_span_metrics_processor.py
if span.get_span_context().trace_flags.sampled:
            print("sampled")
```
2. Replaced resource detectors and manually set resource with
service.name=test-service-name and build ADOT SDK
3. Setup OTel collector with XRay proxy for sampling and AWS credentials
4. In AWS account, create sampling rule to match
service_name=test-service-name
5. Setup sample app in
`sample-applications/simple-client-server/server_automatic_s3client.py`
6. Update `sample-applications/simple-client-server/client.py` to call
server/sample-app a variable number of times to verify sampling rule is
applied
7. Repeat 6 after changing sampling rule rate/reservoir


By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@srprash srprash srprash approved these changes

@AsakerMohd AsakerMohd Awaiting requested review from AsakerMohd

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jj22ee @srprash