apm: Document tail-based sampling performance

solutions/observability/apps/transaction-sampling.md

@@ -133,6 +133,24 @@ Tail-based sampling is implemented entirely in APM Server, and will work with tr
 
 Due to [OpenTelemetry tail-based sampling limitations](../../../solutions/observability/apps/limitations.md#apm-open-telemetry-tbs) when using [tailsamplingprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor), we recommend using APM Server tail-based sampling instead.
 
+### Tail-based sampling performance and requirements [_tail_based_sampling_performance_and_requirements]
+
+Tail-based sampling, by definition, requires storing events locally temporarily, such that they can be retrieved and forwarded once sampling decision is made. 
+
+In APM Server implementation, the events are stored temporarily on disk instead of memory for better scalability. Therefore, it requires local disk storage proportional to APM event ingestion rate, and additional memory to facilitate disk reads and writes. Insufficient [storage limit](../../../solutions/observability/apps/transaction-sampling.md#sampling-tail-storage_limit) causes sampling to be bypassed.
+
+It is recommended to use fast disks, for example, NVMe SSDs, when enabling tail-based sampling, as disk throughput and IO may be the performance bottleneck to tail-based sampling, and APM event ingestion as a whole. Disk writes are proportional to event ingest rate, and disk reads are proportional to event ingest rate and sampling rate.
+
+To demonstrate the performance overhead and requirements, here are some numbers from a standalone APM Server 9.0 deployed on AWS EC2, under full load receiving APM events containing only traces, assuming no backpressure from Elasticsearch, and 10% sample rate in tail sampling policy. They are for reference only, and may vary depending on factors like sampling rate, average event size, and average number of events per distributed trace.
+
+| APM Server EC2 instance size | TBS enabled, Disk                                           | Event ingestion rate (throughput from APM agent to APM Server) in events/s | Memory usage (max Resident Set Size) in GB | Disk usage in GB |
+|:-----------------------------|:------------------------------------------------------------|----------------------------------------------------------------------------|--------------------------------------------|------------------|
+| c6i.2xlarge or c6id.2xlarge  | TBS disabled                                                | 47220                                                                      | 0.98                                       | 0                |
+| ..                           | TBS enabled, gp3 volume with the baseline IOPS of 3000 IOPS | 21310                                                                      | 1.41                                       | 13.1             |
+| ..                           | TBS enabled, local NVMe SSD from c6id instance              | 21210                                                                      | 1.34                                       | 12.9             |
+| c6i.4xlarge or c6id.4xlarge  | TBS disabled                                                | 142200                                                                     | 1.12                                       | 0                |
+| ..                           | TBS enabled, gp3 volume with the baseline IOPS of 3000 IOPS | 32410                                                                      | 1.71                                       | 19.4             |
+| ..                           | TBS enabled, local NVMe SSD from c6id instance              | 47370                                                                      | 1.73                                       | 23.6             |
 
 ## Sampled data and visualizations [_sampled_data_and_visualizations]