`evaluate` supporting replicates

[jamesbraza]

Describe the Feature

It would be nice to do something like evaluate(..., num_replicates=30) so I can calculate mean/std dev of accuracy on a benchmark EvaluationDataset.

What I mean by replicates is basically running the task N times in parallel, and computing aggregate metrics across the parallel runs.

Why is the feature important for you?

Statistical significance is important.

Additional context

I have a custom task, and am trying to compared trained models' performance on that task.

[Vinisha-projects]

Hi
I’ve implemented support for evaluate(..., num_replicates=N) as requested in this issue.

It wraps the existing evaluate() from RAGAS and runs it N times in parallel. It then returns statistical summaries like mean, std, median, min, max, and 95% confidence intervals for each metric.

Code: Vinisha-Projects/Evaluate_with_replicates-for-RAGAS

No API keys are hardcoded, and the implementation uses your existing dataset and metrics interface.

Let me know if you'd like me to submit a PR with this functionality!

— @Vinisha-Projects