[Feautre Reuqest] [Bug] Async + RAGAS

[logan-markewich]

Describe the Feature
As far as I can tell, it seems like there is a lack of async functions in ragas, which leads to a lot of nesting async issues (since sync functions are calling async functions somewhere in the stack). Furthermore, nest_asyncio.apply() is automatically called in the import stack, which breaks a lot of applications

File "/usr/local/lib/python3.10/site-packages/ragas/executor.py", line 15, in <module>
2025-01-07 23:06:08     nest_asyncio.apply()

It would be awesome to have async functions like aevaluate instead of evaluate -- the approach I might take is make everything async first, and then get/create an event loop and run it there for the sync versions to avoid code duplication.

At the very least, removing the automatic nest_asyncio.apply() might be a good idea?

Why is the feature important for you?
Async code is the backbone of production systems -- without it, ragas will block the event loop of the server

Additional context
I THINK the workaround here for now is doing something like

def run_ragas(...):
  from app.evaluation.ragas import generate_dev_set, generate_dev_set_csv
  # other ragas imports

# Then, later on in your code
result = await asyncio.to_thread(run_ragas(...))

Which delays the ragas imports and runs it behind an isolated thread, which should be safe. But obviously not ideal

[jjmachan]

hey @logan-markewich 🙂 👋🏽

like you suggested the main culprit is the

File "/usr/local/lib/python3.10/site-packages/ragas/executor.py", line 15, in <module>
2025-01-07 23:06:08     nest_asyncio.apply()

now we added this as a dirty fix for the executor thread which was responsible for handling the logic "he approach I might take is make everything async first, and then get/create an event loop and run it there for the sync versions to avoid code duplication."

I'll do a bit more experiments and testing to remove the need for nesting_asyncio call.

[jjmachan]

OpenHands started fixing the issue! You can monitor the progress here.

[jjmachan]

OpenHands started fixing the issue! You can monitor the progress here.

[jjmachan]

OpenHands started fixing the issue! You can monitor the progress here.

[richness]

This problem hasn't been solved yet at present?

[nmakhotkin]

As far as I can see, it wasn't. It still crashes for me with

ValueError: Can't patch loop of type <class 'uvloop.Loop'>

as for version 0.2.15.
I used asyncio loop implementation for a while but then I noticed that it leads to unexpected delays and slowdowns during FastAPI requests and then I switched back to uvloop.
The workaround is to run evaluate in executor like this:

loop = asyncio.get_event_loop()
results = await loop.run_in_executor(None, evaluate, dataset, metrics, ...)

But this does not work together with Google gemini models using ChatGoogleGenerativeAI as LLM: google genai throws an error claiming that the thread was attached to another loop

UPD: No, it worked earlier in 0.2.6 but in the latest version does not
UPD2": Ended up with small refactoring and making my own fork with async-only ragas without nest_asyncio at all: https://github.com/nmakhotkin/ragas

[ahgraber]

Re: removing nest_asyncio - I think the major reason it was added was to allow folks to use it in ipython/jupyter kernels? Modern ipython can just await directly rather than mucking about with nesting the asyncio loop. If that's the case, you could just detect an ipython or jupyter environment and tell the user to await directly and completely pull the nest_asyncio dependency?

# Check if we're in Jupyter/IPython
try:
    from IPython.core.getipython import get_ipython

    if get_ipython() is not None:
        raise RuntimeError(
            "In Jupyter/IPython, use `await executor.aresults()` directly; this avoids the need for nest_asyncio."
        )
except ImportError:
    pass