[Feature] Async Context Manager Support for Settings

[jd02904]

What feature would you like to see?

It would be great to have the ability to use asyncify within dspy.settings.context(lm=model) by implementing an asynchronous version of the context manager. Currently seeing race conditions using asyncify with this context manager.

Would you like to contribute?

• Yes, I'd like to help implement this.
• No, I just want to request it.

Additional Context

No response

[chenmoneygithub]

@jd02904 Thank for reporting the issue! This shouldn't be the case, could you share a reproducible code? Thank you!

[jd02904]

@chenmoneygithub I checked the source code and it seems the way DSPy handles race conditions on settings is through thread_local_overrides but thread-local storage would only work for threads, not coroutines (asyncify). So it would make sense for there to be a potential for race conditions on settings if two different coroutines write to it. Let me know if I am missing something.

Let me try to build a minimal reproducible example but might be tricky because this only happens in production with a high number of parallel requests.

[chenmoneygithub]

For some context - asyncify builds on top of the fake streaming, which essentially uses multithreading. Our code could indeed cause an issue on native async, and I am working on the fixes. My question was to clarify which path you are using.

[okhat]

FYI -- fix from Chen at #8203