[Feature] Add a ContextSeeker module for interactive clarification-based QA with human-in-the-loop support

[pringlesinghal]

What feature would you like to see?

Summary
In many real-world interactions with LLMs, users provide underspecified prompts—either omitting key details due to cognitive constraints or assuming shared context. Current DSPy modules are largely designed for static, self-contained queries and do not support dynamic clarification loops involving the user.

This often results in:

• LLMs hallucinating missing details and producing incorrect outputs
• Users needing to rephrase or re-prompt manually
• An inability to collect fine-grained feedback about what’s missing

We propose a general-purpose ContextSeeker module that can handle human-in-the-loop interactive QA, in which the agent explicitly asks the user for missing context before answering.

Motivation
This module aims to support workflows where:

• The query is incomplete, and the missing information exists only in the user’s head
• The agent must ask follow-up questions before answering
• Each component of the agent (question generation, answer generation, stopping criteria) should be modular and swappable, allowing for integration with existing DSPy modules (e.g., ChainOfThought, ReAct)

This problem is distinct from tool use or web search:

• In tool use, missing information can be programmatically retrieved.
• In interactive QA, only the user can provide the required detail (e.g., personal preferences, problem-specific parameters)

Design Proposal
We propose the following high-level interface:

def forward():
    context = []
    ready = stoppingAgent(prompt, context)
    while not ready:
        query = queryAgent(prompt, context)
        answer = oracle(query)  # Can be a human, simulator, or programmatic oracle
        context.append((query, answer))
        ready = stoppingAgent(prompt, context)

    answer = answerAgent(prompt, context)
    return answer

Key components:

• stoppingAgent: Determines whether enough context has been gathered to answer the question
• queryAgent: Chooses the most informative question to ask next
• oracle: Provides the answer (human, simulated user with another llm, or a simple lookup program)
• answerAgent: Produces the final answer, given the accumulated context

Each of these components can be implemented as existing or custom dspy.Modules. Any DSPy-compatible answering strategy can be used for answerAgent. Support for interactive clarification can then be layered on by implementing the queryAgent and stoppingAgent.

The oracle is flexible by design:

• In development and debugging, it can be simulated via lookup tables or scripted logic
• In real-world settings, it would be a human user providing clarifications interactively
• In evaluation, it may be another LLM with access to privileged context, simulating a human user

Ideally, each component of the ContextSeeker should be built as a flexible dspy.Module, supporting the full range of DSPy features such as tool integration and prompt optimization. In cases where the user’s original query contains sufficient information, the agent should immediately proceed to generate an answer without asking further questions. However, when key information is missing, the agent should initiate a clarification loop, asking targeted follow-up questions to fill in the gaps before attempting an answer.

The module could optionally accept a predefined list of allowed follow-up questions, which is useful in structured domains where the space of valid queries is known a priori (e.g., form-filling, medical triage, or educational assessments).

Concluding Notes
This proposal introduces a flexible and modular interface for human-in-the-loop information gathering, which we believe fits naturally within DSPy's declarative and compositional paradigm. By formalizing and supporting interactive clarification loops, DSPy can more robustly handle real-world scenarios where user intent is not fully specified upfront, significantly improving both user experience and agent reliability. We’re happy to contribute an implementation of ContextSeeker if this aligns with the core vision for DSPy.

Would you like to contribute?

• Yes, I'd like to help implement this.
• No, I just want to request it.

Additional Context

No response

[chenmoneygithub]

@pringlesinghal Thanks for opening the feature request!

The code snippet is still quite abstract, would you mind putting up a colab/github gist to explain the proposed API with a concrete use case?

[pringlesinghal]

@chenmoneygithub Thank you for your response!
I’ll put together a Colab notebook that demos the proposed module in a concrete use‑case and share it with you by Sunday, 8 June.