[Feature] SemanticF1: Consider decomposing LLM scoring from numerical P/R calculation

[komikat]

What feature would you like to see?

Hi DSPy team,

While exploring the SemanticF1 module I noticed that it asks LLMs to directly output recall: float and precision: float values. While LLMs excel at semantic understanding, they may be less reliable at generating precise, calibrated numerical scores.

Current approach:

• LLM directly outputs precision/recall floats (0-1)
• Potential for subjective or "impressionistic" scoring

Suggestion:
Consider a more decomposed approach where:

1. LLM extracts and matches key ideas/claims from both texts
2. A separate step (algorithmic or constrained LLM call) calculates P/R from these mappings

This could provide more fine-grained, reliable, and interpretable semantic evaluation.

Would love to hear the team's thoughts on the design philosophy here and whether there's interest in exploring alternative semantic evaluation approaches.

Thanks for the great work!

Would you like to contribute?

• Yes, I'd like to help implement this.
• No, I just want to request it.

Additional Context

relevant twitter thread

[okhat]

Yes that is fair!