[R-310] Test set generation improvements

[shahules786]

For Ragas 0.2, we released our third iteration of synthetic test generation for RAG. While developing this new approach we have kept some important feedback that we gathered from earlier versions

1. Ability to generalize over more data formats and domains.
2. Ability to customize and create scenarios related to one's own domain.
3. Ability to persist and resample Q&A from same list of documents w/o redoing preprocessing
4. Lower cost and faster iteration.

We have already noted some feedback from the community and we want to welcome more, hence this discussion thread. Please feel free share your thoughts and queries on the same. This will help us greatly improve the feature in coming weeks.

_R-310

Known issues

Feature enhancement

• Seeding test generation with queries Seed questions for guiding test generation #1642

Quality enhancement

• Testset generation feedback #1568
• Creating Testdataset from dictionary/json file #1477
• Add support for synthesising multiturn conversation by @ahgraber
• [R-312] support adding metadata into testset generation #1614

Documentation improvements

• Language adaptation for test generation
  • When I want to generate data in Chinese, this error message appears. What should I do? #1488
  • How to set the language type of the generated questions when creating a test set #1485
• Custom prompts for test generation
  • also noted in Testset generation feedback #1568
  • custom True/False query generation Can the generated dataset be filled in the blank or true/false questions #1620

TODO

• Improve quality of generated questions
• Add node filter to avoid query creation from nodes with poor information quality.
• Heading splitter - add minium and max chunk size
• Make sure default settings work in small document set (PubMed summaries,etc, <500 tokens per doc)
• Fix long context issues with extraction (OOC)
• Adjust transforms/query generation for small number of documents (n=1)
• Improve sampling algorithm to sample diverse scenarios from available scenarios.
• Simplify code for writing custom scenarios
• Add documentation for prompt adaptation in test set generation
• Add documentation for writing custom scenarios and query types, For example Yes/No questions

[shahules786]

#1621
#1618
Both these items addresses quality-related issues raised in issues like #1568

[ahgraber]

Personas and improvements to multihop look really good, I'll have to spend some time with the new multihop generation.

[shahules786]

Update on node filter and headline splitter

• Added custom node filter to remove nodes w/o good content for query creation - such as references in research papers. One could modify and their own filters
• Tuned headline splitter for better node creation

merged #1645

[shahules786]

@ahgraber with new simplified way of query generation in multi hop, I am hoping it would also be easier to create multi turn conversations. Will explore this soon.

[shahules786]

This week's updates

1. feat: improvements in default test generation #1661 make default settings work with different types of input docs
2. fixes: handle long context extraction #1680 handle long context extraction

[shahules786]

language adaptation for testgen #1734