Update embedders settings, hybrid search, and add tests for AI search methods
#1087
Conversation
Strift
changed the title
embedders settings, hybrid search, and add tests for AI search methods
meilisearch/index.py
Outdated
| Supported embedder sources: | ||
| - 'openAi': OpenAI embedder | ||
| - 'huggingFace': HuggingFace embedder | ||
| - 'ollama': Ollama embedder | ||
| - 'rest': REST API embedder | ||
| - 'userProvided': User-provided embedder | ||
|
|
||
| Required fields depend on the embedder source: | ||
| - 'rest' requires 'request' and 'response' fields | ||
| - 'userProvided' requires 'dimensions' field | ||
|
|
||
| Optional fields (availability depends on source): | ||
| - 'url': The URL Meilisearch contacts when querying the embedder | ||
| - 'apiKey': Authentication token for the embedder | ||
| - 'model': The model used for generating vectors | ||
| - 'documentTemplate': Template defining the data sent to the embedder | ||
| - 'documentTemplateMaxBytes': Maximum size of rendered document template | ||
| - 'dimensions': Number of dimensions in the chosen model | ||
| - 'revision': Model revision hash (only for 'huggingFace') | ||
| - 'distribution': Object with 'mean' and 'sigma' fields | ||
| - 'binaryQuantized': Boolean to convert vector dimensions to 1-bit values |
There was a problem hiding this comment.
I wonder if this docs follow any pattern to describe the fields and if this is the correct definition, can you confirm?
There was a problem hiding this comment.
It's AI-generated based on my input (the meilisearch docs). It does not follow any particular conventions.
I think it might be better to have less information here, and let the user refer to the documentation. I will remove it.
|
|
||
| if body: | ||
| for _, v in body.items(): | ||
| if "documentTemplateMaxBytes" in v and v["documentTemplateMaxBytes"] is None: |
There was a problem hiding this comment.
Is this handling done by Meili now?
There was a problem hiding this comment.
Removing it did not trigger any test failure but it might simply be untested, so I added it back to avoid any unwanted side effects
| assert "default" in response["hits"][0]["_vectors"] | ||
|
|
||
|
|
||
| def test_get_similar_documents_with_identical_vectors(empty_index): |
There was a problem hiding this comment.
If I understood it correctly, you're manually creating the vector for that given document, so you don't need to define any model in your test instance before, right?
There was a problem hiding this comment.
If you're referring to the test_get_similar_documents_with_identical_vectors test, that's correct.
I'm only creating an embedder so Meilisearch knows which embedder to use to compute the vector similarity:
# Configure the embedder
settings_update_task = index.update_embedders(
{
"default": {
"source": "userProvided",
"dimensions": 2,
}
}
)
There was a problem hiding this comment.
Ty for the review @brunoocasali, I answered your comments and made the changes
meilisearch/index.py
Outdated
| Supported embedder sources: | ||
| - 'openAi': OpenAI embedder | ||
| - 'huggingFace': HuggingFace embedder | ||
| - 'ollama': Ollama embedder | ||
| - 'rest': REST API embedder | ||
| - 'userProvided': User-provided embedder | ||
|
|
||
| Required fields depend on the embedder source: | ||
| - 'rest' requires 'request' and 'response' fields | ||
| - 'userProvided' requires 'dimensions' field | ||
|
|
||
| Optional fields (availability depends on source): | ||
| - 'url': The URL Meilisearch contacts when querying the embedder | ||
| - 'apiKey': Authentication token for the embedder | ||
| - 'model': The model used for generating vectors | ||
| - 'documentTemplate': Template defining the data sent to the embedder | ||
| - 'documentTemplateMaxBytes': Maximum size of rendered document template | ||
| - 'dimensions': Number of dimensions in the chosen model | ||
| - 'revision': Model revision hash (only for 'huggingFace') | ||
| - 'distribution': Object with 'mean' and 'sigma' fields | ||
| - 'binaryQuantized': Boolean to convert vector dimensions to 1-bit values |
There was a problem hiding this comment.
It's AI-generated based on my input (the meilisearch docs). It does not follow any particular conventions.
I think it might be better to have less information here, and let the user refer to the documentation. I will remove it.
|
|
||
| if body: | ||
| for _, v in body.items(): | ||
| if "documentTemplateMaxBytes" in v and v["documentTemplateMaxBytes"] is None: |
There was a problem hiding this comment.
Removing it did not trigger any test failure but it might simply be untested, so I added it back to avoid any unwanted side effects
Co-authored-by: Bruno Casali <[email protected]>
Strift
force-pushed
the
feat/add-embedders-settings
branch
from
bf4e7cb to
268aa4c
Compare
There was a problem hiding this comment.
Sorry, I've been busy and only had a chance to take a really quick look at this. Can we test this against some older v1 meilisearch? I'm thinking there may be some changes here that make the package incompatible with older versions, but haven't had time to test myself.
This is expected @sanders41 since this version introduced the stabilization of the AI capabilities. @curquiza it looks good to me, but I would wait for @sanders41 review before merging :) |
There was a problem hiding this comment.
@Strift let's move on with this PR it looks good enough to me, let's put it in production so users can guide us with further improvements :)
|
Caution Review failedThe pull request is closed. WalkthroughThe changes introduce a new Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Index
participant EmbeddersModel
Client->>Index: update_embedders(embedder_dict)
Index->>EmbeddersModel: Parse embedder_dict by source
EmbeddersModel-->>Index: Return structured embedder objects
Index->>Index: Serialize embedders for API
Index->>MeiliSearch API: PATCH /indexes/:uid/settings/embedders
Client->>Index: get_embedders()
Index->>MeiliSearch API: GET /indexes/:uid/settings/embedders
MeiliSearch API-->>Index: Return embedder configs
Index->>EmbeddersModel: Parse configs into objects
EmbeddersModel-->>Index: Return structured embedders
Index-->>Client: Return embedders object
sequenceDiagram
participant Client
participant Index
Client->>Index: search(query, {hybrid: {semanticRatio, embedder}})
Index->>MeiliSearch API: POST /indexes/:uid/search (with hybrid params)
MeiliSearch API-->>Index: Return results with semanticHitCount, etc.
Index-->>Client: Return search results
Assessment against linked issues
Poem
Note ⚡️ AI Code Reviews for VS Code, Cursor, WindsurfCodeRabbit now has a plugin for VS Code, Cursor and Windsurf. This brings AI code reviews directly in the code editor. Each commit is reviewed immediately, finding bugs before the PR is raised. Seamless context handoff to your AI code agent ensures that you can easily incorporate review feedback. Note ⚡️ Faster reviews with cachingCodeRabbit now supports caching for code and dependencies, helping speed up reviews. This means quicker feedback, reduced wait times, and a smoother review experience overall. Cached data is encrypted and stored securely. This feature will be automatically enabled for all accounts on May 16th. To opt out, configure 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (8)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Pull Request
Related issue
Requires #1086
Fixes #1081
What does this PR do?
Update settings to handle embedders
Docs: https://www.meilisearch.com/docs/reference/api/settings#embedders
Add
embedderssetting: Update methodsgetEmbedders,updateEmbedders,resetEmbedders. Also, the methodupdateSettingsshould be able to accept the newembeddersfield. Here is the list of the acceptable sub fields:sourcesub field is available and accepts:ollama,rest,openAI,huggingFaceanduserProvidedapiKeysub field is available (string) - optional because not compatible with all sources. Only foropenAi,ollama,rest.modelsub field is available (string) - optional because not compatible with all sources. Only forollama,openAI,huggingFacedocumentTemplatesub field is available (string) - optionaldimensions- optional because not compatible with all sources. Only foropenAi,huggingFace,ollama, andrestdistribution- optionalrequest- mandatory only if usingrestembedderresponse- mandatory only if usingrestembedderdocumentTemplateMaxBytes- optionalrevision- optional, only forhuggingFaceheaders- optional, only forrestbinaryQuantized- optionalReview comment:
Update search to handle vector search and hybrid search
Docs: https://www.meilisearch.com/docs/reference/api/search
Update
searchmethod:hybridsearch parameter, with sub fieldssemanticRatioandembedder.embedderis mandatory ifhybridis set.vectorparameter is availableretrieveVectorsparameter availablesemanticHitCountin search response_semanticScorein the search response (optional)_vectorsare returned in the hit objects, whenretrieveVectorsis true_vectorsNOT present in the search responseAdd similar documents endpoint
Docs: https://www.meilisearch.com/docs/reference/api/similar
PR checklist
Please check if your PR fulfills the following requirements:
Thank you so much for contributing to Meilisearch!
Summary by CodeRabbit
New Features
Documentation
Tests