Support for Service Principal Auth for Databricks Retrieve with DSPy

[willsmithDB]

This PR adds support for authenticating with Databricks Vector Search using service principals. Users can now provide client ID and secret credentials as an alternative to personal access tokens.

Changes

• Added support for service principal authentication in the DatabricksRM class
• Added two new optional parameters to the constructor:
  • databricks_client_id: Databricks service principal ID
  • databricks_client_secret: Databricks service principal secret
• Updated authentication logic to use service principal credentials when provided
• Added appropriate error handling and validation
• Added clear logging statements to indicate which authentication method is being used
• Updated documentation with new parameter descriptions

Implementation Details

The implementation prioritizes service principal authentication when both client credentials and tokens are provided. This follows the standard pattern where more specific authentication methods take precedence over more general ones.

Authentication now follows this logic:

• If client ID and secret are provided, use service principal authentication
• Otherwise, fall back to token-based authentication (existing behavior)

Testing

Tested with:

• Service principal authentication using client ID and secret
• Token-based authentication (ensuring backward compatibility)
• Various combinations of missing credentials to verify error handling

Documentation

Updated the docstring to include descriptions of the new parameters and their usage.

[chenmoneygithub]

thanks for the PR! The code LGTM, could you include a screenshot or something that shows this new auth path works?

[willsmithDB]

Absolutely, thank you so much for reviewing!

Are there standards for contributing tests that you would recommend for third party retrievers?

SP Flow

PAT Flow

Invalid PAT Flow

Cheers!

[chenmoneygithub]

@willsmithDB We are actually going to remove most of the 3rd party retreivers from DSPy due to the toughness of testing and maintaining. DatabricksRM will remain for a bit while longer, and probably get moved to Databricks internal repo later. Regarding testing, since it's hard to write unit test and run on CI, we just ask for proof that the change works.

[chenmoneygithub]

LGTM!