Updated the DatabricksRM class to use Databricks service principals with REST API path

[willsmithDB]

This PR is to add service principal support with the Databricks REST API path for the databricks retriever in DSPy. This is mainly the path for work outside of Databricks environments (i.e VS Code with Databricks Connect) where the Databricks SDK is not automatically installed or if the REST API is preferred.

• Updated logic for the _databricks_sdk_installed flag to not fail catastrophically when working locally without the package installed.
• Updated the base class to include the optional parameters.
• Updated print statements for which auth method is invoked.
• Modified parameter checks to not incorrectly trigger an exception when service principal parameters exist, but SDK is not installed and the personal access token / endpoint are missing.
• Updated docstrings.
• Updated logic for databricks token to use oauth token with SP when using the REST API with client secret and client id.
• Updated logic on the _query_via_databricks_sdk to use the SP credentials if they exist otherwise will fallback to PAT.

Note: I synced the fork with the main branch which added a number of commits.

Screenshots of REST API with SP tests:

[chenmoneygithub]

Thanks for the PR!

I feel this is going too complex with supporting all Databricks auth methods. We should move this implementation into dastabricks-ai-bridge IMO. I will discuss with the team and make a decision.

[willsmithDB]

@chenmoneygithub ,

Pardon me for the extra PR. This was supposed to be included with my other PR to add the same SP support for both methods that were already present (SDK and REST API). It's the same mechanism, but inadvertently developed for the SDK before the API when they should've been added together if that makes sense.

Thank you for your thoughts on this!