Beta 9 release of Python Azure AI Projects SDK (#40280)

Author: dargilco

.vscode/cspell.json

@@ -377,6 +377,7 @@
     "prebuilts",
     "premf",
     "prevsnapshot",
+    "prompty",
     "pschema",
     "PSECRET",
     "pydantic",

sdk/ai/azure-ai-projects/CHANGELOG.md

@@ -1,5 +1,24 @@
 # Release History
 
+## 1.0.0b9 (2025-04-16)
+
+### Features added
+
+* Utilities to load prompt template strings and Prompty file content
+* Added BingCustomSearchTool class with sample
+* Added list_threads API to agents namespace
+* Added image input support for agents create_message
+
+### Sample updates
+
+* Added `project_client.agents.enable_auto_function_calls(toolset=toolset)` to all samples that has `toolcalls` executed by `azure-ai-project` SDK
+* New BingCustomSearchTool sample
+* New samples added for image input from url, file and base64
+
+### Breaking Changes
+
+Redesigned automatic function calls because agents retrieved by `update_agent` and `get_agent` do not support them.  With the new design, the toolset parameter in `create_agent` no longer executes toolcalls automatically during `create_and_process_run` or `create_stream`. To retain this behavior, call `enable_auto_function_calls` without additional changes.
+
 ## 1.0.0b8 (2025-03-28)
 
 ### Features added

sdk/ai/azure-ai-projects/MANIFEST.in

@@ -4,4 +4,4 @@ include azure/ai/projects/py.typed
 recursive-include tests *.py
 recursive-include samples *.py *.md
 include azure/__init__.py
-include azure/ai/__init__.py
+include azure/ai/__init__.py

sdk/ai/azure-ai-projects/README.md

@@ -52,6 +52,7 @@ To report an issue with the client library, or request additional features, plea
     - [Create message](#create-message) with:
       - [File search attachment](#create-message-with-file-search-attachment)
       - [Code interpreter attachment](#create-message-with-code-interpreter-attachment)
+      - [Create Message with Image Inputs](#create-message-with-image-inputs)
     - [Execute Run, Run_and_Process, or Stream](#create-run-run_and_process-or-stream)
     - [Retrieve message](#retrieve-message)
     - [Retrieve file](#retrieve-file)
@@ -280,6 +281,9 @@ toolset = ToolSet()
 toolset.add(functions)
 toolset.add(code_interpreter)
 
+# To enable tool calls executed automatically
+project_client.agents.enable_auto_function_calls(toolset=toolset)
+
 agent = project_client.agents.create_agent(
     model=os.environ["MODEL_DEPLOYMENT_NAME"],
     name="my-assistant",
@@ -290,7 +294,7 @@ agent = project_client.agents.create_agent(
 
 <!-- END SNIPPET -->
 
-Also notices that if you use asynchronous client, you use `AsyncToolSet` instead.  Additional information related to `AsyncFunctionTool` be discussed in the later sections.
+Also notice that if you use the asynchronous client, use `AsyncToolSet` instead. Additional information related to `AsyncFunctionTool` be discussed in the later sections.
 
 Here is an example to use `tools` and `tool_resources`:
 <!-- SNIPPET:sample_agents_vector_store_batch_file_search.create_agent_with_tools_and_tool_resources -->
@@ -463,11 +467,7 @@ print(conn_id)
 
 # Initialize agent AI search tool and add the search index connection id
 ai_search = AzureAISearchTool(
-    index_connection_id=conn_id,
-    index_name="sample_index",
-    query_type=AzureAISearchQueryType.SIMPLE,
-    top_k=3,
-    filter=""
+    index_connection_id=conn_id, index_name="sample_index", query_type=AzureAISearchQueryType.SIMPLE, top_k=3, filter=""
 )
 
 # Create agent with AI search tool and process assistant run
@@ -513,12 +513,7 @@ for message in messages.data:
 
 #### Create Agent with Function Call
 
-You can enhance your Agents by defining callback functions as function tools. These can be provided to `create_agent` via either the `toolset` parameter or the combination of `tools` and `tool_resources`. Here are the distinctions:
-
-- `toolset`: When using the `toolset` parameter, you provide not only the function definitions and descriptions but also their implementations. The SDK will execute these functions within `create_and_run_process` or `streaming` . These functions will be invoked based on their definitions.
-- `tools` and `tool_resources`: When using the `tools` and `tool_resources` parameters, only the function definitions and descriptions are provided to `create_agent`, without the implementations. The `Run` or `event handler of stream` will raise a `requires_action` status based on the function definitions. Your code must handle this status and call the appropriate functions.
-
-For more details about calling functions by code, refer to [`sample_agents_stream_eventhandler_with_functions.py`](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/sample_agents_stream_eventhandler_with_functions.py) and [`sample_agents_functions.py`](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/sample_agents_functions.py).
+You can enhance your Agents by defining callback functions as function tools. These can be provided to `create_agent` via either the `toolset` parameter or the combination of `tools` and `tool_resources`.
 
 For more details about requirements and specification of functions, refer to [Function Tool Specifications](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/FunctionTool.md)
 
@@ -529,6 +524,7 @@ Here is an example to use [user functions](https://github.com/Azure/azure-sdk-fo
 functions = FunctionTool(user_functions)
 toolset = ToolSet()
 toolset.add(functions)
+project_client.agents.enable_auto_function_calls(toolset=toolset)
 
 agent = project_client.agents.create_agent(
     model=os.environ["MODEL_DEPLOYMENT_NAME"],
@@ -553,6 +549,7 @@ functions = AsyncFunctionTool(user_async_functions)
 
 toolset = AsyncToolSet()
 toolset.add(functions)
+project_client.agents.enable_auto_function_calls(toolset=toolset)
 
 agent = await project_client.agents.create_agent(
     model=os.environ["MODEL_DEPLOYMENT_NAME"],
@@ -564,6 +561,9 @@ agent = await project_client.agents.create_agent(
 
 <!-- END SNIPPET -->
 
+Notice that if `enable_auto_function_calls` is called, the SDK will invoke the functions automatically during `create_and_process_run` or streaming.  If you prefer to execute them manually, refer to [`sample_agents_stream_eventhandler_with_functions.py`](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/sample_agents_stream_eventhandler_with_functions.py) or 
+[`sample_agents_functions.py`](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/agents/sample_agents_functions.py)
+
 #### Create Agent With Azure Function Call
 
 The AI agent leverages Azure Functions triggered asynchronously via Azure Storage Queues. To enable the agent to perform Azure Function calls, you must set up the corresponding `AzureFunctionTool`, specifying input and output queues as well as parameter definitions.
@@ -610,7 +610,6 @@ agent = project_client.agents.create_agent(
 
 Currently, the Azure Function integration for the AI Agent has the following limitations:
 
-- Azure Functions integration is available **only for non-streaming scenarios**.
 - Supported trigger for Azure Function is currently limited to **Queue triggers** only.
   HTTP or other trigger types and streaming responses are not supported at this time.
 
@@ -793,10 +792,17 @@ auth = OpenApiAnonymousAuthDetails()
 
 # Initialize agent OpenApi tool using the read in OpenAPI spec
 openapi_tool = OpenApiTool(
-    name="get_weather", spec=openapi_weather, description="Retrieve weather information for a location", auth=auth
+    name="get_weather",
+    spec=openapi_weather,
+    description="Retrieve weather information for a location",
+    auth=auth,
+    default_parameters=["format"],
 )
 openapi_tool.add_definition(
-    name="get_countries", spec=openapi_countries, description="Retrieve a list of countries", auth=auth
+    name="get_countries",
+    spec=openapi_countries,
+    description="Retrieve a list of countries",
+    auth=auth,
 )
 
 # Create agent with OpenApi tool and process assistant run
@@ -885,6 +891,19 @@ thread = project_client.agents.create_thread(tool_resources=file_search.resource
 ```
 
 <!-- END SNIPPET -->
+
+#### List Threads
+
+To list all threads attached to a given agent, use the list_threads API:
+
+<!-- SNIPPET:sample_agents_basics.list_threads -->
+
+```python
+threads = project_client.agents.list_threads()
+```
+
+<!-- END SNIPPET -->
+
 #### Create Message
 
 To create a message for assistant to process, you pass `user` as `role` and a question as `content`:
@@ -966,6 +985,88 @@ message = project_client.agents.create_message(
 
 <!-- END SNIPPET -->
 
+#### Create Message with Image Inputs
+
+You can send messages to Azure agents with image inputs in following ways:
+
+- **Using an image stored as a uploaded file**
+- **Using a public image accessible via URL**
+- **Using a base64 encoded image string**
+
+The following examples demonstrate each method:
+
+##### Create message using uploaded image file
+
+```python
+# Upload the local image file
+image_file = project_client.agents.upload_file_and_poll(file_path="image_file.png", purpose="assistants")
+
+# Construct content using uploaded image
+file_param = MessageImageFileParam(file_id=image_file.id, detail="high")
+content_blocks = [
+    MessageInputTextBlock(text="Hello, what is in the image?"),
+    MessageInputImageFileBlock(image_file=file_param),
+]
+
+# Create the message
+message = project_client.agents.create_message(
+    thread_id=thread.id,
+    role="user",
+    content=content_blocks
+)
+```
+
+##### Create message with an image URL input
+
+```python
+# Specify the public image URL
+image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
+
+# Create content directly referencing image URL
+url_param = MessageImageUrlParam(url=image_url, detail="high")
+content_blocks = [
+    MessageInputTextBlock(text="Hello, what is in the image?"),
+    MessageInputImageUrlBlock(image_url=url_param),
+]
+
+# Create the message
+message = project_client.agents.create_message(
+    thread_id=thread.id,
+    role="user",
+    content=content_blocks
+)
+```
+
+##### Create message with base64-encoded image input
+
+```python
+import base64
+
+def image_file_to_base64(path: str) -> str:
+    with open(path, "rb") as f:
+        return base64.b64encode(f.read()).decode("utf-8")
+
+# Convert your image file to base64 format
+image_base64 = image_file_to_base64("image_file.png")
+
+# Prepare the data URL
+img_data_url = f"data:image/png;base64,{image_base64}"
+
+# Use base64 encoded string as image URL parameter
+url_param = MessageImageUrlParam(url=img_data_url, detail="high")
+content_blocks = [
+    MessageInputTextBlock(text="Hello, what is in the image?"),
+    MessageInputImageUrlBlock(image_url=url_param),
+]
+
+# Create the message
+message = project_client.agents.create_message(
+    thread_id=thread.id,
+    role="user",
+    content=content_blocks
+)
+```
+
 #### Create Run, Run_and_Process, or Stream
 
 To process your message, you can use `create_run`, `create_and_process_run`, or `create_stream`.
@@ -988,7 +1089,7 @@ while run.status in ["queued", "in_progress", "requires_action"]:
 
 <!-- END SNIPPET -->
 
-To have the SDK poll on your behalf and call `function tools`, use the `create_and_process_run` method. Note that `function tools` will only be invoked if they are provided as `toolset` during the `create_agent` call.
+To have the SDK poll on your behalf and call `function tools`, use the `create_and_process_run` method.
 
 Here is an example: