Fix BedrockAugmentedLLM config and generate_structured() #253
Conversation
…e content structure for instructor client
WalkthroughThe changes update error handling in the Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Orchestrator
participant BedrockAugmentedLLM
participant BedrockClient
User->>Orchestrator: Initiate task (generate/generate_structured)
Orchestrator->>BedrockAugmentedLLM: Request completion/structured completion
BedrockAugmentedLLM->>BedrockClient: Send request with corrected region and message format
BedrockClient-->>BedrockAugmentedLLM: Return response
BedrockAugmentedLLM-->>Orchestrator: Return processed response
Orchestrator-->>User: Deliver result
Assessment against linked issues
Poem
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
src/mcp_agent/workflows/llm/augmented_llm_bedrock.py(4 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
src/mcp_agent/workflows/llm/augmented_llm_bedrock.py (2)
src/mcp_agent/app.py (2)
logger(164-168)config(128-129)src/mcp_agent/logging/logger.py (1)
error(135-143)
🔇 Additional comments (3)
src/mcp_agent/workflows/llm/augmented_llm_bedrock.py (3)
72-78: Excellent error handling for missing Bedrock configuration.This early validation prevents runtime errors and provides clear feedback when the required Bedrock configuration is missing. The combination of logging and raising a descriptive ValueError follows best practices.
423-423: Consistent AWS region configuration fix.Good consistency with the previous fix at line 387. Both completion task methods now correctly access the AWS region configuration.
434-434:✅ Verification successful
Message format simplification looks good.
The change from nested text structure to direct string content simplifies the message format for structured completion. This likely fixes compatibility with the Bedrock API or instructor library expectations.
🌐 Web query:
What is the expected message format for the instructor library when using instructor.from_bedrock() with Bedrock's chat completions API?💡 Result:
The instructor library, when used with AWS Bedrock via
instructor.from_bedrock(), follows a specific message format for the chat completions API. This format enables structured outputs by integrating Pydantic models with Bedrock's foundation models.Message Format Structure
When using instructor with Bedrock's chat completions API, you'll need to structure your messages as a list of dictionaries, with each dictionary representing a message in the conversation. The basic structure includes:
messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Extract: Jason is 25 years old"}, ]Each message dictionary contains two key components:
- A
rolefield that can be "system", "user", or "assistant"- A
contentfield that contains the actual message textWorking with Bedrock Models
When using instructor with Bedrock, the implementation follows this pattern:
import boto3 import instructor from pydantic import BaseModel # Initialize the Bedrock client bedrock_client = boto3.client('bedrock-runtime') # Enable instructor patches for Bedrock client client = instructor.from_bedrock(bedrock_client) class User(BaseModel): name: str age: int # Create structured output user = client.chat.completions.create( modelId="anthropic.claude-3-sonnet-20240229-v1:0", messages=[ {"role": "user", "content": [{"text": "Extract: Jason is 25 years old"}]}, ], response_model=User, )Note that with some models like Claude, the content field can take a list of content objects with a
textkey[4].Response Structure
When using instructor with Bedrock, the response will be a Pydantic object instance of your defined model rather than the raw API response. For example, the above code would return a
Userobject withname="Jason"andage=25[4].Behind the scenes, instructor is managing:
- Sending your messages to the model
- Adding special instructions to extract structured data
- Parsing the response into your Pydantic model
- Validating that the response matches your model's schema
Advanced Usage
For more complex scenarios, you can incorporate system messages to provide context:
client.chat.completions.create( modelId="anthropic.claude-3-sonnet-20240229-v1:0", messages=[ {"role": "system", "content": "Extract structured information based on the user's request."}, {"role": "user", "content": "Create a user for a model with a name, age, and properties."}, ], response_model=User, )This format aligns with how the Anthropic Claude models work in Bedrock, allowing you to leverage structured outputs while maintaining the familiar chat completion interface[6].
Citations:
- 1: https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-text-completion.html
- 2: https://philschmid.github.io/easyllm/examples/bedrock-chat-completion-api/
- 3: https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html
- 4: https://python.useinstructor.com/integrations/bedrock/
- 5: https://docs.litellm.ai/docs/providers/bedrock
- 6: https://python.useinstructor.com/integrations/anthropic/
- 7: https://neo4j.com/apoc/5/ml/bedrock/
- 8: https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-claude.html
- 9: Proposal for new Instructor API ahead of 1.0 567-labs/instructor#544
- 10: https://haystack.deepset.ai/integrations/instructor-embedder
Approve message format update
The instructor.from_bedrock chat completions API accepts messages as a list of
{role: string, content: string}dictionaries. Switching from a nested text-object list to a direct string aligns with the documented contract and maintains compatibility with structured outputs. No further changes needed.
Fixes #247
Summary by CodeRabbit