Skip to content

docs: adds code agent tutorial #382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 10, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 30 additions & 31 deletions docs/introduction/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -46,27 +46,16 @@ pip install codegen

## What can I do with Codegen?

Codegen enables you to programmatically manipulate code with scale and precision.

<Frame caption="Call graph visualization for modal/modal-client/_Client">
<iframe
width="100%"
height="500px"
scrolling="no"
src={`https://codegen.sh/embedded/graph?id=66e2e195-ceec-4935-876a-ed4cfc1731c7&zoom=0.5&targetNodeName=_Client`}
className="rounded-xl"
style={{
backgroundColor: "#15141b",
}}
></iframe>
</Frame>
<Info>
View source code on [modal/modal-client](https://github.com/modal-labs/modal-client/blob/cbac0d80dfd98588027ecd21850152776be3ab82/modal/client.py#L70). View codemod on [codegen.sh](https://www.codegen.sh/codemod/66e2e195-ceec-4935-876a-ed4cfc1731c7/public/diff)
</Info>

Common use cases include:
Codegen's simple yet powerful APIs enable a range of applications, including:

<CardGroup cols={2}>
<Card
title="Build a Code Agent"
icon="robot"
href="/tutorials/build-code-agent"
>
Create an intelligent agent that can analyze and manipulate your codebase using natural language.
</Card>
<Card
title="Visualize Your Codebase"
icon="diagram-project"
Expand All @@ -76,27 +65,37 @@ Common use cases include:
</Card>
<Card
title="Mine Codebase Data"
icon="robot"
icon="database"
href="/tutorials/training-data"
>
Create high-quality training data for fine-tuning LLMs on your codebase.
</Card>
<Card
title="Eliminate Feature Flags"
icon="flag"
href="/tutorials/manage-feature-flags"
title="Build Codemods"
icon="wand-magic-sparkles"
href="/tutorials/at-a-glance#api-migrations"
>
Add, remove, and update feature flags across your application.
</Card>
<Card
title="Organize Your Codebase"
icon="folder-tree"
href="/tutorials/organize-your-codebase"
>
Restructure files, enforce naming conventions, and improve project layout.
Create powerful code transformations to automate large-scale changes.
</Card>
</CardGroup>

See below for an example call graph visualization generated with Codegen.

<Frame caption="Call graph visualization for modal/modal-client/_Client">
<iframe
width="100%"
height="500px"
scrolling="no"
src={`https://codegen.sh/embedded/graph?id=66e2e195-ceec-4935-876a-ed4cfc1731c7&zoom=0.5&targetNodeName=_Client`}
className="rounded-xl"
style={{
backgroundColor: "#15141b",
}}
></iframe>
</Frame>
<Info>
View source code on [modal/modal-client](https://github.com/modal-labs/modal-client/blob/cbac0d80dfd98588027ecd21850152776be3ab82/modal/client.py#L70). View codemod on [codegen.sh](https://www.codegen.sh/codemod/66e2e195-ceec-4935-876a-ed4cfc1731c7/public/diff)
</Info>

## Get Started

Expand Down
5 changes: 3 additions & 2 deletions docs/mint.json
Original file line number Diff line number Diff line change
Expand Up @@ -82,9 +82,10 @@
"group": "Tutorials",
"pages": [
"tutorials/at-a-glance",
"tutorials/migrating-apis",
"tutorials/codebase-visualization",
"tutorials/build-code-agent",
"tutorials/training-data",
"tutorials/codebase-visualization",
"tutorials/migrating-apis",
"tutorials/organize-your-codebase",
"tutorials/modularity",
"tutorials/manage-feature-flags",
Expand Down
16 changes: 8 additions & 8 deletions docs/tutorials/at-a-glance.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,13 @@ Explore our tutorials to learn how to use Codegen for various code transformatio
## Featured Tutorials

<CardGroup cols={2}>
<Card
title="Build a Code Agent"
icon="robot"
href="/tutorials/build-code-agent"
>
Create an intelligent code agent with Langchain and powerful, codegen-powered tools
</Card>
<Card
title="Visualize Your Codebase"
icon="diagram-project"
Expand All @@ -19,18 +26,11 @@ Explore our tutorials to learn how to use Codegen for various code transformatio
</Card>
<Card
title="Mine Training Data"
icon="robot"
icon="database"
href="/tutorials/training-data"
>
Create high-quality training data for LLM pre-training similar to word2vec or node2vec
</Card>
<Card
title="Manage Feature Flags"
icon="flag"
href="/tutorials/manage-feature-flags"
>
Add, remove, and update feature flags across your application.
</Card>
<Card
title="Delete Dead Code"
icon="broom"
Expand Down
169 changes: 169 additions & 0 deletions docs/tutorials/build-code-agent.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
---
title: "Building a Code Agent with LangChain"
sidebarTitle: "Code Agent"
icon: "robot"
iconType: "solid"
---

This guide demonstrates how to build an intelligent code agent that can analyze and manipulate codebases using Codegen's LangChain integration.

This agent access to powerful code viewing and manipulation tools powered by Codegen, including:
- `RevealSymbolTool`: reveal all N-th degree dependencies and usages of a function
- `MoveSymbolTool`: move a symbol between files, updating all imports etc. (guaranteed correctness)
- `SemanticEditTool`: implementation of Cursor-style smart file editing
- `SemanticSearchTool`: search over an index of vector embeddings for files

<Info>View the full code in our [examples repository](https://github.com/codegen-sh/codegen-sdk/tree/develop/src/codegen/extensions/langchain)</Info>

## Step 1: Setting Up the Agent

First, let's import the necessary components and create our agent:

```python
from langchain_openai import ChatOpenAI
from codegen import Codebase
from codegen.extensions.langchain import create_codebase_agent

# Initialize codebase
codebase = Codebase.from_repo("fastapi/fastapi")

# Create the agent with GPT-4
agent = create_codebase_agent(
codebase=codebase,
model_name="gpt-4",
temperature=0,
verbose=True
)
```

The agent is initialized with:
- A Codebase instance to operate on
- An LLM (GPT-4 in this case)
- Tools for code manipulation
- A conversation memory to maintain context

## Step 2: Available Tools

The agent comes with several built-in tools for code operations:

```python
tools = [
ViewFileTool(codebase), # View file contents
ListDirectoryTool(codebase), # List directory contents
SearchTool(codebase), # Search code
EditFileTool(codebase), # Edit files
CreateFileTool(codebase), # Create new files
DeleteFileTool(codebase), # Delete files
RenameFileTool(codebase), # Rename files
MoveSymbolTool(codebase), # Move functions/classes
RevealSymbolTool(codebase), # Analyze symbol relationships
SemanticEditTool(codebase), # Make semantic edits
CommitTool(codebase), # Commit changes
]
```

Each tool provides specific capabilities to the agent, allowing it to perform complex code operations.

## Step 3: Interacting with the Agent

Let's see some examples of how to interact with the agent:

```python
# Analyze dependencies
result = agent.invoke(
{
"input": "What are the dependencies of the FastAPI class?",
"config": {"configurable": {"session_id": "demo"}}
}
)
print(result["output"])

# Find usage patterns
result = agent.invoke(
{
"input": "Show me examples of dependency injection in the codebase",
"config": {"configurable": {"session_id": "demo"}}
}
)
print(result["output"])

# Perform code analysis
result = agent.invoke(
{
"input": "What's the most complex function in terms of dependencies?",
"config": {"configurable": {"session_id": "demo"}}
}
)
print(result["output"])
```

The agent maintains conversation history, so it can reference previous queries and build context over time.

## Step 4: Code Manipulation

The agent can also perform code changes:

```python
# Move a function to a new file
result = agent.invoke(
{
"input": "Move the validate_email function to validation_utils.py",
"config": {"configurable": {"session_id": "demo"}}
}
)

# Rename a class and update all references
result = agent.invoke(
{
"input": "Rename the UserModel class to User and update all imports",
"config": {"configurable": {"session_id": "demo"}}
}
)

# Add error handling
result = agent.invoke(
{
"input": "Add proper error handling to the process_data function",
"config": {"configurable": {"session_id": "demo"}}
}
)
```

The agent will:
1. Analyze the current code state
2. Plan the necessary changes
3. Execute the changes while maintaining code correctness
4. Update all related imports and references

## Advanced Usage

### Adding Custom Tools

You can extend the agent with custom tools:

```python
from langchain.tools import BaseTool
from pydantic import BaseModel, Field

class CustomToolInput(BaseModel):
"""Input schema for custom tool."""
param: str = Field(..., description="Parameter description")

class CustomCodeTool(BaseTool):
"""A custom tool for the code agent."""
name = "custom_tool"
description = "Description of what the tool does"
args_schema = CustomToolInput

def _run(self, param: str) -> str:
# Tool implementation
return f"Processed {param}"

# Add custom tool to agent
tools.append(CustomCodeTool())
agent = create_codebase_agent(
codebase=codebase,
tools=tools,
model_name="gpt-4"
)
```
10 changes: 1 addition & 9 deletions src/codegen/extensions/langchain/tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@
class ViewFileTool(BaseTool):
"""Tool for viewing file contents and metadata."""

name: ClassVar[str] = "view_file"

Check failure on line 36 in src/codegen/extensions/langchain/tools.py

View workflow job for this annotation

GitHub Actions / mypy

error: Cannot override instance variable (previously declared on base class "BaseTool") with class variable [misc]
description: ClassVar[str] = "View the contents and metadata of a file in the codebase"

Check failure on line 37 in src/codegen/extensions/langchain/tools.py

View workflow job for this annotation

GitHub Actions / mypy

error: Cannot override instance variable (previously declared on base class "BaseTool") with class variable [misc]
args_schema: ClassVar[type[BaseModel]] = ViewFileInput

Check failure on line 38 in src/codegen/extensions/langchain/tools.py

View workflow job for this annotation

GitHub Actions / mypy

error: Cannot override instance variable (previously declared on base class "BaseTool") with class variable [misc]
codebase: Codebase = Field(exclude=True)

def __init__(self, codebase: Codebase) -> None:
Expand All @@ -56,9 +56,9 @@
class ListDirectoryTool(BaseTool):
"""Tool for listing directory contents."""

name: ClassVar[str] = "list_directory"

Check failure on line 59 in src/codegen/extensions/langchain/tools.py

View workflow job for this annotation

GitHub Actions / mypy

error: Cannot override instance variable (previously declared on base class "BaseTool") with class variable [misc]
description: ClassVar[str] = "List contents of a directory in the codebase"

Check failure on line 60 in src/codegen/extensions/langchain/tools.py

View workflow job for this annotation

GitHub Actions / mypy

error: Cannot override instance variable (previously declared on base class "BaseTool") with class variable [misc]
args_schema: ClassVar[type[BaseModel]] = ListDirectoryInput

Check failure on line 61 in src/codegen/extensions/langchain/tools.py

View workflow job for this annotation

GitHub Actions / mypy

error: Cannot override instance variable (previously declared on base class "BaseTool") with class variable [misc]
codebase: Codebase = Field(exclude=True)

def __init__(self, codebase: Codebase) -> None:
Expand All @@ -79,9 +79,9 @@
class SearchTool(BaseTool):
"""Tool for searching the codebase."""

name: ClassVar[str] = "search"

Check failure on line 82 in src/codegen/extensions/langchain/tools.py

View workflow job for this annotation

GitHub Actions / mypy

error: Cannot override instance variable (previously declared on base class "BaseTool") with class variable [misc]
description: ClassVar[str] = "Search the codebase using text search"

Check failure on line 83 in src/codegen/extensions/langchain/tools.py

View workflow job for this annotation

GitHub Actions / mypy

error: Cannot override instance variable (previously declared on base class "BaseTool") with class variable [misc]
args_schema: ClassVar[type[BaseModel]] = SearchInput

Check failure on line 84 in src/codegen/extensions/langchain/tools.py

View workflow job for this annotation

GitHub Actions / mypy

error: Cannot override instance variable (previously declared on base class "BaseTool") with class variable [misc]
codebase: Codebase = Field(exclude=True)

def __init__(self, codebase: Codebase) -> None:
Expand All @@ -102,7 +102,7 @@
class EditFileTool(BaseTool):
"""Tool for editing files."""

name: ClassVar[str] = "edit_file"

Check failure on line 105 in src/codegen/extensions/langchain/tools.py

View workflow job for this annotation

GitHub Actions / mypy

error: Cannot override instance variable (previously declared on base class "BaseTool") with class variable [misc]
description: ClassVar[str] = "Edit a file by replacing its entire content"
args_schema: ClassVar[type[BaseModel]] = EditFileInput
codebase: Codebase = Field(exclude=True)
Expand Down Expand Up @@ -205,15 +205,7 @@
collect_usages: bool = True,
) -> str:
# Find the symbol first
found_symbol = None
for file in self.codebase.files:
for symbol in file.symbols:
if symbol.name == symbol_name:
found_symbol = symbol
break
if found_symbol:
break

found_symbol = self.codebase.get_symbol(symbol_name)
result = reveal_symbol(
found_symbol,
degree,
Expand Down