Skip to content

Commit 06b7ba6

Browse files
jayhackcodegen-bot
and
codegen-bot
authored
feat: fixes codegen run and codegen create (#107)
# Motivation <!-- Why is this change necessary? --> # Content <!-- Please include a summary of the change --> # Testing <!-- How was the change tested? --> # Please check the following before marking your PR as ready for review - [ ] I have added tests for my changes - [ ] I have updated the documentation or added new documentation as needed - [ ] I have read and agree to the [Contributor License Agreement](../CLA.md) --------- Co-authored-by: codegen-bot <[email protected]>
1 parent 0fafffd commit 06b7ba6

21 files changed

+758
-373
lines changed

docs/blog/act-via-code.mdx

Lines changed: 46 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,22 @@
22
title: "Act via Code"
33
icon: "code"
44
iconType: "solid"
5-
description: "The path to advanced code manipulation agents"
5+
description: "The path to fully-automated software engineering"
66
---
77

88
<Frame caption="Voyager (2023) solved agentic tasks with code execution">
9-
<img src="/images/nether-portal.png" />
9+
<img src="/images/mine-amethyst.png" />
1010
</Frame>
1111

1212

13-
Two and a half years since the launch of the GPT-3 API, code assistants have emerged as potentially the premier use case of LLMs. The rapid adoption of AI-powered IDEs and prototype builders isn't surprising — code is structured, deterministic, and rich with patterns, making it an ideal domain for machine learning. Experienced developers working with tools like Cursor (myself included) can tell that the field of software engineering is about to go through rapid change.
13+
Two and a half years since the launch of the GPT-3 API, code assistants have emerged as potentially the premier use case of LLMs. The rapid adoption of AI-powered IDEs and prototype builders isn't surprising — code is structured, deterministic, and rich with patterns, making it an ideal domain for machine learning. Developers actively working with tools like Cursor (myself included) have an exhiliarating yet uncertain sense that the field of software engineering is approaching an inflection point.
1414

15-
Yet there's a striking gap between understanding and action. Today's AI agents can analyze enterprise codebases and propose sophisticated improvements—eliminating tech debt, untangling dependencies, improving modularity. But ask them to actually implement these changes across millions of lines of code, and they hit a wall. Their ceiling isn't intelligence—it's the ability to safely and reliably execute large-scale modifications on real, enterprise codebases.
15+
Yet there's a striking gap between understanding and action for today's code assistants. When provided proper context, frontier LLMs can analyze massive enterprise codebases and propose practical paths towards sophisticated, large-scale improvements. But implementing changes that impact more than a small set of files with modern AI assistants is fundamentally infeasible. The good news is that for focused, file-level changes, we've found real success: AI-powered IDEs ([Windsurf](https://codeium.com/windsurf), [Cursor](https://www.cursor.com/)) are transforming how developers write and review code, while chat-based assistants are revolutionizing how we bootstrap and prototype new applications (via tools like [v0](https://v0.dev/), [lovable.dev](https://lovable.dev/), and [bolt.new](https://bolt.new/)).
1616

17-
The bottleneck isn't intelligence — it's tooling. By giving AI models the ability to write and execute code that modifies code, we're about to unlock an entire class of tasks that agents already understand but can't yet perform. Code execution environments represent the most expressive tool we could offer an agent—enabling composition, abstraction, and systematic manipulation of complex systems. When paired with ever-improving language models, this will unlock another step function improvement in AI capabilities.
17+
However, there's a whole class of critical engineering tasks that remain out of reach - tasks that are fundamentally programmatic and deal with codebase structure at scale. A significant amount of effort on modern engineering teams is directed towards eliminating tech debt, managing large-scale migrations, analyzing dependency graphs, enforcing type coverage across the codebase, and similar tasks that require a global view of a codebase. Today's AI assistants can fully understand these challenges and even propose solutions, but they lack the mechanisms to actually implement them. The intelligence is there, but it's trapped in your IDE's text completion window.
18+
19+
20+
The bottleneck isn't intelligence — it's tooling. The solution requires letting AI systems programmatically interact with codebases and software systems through code execution environments. Code execution environments represent the most expressive tool we could offer an agent—enabling composition, abstraction, and systematic manipulation of complex systems. By combining code execution environments with custom APIs that correspond to powerful large-scale operations, we can unlock a new set of tasks in which agents can be significant contributors. When paired with ever-improving foundation models, this will lead to a step function improvement for code assistants, enabling their application in an entirely new set of valuable tasks.
1821

1922
## Beating Minecraft with Code Execution
2023

@@ -44,7 +47,7 @@ async function chopSpruceLogs(bot) {
4447
}
4548
```
4649

47-
This approach transformed the agent's capabilities. Rather than being constrained to atomic actions like `equipItem(...)`, it could create higher-level operations like [`craftShieldWithFurnace()`](https://github.com/MineDojo/Voyager/blob/main/skill_library/trial2/skill/code/craftShieldWithFurnace.js) through composing JS APIs. Furthermore, Wang et al. implemented a memory mechanism, in which successful "action programs" could later be recalled, copied, and built upon, effectively enabling the agent to accumulate experience.
50+
This approach transformed the agent's capabilities. Rather than being constrained to atomic actions like `equipItem(...)` (this would be typical of "traditional" agent algorithms, such as ReAct), it could create higher-level operations like [craftShieldWithFurnace()](https://github.com/MineDojo/Voyager/blob/main/skill_library/trial2/skill/code/craftShieldWithFurnace.js) through composing the atomic APIs. Furthermore, Wang et al. implemented a memory mechanism, in which these successful "action programs" could later be recalled, copied, and built upon, effectively enabling the agent to accumulate experience.
4851

4952
<Frame>
5053
<img src="/images/voyager-retrieval.png" />
@@ -56,9 +59,26 @@ As the Voyager authors noted:
5659

5760
## Code is an Ideal Action Space
5861

59-
The implications of code as an action space extend far beyond gaming. This architectural insight — letting AI act through code rather than atomic commands — will lead to a step change in the capabilities of AI systems. Nowhere is this more apparent than in software engineering, where agents already understand complex transformations but lack the tools to execute them effectively.
62+
What these authors demonstrated is a fundamental insight that extends far beyond gaming. Letting AI act through code rather than atomic commands will lead to a step change in the capabilities of AI systems. Nowhere is this more apparent than in software engineering, where agents already understand complex transformations but lack the tools to execute them effectively.
63+
64+
Today's productionized code assistants operate though an interface where they can directly read/write to text files and perform other bespoke activities, like searching through file embeddings or running terminal commands.
65+
66+
In the act via code paradigm, all of these actions are expressed through writing and executing code, like the below:
67+
68+
```python
69+
# Implement `grep` via for loops and if statements
70+
for function in codebase.functions:
71+
if 'Page' in function.name:
72+
73+
# Implement systematic actions, like moving things around, through an API
74+
function.move_to_file('/pages/' + function.name + '.tsx')
75+
```
76+
77+
Provided a sufficiently comprehensive set of APIs, this paradigm has many clear advantages:
6078

61-
When an agent writes code, it gains several critical advantages over traditional atomic tools:
79+
- **API-Driven Extensibility**: Any operation that can be expressed through an API becomes accessible to the agent. This means the scope of tasks an agent can handle grows with our ability to create clean APIs for complex operations.
80+
81+
- **Programmatic Efficiency**: Many agent tasks involve systematic operations across large codebases. Expressing these as programs rather than individual commands dramatically reduces computational overhead and allows for batch operations.
6282

6383
- **Composability**: Agents can build their own tools by combining simpler operations. This aligns perfectly with LLMs' demonstrated ability to compose and interpolate between examples to create novel solutions.
6484

@@ -68,17 +88,29 @@ When an agent writes code, it gains several critical advantages over traditional
6888

6989
- **Natural Collaboration**: Programs are a shared language between humans and agents. Code explicitly encodes reasoning in a reviewable format, making actions transparent, debuggable, and easily re-runnable.
7090

71-
## For Software Engineering
91+
## Code Manipulation Programs
7292

73-
Software engineering tasks are inherently programmatic and graph-based — dependency analysis, refactors, control flow analysis, etc. Yet today's AI agents interface with code primarily through string manipulation, missing the rich structure that developers and their tools rely on. By giving agents APIs that operate on the codebase's underlying graph structure rather than raw text, we can unlock a new tier of capabilities. Imagine agents that can rapidly traverse dependency trees, analyze control flow, and perform complex refactors while maintaining perfect awareness of the codebase's structure.
93+
For software engineering, we believe the path forward is clear: agents need a framework that matches how developers think about and manipulate code. While decades of static analysis work gives us a strong foundation, traditional code modification frameworks weren't designed with AI-human collaboration in mind - they expose low-level APIs that don't match how developers (or AI systems) think about code changes.
7494

75-
Consider how a developer thinks about refactoring: it's rarely about direct text manipulation. Instead, we think in terms of high-level operations: "move this function," "rename this variable everywhere," "split this module." These operations can be encoded into a powerful Python API:
95+
We're building a framework with high-level APIs that correspond to how engineers actually think about code modifications. The APIs are clean and intuitive, following clear [principles](/docs/principles) that eliminate sharp edges and handle edge cases automatically. Most importantly, the framework encodes rich structural understanding of code. Consider this example:
7696

7797
```python
78-
# simple access to high-level code constructs
98+
# Access to high-level semantic operations
7999
for component in codebase.jsx_components:
80-
# access detailed code structure and relations
100+
# Rich structural analysis built-in
81101
if len(component.usages) == 0:
82-
# powerful edit APIs that handle edge cases
102+
# Systematic operations across the codebase
83103
component.rename(component.name + 'Page')
84104
```
105+
106+
This isn't just string manipulation - the framework understands React component relationships, tracks usage patterns, and can perform complex refactors while maintaining correctness. By keeping the codebase representation in memory, we can provide lightning-fast operations for both analysis and systematic edits.
107+
108+
The documentation for such a framework isn't just API reference - it's education for advanced intelligence about how to successfully manipulate code at scale. We're building for a future where AI systems are significant contributors to codebases, and they need to understand not just the "how" but the "why" behind code manipulation patterns.
109+
110+
Crucially, we believe these APIs will extend beyond the codebase itself into the broader software engineering ecosystem. When agents can seamlessly interact with tools like Datadog, AWS, and other development platforms through the same clean interfaces, we'll take a major step toward [autonomous software engineering](/about#our-mission). The highest leverage move isn't just giving agents the ability to modify code - it's giving them programmatic access to the entire software development lifecycle.
111+
112+
## Codegen is now OSS
113+
114+
We're excited to release [Codegen](https://github.com/codegen-sh/codegen-sdk) as open source [Apache 2.0](https://github.com/codegen-sh/codegen-sdk?tab=Apache-2.0-1-ov-file) and build out this vision with the broader developer community. [Get started with Codegen](/introduction/getting-started) today or please join us in our [Slack community](https://community.codegen.com) if you have feedback or questions about a use case!
115+
116+
Jay Hack, Founder

docs/blog/posts.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@ iconType: "solid"
1111
Why code as an action space will lead to a step function improvement in agent capabilities.
1212

1313
<Card
14-
img="/images/nether-portal.png"
15-
title="Act via Code"
14+
img="/images/mine-amethyst.png"
15+
title="Voyager (2023) solved Minecraft with code execution"
1616
href="/blog/act-via-code"
1717
/>
1818

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
---
2+
title: "The .codegen Directory"
3+
sidebarTitle: ".codegen Directory"
4+
icon: "folder"
5+
iconType: "solid"
6+
---
7+
8+
The `.codegen` directory contains your project's Codegen configuration, codemods, and supporting files. It's automatically created when you run `codegen init`.
9+
10+
## Directory Structure
11+
12+
```bash
13+
.codegen/
14+
├── config.toml # Project configuration
15+
├── codemods/ # Your codemod implementations
16+
├── jupyter/ # Jupyter notebooks for exploration
17+
├── docs/ # API documentation
18+
├── examples/ # Example code
19+
└── prompts/ # AI system prompts
20+
```
21+
22+
## Initialization
23+
24+
The directory is created and managed using the `codegen init` command:
25+
26+
```bash
27+
codegen init [--fetch-docs] [--repo-name NAME] [--organization-name ORG]
28+
```
29+
30+
<Note>
31+
The `--fetch-docs` flag downloads API documentation and examples specific to your project's programming language.
32+
</Note>
33+
34+
### Configuration
35+
36+
The `config.toml` file stores your project settings:
37+
38+
```toml
39+
organization_name = "your-org"
40+
repo_name = "your-repo"
41+
programming_language = "python" # or other supported language
42+
```
43+
44+
This configuration is used by Codegen to provide language-specific features and proper repository context.
45+
46+
## Git Integration
47+
48+
Codegen automatically adds appropriate entries to your `.gitignore`:
49+
50+
```gitignore
51+
# Codegen
52+
.codegen/prompts/
53+
.codegen/docs/
54+
.codegen/examples/
55+
```
56+
57+
<Info>
58+
While prompts, docs, and examples are ignored, your codemods in `.codegen/codemods/` are tracked in Git.
59+
</Info>
60+
61+
## Working with Codemods
62+
63+
The `codemods/` directory is where your transformation functions live. You can create new codemods using:
64+
65+
```bash
66+
codegen create my-codemod [--description "what it does"]
67+
```
68+
69+
This will:
70+
1. Create a new file in `.codegen/codemods/`
71+
2. Generate a system prompt in `.codegen/prompts/` (if using `--description`)
72+
3. Set up the necessary imports and decorators
73+
74+
<Tip>
75+
Use `codegen list` to see all codemods in your project.
76+
</Tip>
77+
78+
## Jupyter Integration
79+
80+
The `jupyter/` directory contains notebooks for interactive development:
81+
82+
```python
83+
from codegen import Codebase
84+
85+
# Initialize codebase
86+
codebase = Codebase('../../')
87+
88+
# Print stats
89+
print(f"📚 Total Files: {len(codebase.files)}")
90+
print(f"⚡ Total Functions: {len(codebase.functions)}")
91+
```
92+
93+
<Note>
94+
A default notebook is created during initialization to help you explore your codebase.
95+
</Note>
96+
97+
## Next Steps
98+
99+
After initializing your `.codegen` directory:
100+
101+
1. Create your first codemod:
102+
```bash
103+
codegen create my-codemod -d "describe what you want to do"
104+
```
105+
106+
2. Run it:
107+
```bash
108+
codegen run my-codemod --apply-local
109+
```
110+
111+
3. Deploy it for team use:
112+
```bash
113+
codegen deploy my-codemod
114+
```

docs/building-with-codegen/files-and-directories.mdx

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -7,19 +7,19 @@ iconType: "solid"
77

88
Codegen provides two primary abstractions for working with your codebase's file structure:
99

10-
- [`File`](../api-reference/core/File)
11-
- [`Directory`](../api-reference/core/Directory)
10+
- [File](/api-reference/core/File)
11+
- [Directory](/api-reference/core/Directory)
1212

1313
Both of these expose a rich API for accessing and manipulating their contents.
1414

1515
This guide explains how to effectively use these classes to manage your codebase.
1616

1717
## Accessing Files and Directories
1818

19-
You typically access files from the [`Codebase`](/api-reference/core/Codebase) object with two APIs:
19+
You typically access files from the [codebase](/api-reference/core/Codebase) object with two APIs:
2020

21-
- [`Codebase.get_file(...)`](../api-reference/core/Codebase#get_file) - Get a file by its path
22-
- [`Codebase.files`](../api-reference/core/Codebase#files) - Enables iteration over all files in the codebase
21+
- [codebase.get_file(...)](/api-reference/core/Codebase#get_file) - Get a file by its path
22+
- [codebase.files](/api-reference/core/Codebase#files) - Enables iteration over all files in the codebase
2323

2424
```python
2525
# Get a file from the codebase
@@ -33,7 +33,7 @@ for file in codebase.files:
3333
exists = codebase.has_file("path/to/file.py")
3434
```
3535

36-
These APIs are similar for [`Directory`](../api-reference/core/Directory), which provides similar methods for accessing files and subdirectories.
36+
These APIs are similar for [Directory](/api-reference/core/Directory), which provides similar methods for accessing files and subdirectories.
3737

3838
```python
3939
# Get a directory
@@ -74,7 +74,7 @@ docs = codebase.files(extensions=[".md", ".mdx"])
7474
config_files = codebase.files(extensions=[".json", ".yaml", ".toml"])
7575
```
7676

77-
These APIs are similar for [`Directory`](../api-reference/core/Directory), which provides similar methods for accessing files and subdirectories.
77+
These APIs are similar for [Directory](/api-reference/core/Directory), which provides similar methods for accessing files and subdirectories.
7878

7979
## Raw Content and Metadata
8080

@@ -102,10 +102,10 @@ Files and Directories provide several APIs for accessing and iterating over thei
102102

103103
See, for example:
104104

105-
- `.functions` ([`File`](../api-reference/core/File#functions) / [`Directory`](../api-reference/core/Directory#functions)) - All [`Functions`](../api-reference/core/Function) in the file/directory
106-
- `.classes` ([`File`](../api-reference/core/File#classes) / [`Directory`](../api-reference/core/Directory#classes)) - All [`Classes`](../api-reference/core/Class) in the file/directory
107-
- `.imports` ([`File`](../api-reference/core/File#imports) / [`Directory`](../api-reference/core/Directory#imports)) - All [`Imports`](../api-reference/core/Import) in the file/directory
108-
- [`File.code_block`](../api-reference/core/File#code-block) - The top-level [`CodeBlock`](../api-reference/core/CodeBlock) containing the file's statements
105+
- `.functions` ([File](/api-reference/core/File#functions) / [Directory](/api-reference/core/Directory#functions)) - All [Functions](../api-reference/core/Function) in the file/directory
106+
- `.classes` ([File](/api-reference/core/File#classes) / [Directory](/api-reference/core/Directory#classes)) - All [Classes](../api-reference/core/Class) in the file/directory
107+
- `.imports` ([File](/api-reference/core/File#imports) / [Directory](/api-reference/core/Directory#imports)) - All [Imports](../api-reference/core/Import) in the file/directory
108+
109109

110110
```python
111111
# Get all functions in a file
@@ -120,10 +120,10 @@ for cls in file.classes:
120120
print(f"Methods: {[m.name for m in cls.methods]}")
121121
print(f"Attributes: {[a.name for a in cls.attributes]}")
122122

123-
# Get imports
124-
for import_stmt in file.import_statements:
125-
print(f"Import from: {import_stmt.module}")
126-
print(f"Imported symbols: {[s.name for s in import_stmt.symbols]}")
123+
# Get imports (can also do `file.import_statements`)
124+
for imp in file.imports:
125+
print(f"Import from: {imp.module}")
126+
print(f"Imported symbol: {[s.name for s in imp.imported_symbol]}")
127127

128128
# Get specific symbols
129129
main_function = file.get_function("main")

docs/building-with-codegen/parsing-codebases.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: "Parsing Codebases"
3-
sidebarTitle: "Initialization"
3+
sidebarTitle: "Parsing Codebases"
44
icon: "power-off"
55
iconType: "solid"
66
---

0 commit comments

Comments
 (0)