Add Comprehensive Codebase Analyzer

[codegen-sh]

User description

This PR adds a comprehensive codebase analyzer that provides extensive static code analysis using the Codegen SDK.

Features

The analyzer provides detailed information about:

1. Codebase Structure Analysis

   • File statistics (count, language, size)
   • Symbol tree analysis
   • Import/export analysis
   • Module organization

2. Symbol-Level Analysis

   • Function analysis (parameters, return types, complexity)
   • Class analysis (methods, attributes, inheritance)
   • Variable analysis
   • Type analysis

3. Dependency and Flow Analysis

   • Call graph generation
   • Data flow analysis
   • Control flow analysis
   • Symbol usage analysis

4. Code Quality Analysis

   • Unused code detection
   • Code duplication analysis
   • Complexity metrics
   • Style and convention analysis

5. Code Metrics

   • Monthly commits
   • Cyclomatic complexity
   • Halstead volume
   • Maintainability index

Usage

# Analyze from URL
python codebase_analyzer.py --repo-url https://github.com/username/repo

# Analyze local repository
python codebase_analyzer.py --repo-path /path/to/repo

# Generate HTML report
python codebase_analyzer.py --repo-url https://github.com/username/repo --output-format html

This implementation provides a complete solution for static code analysis that can be deployed as a server-side system, giving you maximum comprehension of your codebase structure and quality.

---

💻 View my work • About Codegen

Summary by Sourcery

Add a comprehensive codebase analyzer that provides extensive static code analysis using the Codegen SDK, enabling detailed insights into code structure, dependencies, quality, and metrics.

New Features:

• Comprehensive static code analysis system
• Detailed codebase structure analysis
• Symbol-level code analysis
• Dependency and flow tracking
• Code quality metrics
• Multiple output formats (JSON, HTML, console)

Enhancements:

• Modular analysis approach with multiple categories
• Advanced complexity and maintainability metrics
• Visualization of code analysis results

Documentation:

• Update README with new analyzer features and usage instructions
• Add comprehensive documentation for codebase analysis capabilities

---

PR Type

Enhancement, Documentation

---

Description

• Introduce a comprehensive static codebase analyzer leveraging the Codegen SDK

  • Provides detailed structure, symbol, dependency, and quality analysis
  • Supports multiple output formats: JSON, HTML, and console
  • Includes advanced metrics: complexity, maintainability, duplication, and more

• Add user documentation and usage instructions to README

• Add requirements file for dependencies

• Refactor OpenAPI client to remove external dateutil dependency

---

Changes walkthrough 📝

	Relevant files
Enhancement	codebase_analyzer.py Add comprehensive static codebase analyzer script                codebase_analyzer.py Add a new, full-featured static codebase analyzer script Implements extensive analysis: structure, symbols, dependencies, quality, metrics Supports command-line interface and multiple output formats (JSON, HTML, console) Integrates with Codegen SDK and uses rich, networkx for reporting and visualization +1866/-0
Documentation	README.md Add documentation for codebase analyzer and usage                README.md Add detailed documentation for the new codebase analyzer Describe features, installation, usage, and available analysis categories Provide example commands and requirements +83/-78
Dependencies	requirements.txt Add requirements file for analyzer dependencies                    requirements.txt Add requirements file specifying dependencies for analyzer List codegen-sdk, networkx, matplotlib, and rich +4/-0
Bug fix	api_client.py Refactor date parsing to use built-in datetime                      src/codegen/agents/client/openapi_client/api_client.py Remove dependency on dateutil for date/datetime parsing Use built-in datetime methods for deserialization Minor docstring and formatting fixes +4/-9

---

Need help?

Type /help how to ... in the comments thread for any questions about Qodo Merge usage.

Check out the documentation for more information.

Description by Korbit AI

What change is being made?

Add a comprehensive static code analysis tool named "Codebase Analyzer" using the Codegen SDK, providing detailed codebase structure, symbol-level, dependency flow, code quality, visualization, and language-specific analysis capabilities.

Why are these changes being made?

These changes are made to enable a robust and thorough analysis of code repositories, aiding developers in understanding code complexity, dependencies, and quality metrics. The Codebase Analyzer serves to facilitate better maintenance, refactoring, and optimization efforts by providing actionable insights into the codebase and supporting multiple output formats such as JSON, HTML, and console. Additionally, the removal of the dateutil dependency and streamlining of date parsing in api_client.py enhances dependency management and performance.

Is this description stale? Ask me to generate a new description by commenting /korbit-generate-pr-description

[korbit-ai]

By default, I don't review pull requests opened by bots. If you would like me to review this pull request anyway, you can request a review via the /korbit-review command in a comment.

[sourcery-ai]

Reviewer's Guide

This pull request introduces a comprehensive codebase analyzer implemented as a Python script (codebase_analyzer.py). The script utilizes the codegen-sdk to perform static analysis, parsing codebases from Git URLs or local paths. It defines a CodebaseAnalyzer class containing methods organized by analysis category (e.g., structure, symbol-level, quality, metrics). The analyzer leverages argparse for command-line arguments, rich for enhanced console output, networkx for dependency graph analysis, and includes setup for matplotlib (though visualization methods might be placeholders). Dependencies are listed in a new requirements.txt file, and the README.md has been updated to document the analyzer's features and usage.

File-Level Changes

Change	Details	Files
Added the main codebase analyzer script and its dependencies.	Created codebase_analyzer.py containing the CodebaseAnalyzer class. Implemented analysis methods leveraging codegen-sdk for various metrics. Integrated argparse for command-line interface. Utilized networkx for graph-based analysis (e.g., circular imports). Used rich for formatted console output. Added requirements.txt listing necessary libraries (codegen-sdk, networkx, matplotlib, rich).	codebase_analyzer.py requirements.txt
Updated documentation to reflect the new analyzer tool.	Replaced the previous codegen-sdk introduction with documentation for the new codebase analyzer. Detailed the features, installation steps, usage examples, and analysis categories.	README.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[codegen-sh]

Hey! 👋 I see one of the checks failed. I am on it! 🫡

💻 View my work

[codegen-sh]

I've received your interrupt. I'll pause my current actions and wait for your further instructions. How would you like me to proceed?

💻 View my work

[codegen-sh]

Hey! 👋 I see one of the checks failed. I am on it! 🫡

💻 View my work

[codegen-sh]

I see that your PR "Add Comprehensive Codebase Analyzer" is failing the pre-commit checks. Let me analyze the issues and fix them for you.

💻 View my work

[codegen-sh]

I'll continue analyzing the PR issues and fix them right away.

💻 View my work

[codegen-sh]

I've fixed the issues in your PR that were causing the pre-commit checks to fail. Here's what I did:

1. Fixed the datetime.datetime.now() call by adding a timezone parameter: datetime.datetime.now(datetime.timezone.utc)

2. Resolved the function redefinition issue by renaming the second occurrence of get_unused_imports to get_unused_imports_analysis

3. Added missing newlines at the end of README.md and requirements.txt files

4. Fixed the deptry issue by removing the dateutil import from api_client.py and updating the date parsing methods to use built-in datetime functions

I've pushed these changes to your PR branch. The pre-commit checks should now pass successfully.

Is there anything else you'd like me to help with?

💻 View my work • React 👍 or 👎

[codegen-sh]

Hey! 👋 I see one of the checks failed. I am on it! 🫡

💻 View my work

[codegen-sh]

I see that your PR for the Comprehensive Codebase Analyzer has failed the mypy type checking. I'll analyze the errors and fix them for you right away!

💻 View my work

[codegen-sh]

I noticed your interrupt message. I'll pause my current analysis. What would you like me to address or change about my approach to fixing the mypy errors in the PR?

💻 View my work • React 👍 or 👎

[korbit-ai]

By default, I don't review pull requests opened by bots. If you would like me to review this pull request anyway, you can request a review via the /korbit-review command in a comment.

[qodo-merge-pro]

PR Reviewer Guide 🔍

(Review updated until commit 503da96)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 5 🔵🔵🔵🔵🔵

🧪 No relevant tests

🔒 Security concerns Sensitive information exposure: The analyzer may inadvertently expose sensitive information from analyzed codebases in its output. When analyzing repositories, it extracts and processes all code including potential API keys, passwords, or other secrets that might be embedded in the code. The HTML report generation (lines 1696-1755) and console output (lines 1757-1797) don't have any filtering mechanism to redact or exclude sensitive information before displaying it. This could lead to accidental exposure of credentials when sharing analysis reports.

⚡ Recommended focus areas for review Error Handling The analyzer has minimal error handling for API failures. When analyzing large codebases, API calls might fail or timeout, but there's no robust retry mechanism or graceful degradation when specific analysis methods fail. def analyze(self, categories: Optional[list[str]] = None, output_format: str = "json", output_file: Optional[str] = None): """Perform a comprehensive analysis of the codebase. Args: categories: List of categories to analyze. If None, all categories are analyzed. output_format: Format of the output (json, html, console) output_file: Path to the output file Returns: Dict containing the analysis results """ if not self.codebase: msg = "Codebase not initialized. Please initialize the codebase first." raise ValueError(msg) # If no categories specified, analyze all if not categories: categories = list(METRICS_CATEGORIES.keys()) # Initialize results dictionary self.results = { "metadata": { "repo_name": self.codebase.ctx.repo_name, "analysis_time": datetime.datetime.now(datetime.timezone.utc).isoformat(), "language": str(self.codebase.ctx.programming_language), }, "categories": {}, } # Analyze each category with Progress( SpinnerColumn(), TextColumn("[bold blue]{task.description}"), BarColumn(), TextColumn("[bold green]{task.completed}/{task.total}"), TimeElapsedColumn(), ) as progress: task = progress.add_task("[bold green]Analyzing codebase...", total=len(categories)) for category in categories: if category not in METRICS_CATEGORIES: self.console.print(f"[bold yellow]Warning: Unknown category '{category}'. Skipping.[/bold yellow]") progress.update(task, advance=1) continue self.console.print(f"[bold blue]Analyzing {category}...[/bold blue]") # Get the metrics for this category metrics = METRICS_CATEGORIES[category] category_results = {} # Run each metric for metric in metrics: try: method = getattr(self, metric, None) if method and callable(method): result = method() category_results[metric] = result else: category_results[metric] = {"error": f"Method {metric} not implemented"} except Exception as e: category_results[metric] = {"error": str(e)} # Add the results to the main results dictionary self.results["categories"][category] = category_results progress.update(task, advance=1) Resource Management The temporary directory created in _init_from_url() is never cleaned up, which could lead to disk space issues over time. There's no mechanism to delete the temporary files after analysis is complete. def _init_from_url(self, repo_url: str, language: Optional[str] = None): """Initialize codebase from a repository URL.""" try: # Extract owner and repo name from URL if repo_url.endswith(".git"): repo_url = repo_url[:-4] parts = repo_url.rstrip("/").split("/") repo_name = parts[-1] owner = parts[-2] repo_full_name = f"{owner}/{repo_name}" # Create a temporary directory for cloning tmp_dir = tempfile.mkdtemp(prefix="codebase_analyzer_") # Configure the codebase config = CodebaseConfig( debug=False, allow_external=True, py_resolve_syspath=True, ) secrets = SecretsConfig() # Initialize the codebase self.console.print(f"[bold green]Initializing codebase from {repo_url}...[/bold green]") prog_lang = None if language: prog_lang = ProgrammingLanguage(language.upper()) self.codebase = Codebase.from_github(repo_full_name=repo_full_name, tmp_dir=tmp_dir, language=prog_lang, config=config, secrets=secrets, full_history=True) self.console.print(f"[bold green]Successfully initialized codebase from {repo_url}[/bold green]") except Exception as e: self.console.print(f"[bold red]Error initializing codebase from URL: {e}[/bold red]") raise Security Risk The analyzer allows arbitrary repository URLs to be cloned and analyzed without validation, which could potentially be exploited to execute malicious code during the analysis process. def _init_from_url(self, repo_url: str, language: Optional[str] = None): """Initialize codebase from a repository URL.""" try: # Extract owner and repo name from URL if repo_url.endswith(".git"): repo_url = repo_url[:-4] parts = repo_url.rstrip("/").split("/") repo_name = parts[-1] owner = parts[-2] repo_full_name = f"{owner}/{repo_name}" # Create a temporary directory for cloning tmp_dir = tempfile.mkdtemp(prefix="codebase_analyzer_") # Configure the codebase config = CodebaseConfig( debug=False, allow_external=True, py_resolve_syspath=True, ) secrets = SecretsConfig() # Initialize the codebase self.console.print(f"[bold green]Initializing codebase from {repo_url}...[/bold green]") prog_lang = None if language: prog_lang = ProgrammingLanguage(language.upper()) self.codebase = Codebase.from_github(repo_full_name=repo_full_name, tmp_dir=tmp_dir, language=prog_lang, config=config, secrets=secrets, full_history=True) self.console.print(f"[bold green]Successfully initialized codebase from {repo_url}[/bold green]") except Exception as e: self.console.print(f"[bold red]Error initializing codebase from URL: {e}[/bold red]") raise

[qodo-merge-pro]

PR Code Suggestions ✨

Latest suggestions up to 503da96
Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Handle None values Handle None values in the datetime deserialization method to prevent potential TypeError when None is passed. This is a common scenario when dealing with optional datetime fields. codebase_analyzer.py [624-635] def __deserialize_datetime(self, string): """Deserializes string to datetime. The string should be in iso8601 datetime format. :param string: str. :return: datetime. """ + if string is None: + return None try: return datetime.datetime.fromisoformat(string.replace('Z', '+00:00')) except ValueError: raise rest.ApiException(status=0, reason=(f"Failed to parse `{string}` as datetime object")) [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 7 __ Why: The suggestion correctly adds a check for None input to prevent a potential AttributeError in string.replace(), improving the robustness of the deserialization method.	Medium
	Ensure graph nodes exist Check if the imported file path exists in the dependency map before adding an edge to prevent KeyError exceptions. This can happen when an import references a file that isn't properly tracked in the codebase. codebase_analyzer.py [497-530] def get_circular_imports(self) -> list[list[str]]: """Detect circular imports in the codebase.""" files = list(self.codebase.files) dependency_map = {} # Build dependency graph for file in files: if file.is_binary: continue file_path = file.file_path imports = [] for imp in file.imports: if hasattr(imp, "imported_symbol") and imp.imported_symbol: imported_symbol = imp.imported_symbol if hasattr(imported_symbol, "file") and imported_symbol.file: imports.append(imported_symbol.file.file_path) dependency_map[file_path] = imports # Create a directed graph G = nx.DiGraph() # Add nodes and edges for file_path, imports in dependency_map.items(): G.add_node(file_path) for imp in imports: + G.add_node(imp) # Ensure the target node exists G.add_edge(file_path, imp) # Find cycles cycles = list(nx.simple_cycles(G)) return cycles Apply this suggestion Suggestion importance[1-10]: 6 __ Why: Explicitly adding the target node imp before adding the edge improves the robustness and clarity of the graph construction, potentially preventing issues if an imported file isn't added as a node during the initial loop.	Low
More

---

Previous suggestions

Suggestions up to commit 503da96

Category	Suggestion	Impact
Possible issue	Improve datetime parsing The fromisoformat method doesn't handle all ISO 8601 formats correctly, especially those with microseconds and timezone offsets. Add a fallback mechanism to handle a wider range of ISO 8601 formats to prevent parsing failures with valid datetime strings. codebase_analyzer.py [624-635] def __deserialize_datetime(self, string): """Deserializes string to datetime. The string should be in iso8601 datetime format. :param string: str. :return: datetime. """ try: return datetime.datetime.fromisoformat(string.replace('Z', '+00:00')) except ValueError: - raise rest.ApiException(status=0, reason=(f"Failed to parse `{string}` as datetime object")) + try: + # Try with different ISO 8601 formats + for fmt in ("%Y-%m-%dT%H:%M:%S", "%Y-%m-%dT%H:%M:%S.%f", "%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M:%S.%f"): + try: + return datetime.datetime.strptime(string, fmt) + except ValueError: + continue + raise ValueError(f"No matching format found for {string}") + except ValueError: + raise rest.ApiException(status=0, reason=(f"Failed to parse `{string}` as datetime object")) Suggestion importance[1-10]: 2 __ Why: The suggestion incorrectly identifies the relevant file. While parsing robustness is a valid concern, datetime.fromisoformat (used in the PR) is generally robust for standard ISO 8601 formats, including those with timezones and microseconds. The proposed strptime fallback adds complexity and may not be a significant improvement.	Low
	Fix division by zero The function doesn't handle the case where n2 (unique operands) is zero, which can lead to a division by zero error in the difficulty calculation. Add a check to handle this edge case to prevent crashes during analysis of codebases with unusual patterns. codebase_analyzer.py [1405-1431] def calculate_halstead_volume(self) -> dict[str, float]: """Calculate Halstead volume metrics.""" operators_and_operands = self.get_operators_and_operands() n1 = operators_and_operands["unique_operators"] n2 = operators_and_operands["unique_operands"] N1 = operators_and_operands["total_operators"] N2 = operators_and_operands["total_operands"] # Calculate Halstead metrics vocabulary = n1 + n2 length = N1 + N2 volume = length * math.log2(vocabulary) if vocabulary > 0 else 0 - difficulty = (n1 / 2) * (N2 / n2) if n2 > 0 else 0 + + # Prevent division by zero + if n2 > 0: + difficulty = (n1 / 2) * (N2 / n2) + else: + difficulty = 0 + effort = volume * difficulty time = effort / 18 # Time in seconds (18 is a constant from empirical studies) bugs = volume / 3000 # Estimated bugs (3000 is a constant from empirical studies) return { "vocabulary": vocabulary, "length": length, "volume": volume, "difficulty": difficulty, "effort": effort, "time": time, # in seconds "bugs": bugs, } Suggestion importance[1-10]: 1 __ Why: The suggestion identifies a potential division by zero error but fails to recognize that the existing code on line 1418 (difficulty = (n1 / 2) * (N2 / n2) if n2 > 0 else 0) already handles this case correctly using a conditional expression. The proposed improved_code is functionally equivalent and offers no improvement.	Low