AI Code Evaluator

01 The Problem

Large language models can now generate code that looks correct — but appearances are deceiving. A generated function might pass basic tests yet fail catastrophically on edge cases, exhibit quadratic complexity where linear suffices, or lack the documentation and type safety needed for production use. Manual review of AI-generated code doesn't scale, and simple pass/fail testing misses the nuance that separates a toy solution from a shippable one.

The fundamental challenge is multi-dimensional evaluation. A code submission can be: correct but slow, fast but undocumented, well-structured but fragile on edge inputs. Existing tools like Labelbox and Scale AI pioneered automated evaluation for data-labeling quality — but the code domain demanded its own framework. The AI Code Evaluator fills this gap: a systematic, weighted scoring engine that measures AI-generated code across four orthogonal axes and produces a single interpretable quality score.

02 Architecture

The framework is designed as a modular pipeline: test cases are defined in YAML, the scoring engine runs each submission through all four dimensions, and results are aggregated into a 0–100 score. Each dimension is independently configurable, allowing different problem domains to re-weight criteria without modifying core logic.

Evaluator Engine
├── Correctness (40%) — Standard test cases
├── Edge Cases    (25%) — Boundaries, empty, large inputs
├── Performance   (20%) — Execution speed, algorithmic efficiency
└── Code Quality  (15%) — Docstrings, type hints, naming, structure
    │
    ▼
Benchmark Runner → Test Cases + Scoring Engine
    │
    ▼
Report Generator → JSON + Markdown
    
CI: GitHub Actions (automated runs)

03 Tech Stack

The stack prioritises portability, automation, and clear output. YAML keeps test-case definitions human-readable and version-controllable; GitHub Actions makes evaluation a push-button CI process; and dual-format reporting (JSON + Markdown) serves both automated consumers and human reviewers.

04 Key Challenges

🎲 1. AI Code Is Fundamentally Unpredictable

Human-written code follows consistent style patterns; AI-generated code varies wildly between models, prompts, and even runs with identical inputs. A framework that assumes standard indentation, conventional naming, or idiomatic Python will penalise valid-but-unusual solutions. The evaluator must be model-agnostic: it scores the functional properties of the code (does it run? is it fast? is it safe?) without assuming how it was produced. This meant designing the Correctness and Edge Case dimensions to reason about runtime behaviour rather than syntax, and weighting style heuristics conservatively.

🏬 2. Automatic Edge-Case Generation

A robust evaluation must test boundary conditions: empty inputs, negative numbers, Unicode strings, maximum recursion depth, concurrent calls. Hand-writing these for every problem doesn't scale. The framework includes an edge-case generator that analyses the function signature and type hints to produce a combinatorial suite of boundary inputs per problem domain. For example, a sort function receives an empty list, a single-element list, a reversed list, a list of duplicates, a list of mixed types (if untyped), and a list at maximum recursion depth. The generator is parameterised so domain experts can extend it with domain-specific cases.

📊 3. Performance Isolation

Measuring algorithm speed in Python is noisy. Garbage collection, OS scheduling, CPU throttling, and even the phase of the moon can skew timing results by 10–30%. The framework addresses this with statistical significance: each performance test runs multiple iterations (default 10), discards the fastest and slowest, takes the median, and compares against a pre-recorded baseline. A submission is only penalised if the 95% confidence interval of its execution time lies strictly above the baseline. This approach eliminates most noise while catching genuine inefficiency — for example, a solution using list.insert(0, ...) repeatedly instead of a collections.deque.

📜 4. Automated Code-Quality Heuristics

Some aspects of code quality are straightforward to measure: presence of docstrings, type hints, meaningful identifiers, cyclomatic complexity. But design patterns — proper use of abstractions, separation of concerns, composability — resist automated measurement. The framework uses a heuristic scoring layer that checks for AST-level patterns (function length < 40 lines, branching factor < 8, consistent return types) and awards partial credit based on a weighted rubric. These heuristics are transparent and configurable per problem, and the scoring rationale is included in the Markdown report so a human reviewer can override if needed.

def evaluate_solution(
    problem_name: str,
    code: str,
    func_name: str,
    test_cases: list[dict],
    edge_cases: list[dict] | None = None,
    time_limit: float = 2.0,
) -> EvaluationResult:
    result = EvaluationResult(problem_name)

    # Dimension 1: Code Quality (static analysis)
    result.code_quality = analyze_code_quality(code)

    # Execute and test
    try:
        exec_globals = {}
        exec(compile(code, '<string>', 'exec'), exec_globals)
        fn = exec_globals[func_name]

        # Dimension 2: Correctness — standard test cases
        result.correctness = run_tests(fn, test_cases, time_limit)
        # Dimension 3: Edge cases — boundary conditions
        result.edge_cases = run_tests(fn, edge_cases or [], time_limit)
        # Dimension 4: Performance — timing analysis
        result.performance = measure_performance(fn, test_cases)
    except Exception as e:
        result.error = str(e)

    # Weighted total: 40% + 25% + 20% + 15%
    result.score = (
        result.correctness * 0.40 +
        result.edge_cases  * 0.25 +
        result.performance * 0.20 +
        result.code_quality * 0.15
    )
    return result

05 Results

The AI Code Evaluator produces a single 0–100 score from four weighted dimensions, enabling side-by-side comparison of good versus flawed solutions. The dual-format output means the same evaluation run feeds both a CI badge in the project README and a detailed report attached to a PR. The architecture directly mirrors production evaluation pipelines at Labelbox and Scale AI, adapted for code rather than data-labelling quality.

• Unified scoring model — Single 0–100 score aggregating correctness, edge cases, performance, and code quality
• YAML-defined test suites — Human-readable, version-controlled, extendable per problem domain
• Automated edge-case generation — Type-driven boundary input generation with configurable pruning
• Statistical performance analysis — Multi-iteration median timing with confidence intervals eliminates noise
• Dual-format reporting — Structured JSON for dashboards and Markdown for human review
• CI-native integration — GitHub Actions badge, automated runs on push/PR, no infrastructure overhead

06 What I Learned

Building an automated code evaluator revealed just how much tacit knowledge goes into human code review — and how systematically it can be encoded. Three takeaways shaped my approach to AI evaluation infrastructure: