Some engineers work inside Cursor. Others live in the terminal using Aider or GitHub Copilot CLI. In high-security environments, developers are not even allowed to use cloud-based models; they run local models via Ollama or vLLM on private workstations.

For open source ecosystems like the Apache Software Foundation, this fragmentation creates a serious dilemma. If you build AI workflows for only one specific agent harness, you alienate contributors and lock your project into a proprietary vendor. If you rewrite every workflow for every tool, you end up with duplicated, brittle code that becomes impossible to maintain.

In the Apache Magpie project, an open source framework designed for security automation across Apache projects, I tackled this problem by designing and landing swappable runtime adapters for five major AI agent harnesses.

Here is how the system was built, the technical hurdles solved, and the measurable impact on open source security operations.

[1] The Fragmentation Problem in Open Source Security

The Apache Software Foundation oversees hundreds of critical open source projects, including Airflow, Kafka, Spark, and Tomcat. Companies around the world depend on these projects daily.

When a critical security vulnerability is reported in an Apache project, maintainers cannot handle it like a standard bug:

1. Strict confidentiality: Vulnerability discussions cannot happen in public GitHub issues before a patch is ready. If exploit details leak prematurely, attackers can target production systems before a fix is released.
2. Complex release governance: Handling a vulnerability requires requesting CVE numbers, preparing private patches, coordinating voting across project management committees, and drafting public security advisories.
3. High manual effort: Managing a single security advisory can take up to twenty manual review cycles.

To assist maintainers, Apache Magpie provides automated agentic workflows called skills. However, maintainers have different setups. Some wanted to run triage from Cursor. Others wanted terminal pair programming in Aider or Copilot. Security teams handling highly sensitive embargoed bugs required completely air-gapped local machines without any network traffic leaving the building.

The project faced an immediate challenge: how do we empower maintainers to use their preferred AI agent harnesses while guaranteeing absolute data confidentiality and zero vendor lock-in?

[2] The Architectural Challenge

The objective was straightforward but technically demanding:

• Maintain one canonical source of truth: Write every security skill once in clean, vendor-neutral markdown, without mentioning proprietary vendor names or hardcoding API calls.
• Support five diverse agent harnesses: Connect IDE assistants, terminal CLI tools, declarative recipe runners, and offline local inference engines.
• Guarantee strict human oversight: Ensure that no AI model can autonomously push code to protected branches, merge pull requests, or send outbound messages without explicit human approval.
• Enforce air-gapped data boundaries: Guarantee that confidential vulnerability data and private repository tokens never leak through cloud telemetry.

To solve this, we decoupled the workflow definition from the execution engine. Skills define what needs to be done. Capabilities define abstract operations like sending mail or querying advisories. Runtime adapters bind those capabilities to concrete developer harnesses.

My objective was to engineer the runtime adapters and operational guides for five distinct AI agent harnesses to achieve 100% harness neutrality.

[3] Engineering the Five Agent Harnesses

Each agent harness has its own execution model, configuration layout, and permission mechanics. I implemented custom runtime adapters tailored to each environment.

1. GitHub Copilot CLI and Coding Agent (PR #1287)

GitHub Copilot serves developers in two distinct ways: interactive command line sessions in the terminal and server-side cloud coding agents running on GitHub infrastructure.

I addressed two critical requirements for this harness:

• Establishing a strict Privacy boundary: Server-side cloud agents were explicitly forbidden from operating on private security trackers and embargoed vulnerability discussions. This ensures private exploit code is never sent to external cloud model servers.
• Enforcing human confirmation: In the terminal, all shell commands require interactive operator approval before execution. On GitHub servers, the agent is restricted strictly to creating draft pull requests; it has zero permission to push to protected branches or merge changes autonomously.

2. Aider Terminal Pair Programming (PR #1286)

Aider is a fast, terminal-based pair programming harness that interacts directly with local git working trees.

To make Aider work reliably with dozens of complex workflows, I solved two engineering problems:

• Preventing context window flooding: Magpie contains over seventy specialized skills. Loading all skill files at startup exhausts model context windows and degrades reasoning quality. I implemented a progressive loading pattern where Aider ingests only the specific skill markdown file needed for the immediate task.
• Enforcing human-reviewed commits: By default, Aider can automatically commit file modifications to git. In security engineering, unreviewed autonomous commits are unacceptable. I configured strict rules disabling automatic and dirty commits, guaranteeing that maintainers inspect every diff before a commit is created.

3. Block Goose and Model Context Protocol (PR #1217)

Goose, developed by Block, is an extensible open source CLI agent harness that automates complex engineering tasks through recipes and extensions.

For the Goose harness, I focused on deep protocol integration:

• Native Model Context Protocol (MCP) integration: I configured native stdio MCP bridges connecting Goose directly to custom Magpie servers for mail triage, project metadata, and security tooling.
• Declarative recipe workflows: I created reusable YAML recipe templates that allow maintainers to launch multi-step security triage routines with a single terminal command.
• PreToolUse guardrails: I leveraged Goose extension hooks to intercept tool execution, giving maintainers the ability to inspect and approve operations before the agent modifies files.

4. Air-Gapped Local LLMs via Ollama, llama.cpp, and vLLM (PR #1206)

For security teams handling top-secret vulnerability disclosures, commercial cloud APIs are out of the question.

In PR #1206, I documented and calibrated the harness for 100% offline, sovereign operations:

• Universal local serving: Documented complete integration with Ollama, llama.cpp server, and vLLM exposing local OpenAI-compatible endpoints.
• Calibrating empirical model capability floors: Through rigorous evaluation testing, I established clear guidance on model sizes:
  • 70B+ parameter reasoning models are required for multi-step security triage, where the agent must evaluate exploit mechanics and follow multi-step governance rules.
  • 8B to 14B parameter models are calibrated for bounded, single-file audits, checklist verifications, and formatting tasks.
• Air-gapped workstations: Enabled maintainers to run fully capable AI assistants on isolated machines with zero internet connectivity and zero cloud telemetry.

5. Cursor Composer and Agent CLI (PR #1204)

Cursor is one of the most widely used AI development harnesses, featuring both an in-editor Composer and a standalone terminal agent.

To integrate Cursor into our universal model:

• Canonical skill discovery: Mapped canonical skill directory paths so Cursor automatically discovers and recommends workflows without requiring duplicated configuration files.
• Modal approval alignment: Mapped Cursor's per-action approval modals to our proposal-then-confirm standard, ensuring the agent never applies destructive changes without consent.
• Sandboxed CLI execution: Wrapped CLI invocations in clean-environment isolation containers to prevent unintended access to local developer tokens.

[4] Production Impact and Measurable Outcomes

Landing these five agent harnesses delivered immediate, concrete benefits to the Apache Magpie framework and the wider open source community:

• 100% Harness Neutrality Score: All five harnesses passed our automated vendor neutrality test suites with a perfect 10/10 score across capability contracts and 24/24 substrate tools.
• Multi-Harness Support: Maintainers across diverse Apache projects can now execute identical security workflows from Cursor, Aider, Copilot, Goose, or fully offline local LLMs.
• Zero Leaked Credentials: By wrapping runtimes with ambient credential stripping, local API keys and cloud tokens remain completely isolated from model sessions.
• Proven Air-Gapped Feasibility: Established empirical benchmarks proving that open-weight 70B models can handle complex, multi-step vulnerability triage without sending a single byte to cloud servers.
• Complete Maintainer Governance: Strict human-in-the-loop policies guarantee that every git mutation, draft advisory, and outbound communication remains under human authority.

[5] Key Takeaways for Engineering Teams

Building adapters across five distinct agent harnesses highlighted three vital lessons for the future of software engineering:

1. Decouple workflows from harnesses: AI developer tools change constantly. If your automation relies on proprietary features from a single harness, your codebase will age poorly. Writing workflows in clean, tool-agnostic markdown protects your engineering investment.
2. Safety is more important than autonomy: Fully autonomous agents that write code, commit to git, and deploy without confirmation are a liability in production. Real-world enterprise systems require proposal-then-confirm patterns where the agent drafts and the human decides.
3. Sovereign AI is ready for production: You no longer need to compromise between powerful AI capabilities and strict confidentiality. With properly calibrated open-weight models, teams can run enterprise-grade security workflows on entirely private infrastructure.

Open source software succeeds when developers have freedom of choice. By building universal agent harnesses for Apache Magpie, we showed that AI automation can be flexible, secure, and truly vendor-neutral.

Huge thanks to Jarek Potiuk for the rigorous architectural reviews, thoughtful feedback, and continuous push for production-grade engineering quality across these PRs. He is the final boss of code reviews in Apache Magpie :)

Thanks for reading, Connect with me :)

• LinkedIn and GitHub

Refactored Microsoft PyRIT’s monolithic 1,000-line Greedy Coordinate Gradient (GCG) adversarial optimization loop into 3 typed, modular components (CandidateProposer, CandidateEvaluator, and ProgressiveScheduleController).

Bounded GPU VRAM consumption to $O(1)$ by introducing sequential candidate evaluation with immediate tensor cleanup del logits, ids), completely eliminating multi-model Out-Of-Memory (OOM) crashes.

Maintained 100% backward compatibility with zero breaking changes, supported by 300+ deterministic automated tests running across Windows, macOS, and Linux runners without requiring active GPUs.

• Tracking Issue: Closes #2665
• Merged PRs: #2671, #2700, #2720

[1] Introduction

Large language models (LLMs) are deployed with safety guardrails to prevent harmful, dangerous, or unauthorized outputs. In AI security and red teaming, testing whether these guardrails can be bypassed is an essential part of defensive engineering.

One of the most effective white-box algorithms for this is Greedy Coordinate Gradient (GCG). It searches for specific suffix tokens that, when appended to a prompt, guide the neural network into bypassing its safety constraints.

While GCG is mathematically powerful, running it reliably at scale is a difficult engineering challenge. In production red teaming frameworks, GCG must handle multiple models, diverse tokenizers, and heavy GPU tensor computations.

This article walks through how I refactored the GCG optimization engine inside Microsoft PyRIT (Python Risk Identification Tool for Generative AI) across three merged pull requests under Issue #2665, resolving critical VRAM spikes and transforming a monolithic loop into a maintainable, typed architecture.

[2] Background: How GCG Works

To understand the engineering constraints, it helps to understand how GCG operates under the hood:

1. Target Loss Formulation: The attacker specifies a target phrase (for example: "Sure, here is how to..."). The goal is to minimize the model loss on this target string.
2. Gradient Computation: The algorithm computes the gradient of the loss with respect to the one-hot token embeddings of the attack suffix.
3. Candidate Proposal: Using the gradient values, the algorithm identifies the top candidate tokens most likely to decrease the loss at each token position.
4. Candidate Evaluation: The algorithm evaluates a large batch of these candidate sequences across the target models using forward passes.
5. Selection: The candidate producing the lowest loss is selected, and the loop repeats until the model produces the target output.

In theory, this is a clean optimization loop. In practice, running hundreds of candidates across multiple neural networks quickly creates memory and architectural bottlenecks.

[3] System Bottlenecks in the Original Architecture

Before the refactoring, the optimization logic was concentrated inside two oversized methods: GCGMultiPromptAttack.step() and ProgressiveMultiPromptAttack.run().

This monolithic design introduced three major engineering problems:

1. High Coupling and Test Fragility

A single function was responsible for dispatching worker tasks across GPUs, aggregating incompatible gradient tensor shapes, sampling replacement tokens, executing model forward passes, and deciding step budgets. Because every phase was interwoven, writing fast unit tests was impossible without spinning up heavy model instances.

2. VRAM Spikes and Out-Of-Memory Crashes

Evaluating candidate tokens requires computing model logits across prompts and models. The original implementation held intermediate tensors in GPU memory throughout the step loop. During multi-prompt or multi-model runs, this caused sudden Out-Of-Memory (OOM) failures on GPU clusters.

3. Entangled Scheduling State

GCG uses a progressive curriculum: it starts on a single prompt and single model, then gradually admits more prompts, more models, and increases control weights. These state transitions were managed through nested conditionals, leading to edge cases where non-finite losses (NaN or Inf) could inadvertently trigger early exit.

[4] Architecture Redesign - Phase Extraction Strategy

Under tracking issue #2665, Microsoft maintainer Roman Lutz outlined the architectural requirements: decouple the pipeline into three independently reviewable, typed modules while maintaining 100% backward compatibility for all existing public APIs.

The refactoring was executed across three distinct engineering phases.

[5] Phase 1: Decoupling Candidate Proposal

Reference -> PR #2671

The first phase focused on extracting gradient aggregation, normalization, and candidate sampling logic out of GCGMultiPromptAttack.step().

Technical Implementation

First, I created an immutable data container using @dataclass(frozen=True, slots=True) to cleanly pass proposed candidates between modules:

@dataclass(frozen=True, slots=True)
class CandidateProposalBatch:
    """Grouped candidate control tokens and filtered text strings across compatible workers."""
    control_candidates_by_group: list[list[str]]
    group_worker_indices: list[int]

    @property
    def num_groups(self) -> int:
        return len(self.control_candidates_by_group)

Inside GCGCandidateProposer, we dispatch gradient computations across worker threads and normalize the gradients with an \(L_2\) norm so models with different loss magnitudes contribute equally:

# Pull gradient from worker and migrate to primary device
new_grad: torch.Tensor = worker.results.get().to(self._main_device)

# L2 normalization across embedding dimensions
new_grad = new_grad / new_grad.norm(dim=-1, keepdim=True)

Next came handling heterogeneous models. If an attack targets both LLaMA and Mistral simultaneously, their vocabularies and gradient tensor shapes differ. Instead of crashing, the proposer groups compatible shapes together:

# If vocabulary or tensor shape differs, flush the preceding group
if grad.shape != new_grad.shape or vocab != new_vocab:
    with torch.no_grad():
        sampled = self._sample_group(worker_idx=j - 1, gradient=grad, batch_size=batch_size, topk=topk)
        filtered = self._filter_group(worker_idx=j - 1, control_cand=sampled)
        control_cands.append(filtered)
        group_worker_indices.append(j - 1)
    grad = new_grad
    vocab = new_vocab
else:
    grad += new_grad

By extracting this into GCGCandidateProposer, we gained the ability to write 6 fast, deterministic unit tests for gradient aggregation and shape grouping without allocating real neural networks.

[6] Phase 2: VRAM-Bounded Candidate Evaluation

Reference -> PR #2700

The second phase addressed GPU memory exhaustion by isolating candidate evaluation into a dedicated, memory-efficient component: GCGCandidateEvaluator.

Technical Implementation

When evaluating 512 or 1,024 candidate suffixes across multiple models, storing all prediction logits simultaneously quickly exceeds the 24GB or 80GB VRAM limits of modern GPUs.

To fix this, the evaluation loop evaluates prompts sequentially and explicitly frees temporary tensors from GPU memory immediately after their losses are accumulated:

# pyrit/executor/promptgen/gcg/attack/gcg/candidate_evaluator.py

with torch.no_grad():
    for j, cand in enumerate(control_candidates_by_group):
        for i in range(num_prompts):
            # Dispatch logits computation to worker models
            for k, worker in enumerate(self._workers):
                worker(self._prompts[k][i], ModelWorkerOperation.LOGITS, cand, return_ids=True)

            # Retrieve worker logits and token IDs
            logits, ids = zip(*[worker.results.get() for worker in self._workers], strict=True)

            # Accumulate loss on main_device across all models
            loss[j * batch_size : (j + 1) * batch_size] += sum(
                self._loss_function.compute_loss(
                    logits=logit,
                    token_ids=token_ids,
                    target_slice=self._prompts[k][i]._target_slice,
                    control_slice=self._prompts[k][i]._control_slice,
                ).to(self._main_device)
                for k, (logit, token_ids) in enumerate(zip(logits, ids, strict=True))
            )

            # Explicit cleanup: delete bulky intermediate tensors immediately
            del logits, ids

Why does del logits, ids matter here? In Python, temporary variables inside loops can remain referenced until the next iteration. In PyTorch, logits tensors for large batch sizes occupy gigabytes of memory. Calling del logits, ids informs Python's garbage collector and the PyTorch caching allocator that these memory blocks can be reused immediately, bounding VRAM consumption to a constant size.

Additionally, the evaluator accounts for numerical edge cases such as NaN or Inf losses, ensuring stability during extreme optimization steps.

[7] Phase 3: Progressive Scheduling and State Machine

Reference -> PR #2720

The final phase extracted multi-model curriculum scheduling and budget accounting into ProgressiveScheduleController.

Technical Implementation

Instead of maintaining nested if/else flags inside ProgressiveMultiPromptAttack.run(), we created an explicit state machine and state container:

class ScheduleTransitionAction(Enum):
    """Action to take following an inner attack result in progressive scheduling."""
    CONTINUE = auto()
    FINALIZE_AND_STOP = auto()


@dataclass
class ProgressiveScheduleState:
    """Typed schedule state tracking curriculum admission and steps."""
    goals_admitted: int
    workers_admitted: int
    steps_completed: int = 0
    loss: float = float("inf")
    stop_inner_on_success: bool = False

The controller enforces an invariant in multi-model attacks: goals must be admitted before target model workers are expanded.

# pyrit/executor/promptgen/gcg/attack/base/progressive_schedule.py

def update_on_success(self, state: ProgressiveScheduleState) -> ScheduleTransitionAction:
    """Advance the progressive curriculum when an attack threshold is met."""
    if state.goals_admitted < self._total_goals:
        # Admit next attack goal
        state.goals_admitted += 1
        return ScheduleTransitionAction.CONTINUE
    elif state.workers_admitted < self._total_workers:
        # Admit next target model worker
        state.workers_admitted += 1
        return ScheduleTransitionAction.CONTINUE
    
    # All goals and workers admitted: finalize attack
    return ScheduleTransitionAction.FINALIZE_AND_STOP

The controller also manages exact step budgeting, guaranteeing the attack terminates cleanly when the user's iteration limit is reached without leaving hanging tasks.

[8] Code Quality, Reviews, and Maintainer Collaboration

Working on core infrastructure in a major open-source repository requires close collaboration with maintainers. Throughout this project, Microsoft maintainer Roman Lutz reviewed each pull request.

During the execution of Phase 2, Roman acknowledged the development velocity:

In Phase 3 (PR #2720), Roman identified an important compatibility detail regarding Python class definitions. When ProgressiveScheduleState was moved into the new module, an existing import in attack_manager.py shadowed the class. As a result, caller isinstance() checks returned False:

"The original ProgressiveScheduleState definition still shadows the new import. That means controller.state is a different class from the one exposed through the existing import path... Dataclass equality also fails between the two classes even when every field matches... A regression assertion using the existing import path would catch this."

I resolved this by removing the duplicate definition, re-exporting the imported class directly, and adding explicit test assertions covering both isinstance and dataclass equality:

# tests/unit/executor/promptgen/gcg/test_run_state.py

assert isinstance(attack.last_schedule_state, attack_manager.ProgressiveScheduleState)
assert attack.last_schedule_state == expected_state

Roman verified the fix and approved the PR:

"Confirmed in 2138122 that both import paths now expose the same class, restoring isinstance checks and dataclass equality for the returned state. This addresses the compatibility issue."

With all three PRs merged, Issue #2665 was officially closed, setting the stage for future performance optimizations:

[9] Results and Engineering Impact

The refactoring achieved significant measurable improvements:

• Clean Separation of Concerns: Replaced a 1,000-line monolithic loop with three modular, single-responsibility components.
• Eliminated VRAM Spikes: Sequential evaluation paired with immediate tensor disposal resolved GPU memory exhaustion across multi-model benchmarks.
• Zero Breaking Changes: Maintained complete backward compatibility for all public methods, parameter signatures, and configuration classes.
• Comprehensive Test Suite: Over 300 automated unit and integration tests now run deterministically across Windows, macOS, and Linux runners without requiring active GPUs.

[10] My Takeaways

1. Decouple Complex Algorithms Early: Large research algorithms tend to accumulate responsibilities over time. Separating proposal, evaluation, and scheduling makes codebases easier to maintain and extend.
2. Deterministic Cleanup in Accelerated Workloads: In PyTorch applications, relying solely on Python automatic garbage collection is insufficient for high-throughput loops. Explicitly deleting intermediate tensors prevents memory fragmentation.
3. Guard Class Contracts During Refactoring: Moving a dataclass or type definition can silently break isinstance checks if legacy export paths are not strictly validated with regression tests.

Thanks for Reading :)

Connect with me on LinkedIn