I Abliterated Gemma 4 on a MacBook

Chat models refuse things. Ask Gemma 4 how to pick a lock and you’ll get “I cannot provide instructions or advice on how to pick locks or engage in any illegal activities.” Every time, for every prompt that touches anything the safety training flagged.

Abliteration is a technique that removes this refusal behavior by editing the model’s weights directly. No fine-tuning, no gradient descent, no training data. Just linear algebra. Find the direction in the model’s internal representation that means “I should refuse,” and project it out of the weight matrices so the model can never produce it again.

It works beautifully on Llama and Mistral. On Gemma 4, it doesn’t work at all. I spent a full day finding out why—and how to fix it.

How Abliteration Works

The idea is from Arditi et al.: refusal in language models is mediated by a single linear direction in the residual stream. When the model processes a harmful prompt, activations shift along this direction, and the model generates “I cannot…”

The process is three steps:

1. Collect activations—run harmful and harmless prompts through the model, capture the hidden state at each layer.
2. Find the refusal direction—compute the mean difference between harmful and harmless activations. That difference vector is the refusal direction.
3. Project it out—modify the weight matrices so they can no longer produce output along the refusal direction.

For Llama, this takes about 20 lines of code and works on the first try. For Gemma 4, it took twelve attempts.

Finding the Refusal Direction

I ran 100 harmful prompts and 100 harmless prompts through Gemma 4 E2B-it and captured the residual stream at all 35 decoder layers. Each layer produces a 1536-dimensional vector. The mean difference between harmful and harmless activations reveals where the refusal signal lives.

Five Ways to Find the Direction

The difference-in-means gives you a refusal direction per layer. But there are multiple ways to combine or compute these directions. I tested five methods to see which best separates harmful from harmless activations.

The Standard Approach—and Why It Fails

Standard abliteration subtracts the refusal direction from the weight matrices. For each layer’s output projection:

W_new = W - refusal_dir @ (refusal_dir @ W)

This directly removes the model’s ability to produce output along the refusal direction. On Llama 3 or Mistral, this works immediately. On Gemma 4, it does nothing. Or it breaks everything.

I tried activation hooks on select layers—no effect. Hooks on all 35 layers—refusal dropped to near zero but the model degenerated into repeating “I I I I I I” on every response. Raw weight modification across various layer ranges—either the refusal persisted or harmless prompts started getting refused too.

Why Gemma Resists

Gemma 4 has an architectural feature that most models don’t: four RMSNorm layers per decoder block (most transformers have two). RMSNorm rescales vectors to a fixed magnitude after every operation.

When standard abliteration subtracts the refusal direction from a weight matrix, it changes both the direction and themagnitude of each row. Then the very next RMSNorm sees the magnitude change and rescales everything back to normal—partially restoring the refusal signal. You push the water one way, the current pushes it back.

Three problems, three fixes:

The Fix: Norm-Preserving Biprojected Ablation

The corrected pipeline:

1. Collect activations with winsorization (clip outliers at 99.5th percentile)
2. Per-layer difference-in-means—each layer gets its own refusal direction, not one shared direction
3. Biprojection—for each layer, subtract the component that overlaps with the harmless mean
4. Norm-preserving weight modification—project out the refusal direction, then restore row magnitudes

def norm_preserving_ablate(W, refusal_dir, scale=1.75):
    r_hat = normalize(refusal_dir)
    W_norms = torch.norm(W, dim=1, keepdim=True)  # save magnitudes

    # Project out refusal direction from output space
    correction = scale * torch.outer(r_hat, r_hat @ W)
    W_new = W - correction

    # Restore original row magnitudes
    W_new = W_new / torch.norm(W_new, dim=1, keepdim=True) * W_norms
    return W_new

Applied to self_attn.o_proj and mlp.down_proj—the two weight matrices that write into the residual stream—across the top 24 layers by refusal signal strength, with a scale factor of 1.75.

It worked on the first try.

Finding the Optimal Config

The technique worked, but there are two hyperparameters: how many layers to modify and how aggressively to scale the projection. I grid-searched 30 combinations (5 layer counts × 6 scale factors) on a 20-prompt test set.

Winner: 24 layers, scale 1.75. The only config that hit zero refusals on the test set with zero harmless damage. Confirmed on the full 100-prompt set: every harmful prompt answered, every harmless prompt clean and coherent.

More layers is not better. L=28 and L=32 actually performed worse—modifying too many layers introduces noise that degrades output quality without improving ablation. And scale has a clear sweet spot: below 1.5 doesn’t overcome Gemma’s norm layers, above 2.0 starts overcorrecting.

Before & After

Inside the Model: Before vs After

Numbers and before/after text are one thing. But what’s actually happening inside the model? I ran the same harmful prompt through both the original and abliterated models, captured the hidden state at every layer, and projected it onto the refusal direction.

What Didn’t Change

Abliteration is only useful if it doesn’t break normal behavior. I tested 130 harmless prompts across eight categories. Every single one produced a full, coherent, helpful response—identical quality to the base model.

What I Learned

Architecture matters more than technique. The exact same abliteration approach that works on Llama fails on Gemma because of RMSNorm. If you’re abliterating a new model and it’s not working, check how many normalization layers it has per block.

Single pass beats iteration. I tried running the full pipeline multiple times, hoping each pass would peel off more refusal signal. It didn’t help—the refusal count plateaued immediately while the model accumulated unnecessary weight modifications.

Your refusal detector matters. Half my “failures” were actually successes. The model was answering harmful prompts but opening with cautious preamble (“this is a complex issue”) that triggered my keyword detector. I spent hours optimizing against detector noise, not real refusals. Build a better detector before running experiments.

The whole thing runs on a MacBook. M4 Pro, 64GB. Model loading, activation collection, direction computation, weight modification, and full evaluation—end to end in under 10 minutes. No cloud GPUs needed.

Try it: Live demo on HuggingFace Spaces

Model: treadon/gemma4-E2B-it-abliterated on HuggingFace

Base model: google/gemma-4-E2B-it (5.1B params, Apache 2.0)

Eval dataset: Abliteration-Eval: A Benchmark for Uncensored LLMs

Paper: Refusal in Language Models Is Mediated by a Single Direction (Arditi et al., 2024)