NeuralPostmortem: An Interactive Framework for LLM Failures Analysis

1 Case Setup & Model Configuration

Pending

📁 Load from Trace (Auto-fills all fields)

Dataset: Trace File:

✍️ Manual Input

Prompt:

Completion:

Ground Truth (Optional):

🤖 Model Configuration

Series: Model: DType: 4-bit Quantization

Model Path:

2 Contrast Token Pair Identification

Locked

🎯 Target Token Localization & Text Truncation

Use multiple LLM validators to identify the first erroneous token in the completion.

Use LLM search (validators)

Validator Models:

GPT-5.2 GPT-OSS-120B DeepSeek-V3.2 Kimi-K2-Thinking

🔄 Contrast Token Selection

After error token localization, identify the contrast token (correct alternative).

Selection Method:

Manual Selection (from Top-K predictions) Beam Search (automatic exploration)

K =

Top-K Token Predictions:

Rank	Token	Logit	Action

3 Input Attribution Analysis

Locked

⚙️ Input Attribution Configuration

Rule: Attn/MLP Split

🎨 Input Attribution Visualization

Mode: Strategy:

🧪 Perturbation Evaluation (Ground-Truth Comparison)

Evaluate attribution quality by zeroing out top-attributed tokens and checking if the model's error is fixed. A better attribution method should fix the error at smaller K values.

K values (comma-separated):

Perturbation Results:

K	Perturbed Tokens	New Top-1 Token	Error Fixed?	Target Rank After	Logit Change

🖱️ Manual Token Perturbation

Click tokens in the Attribution Map above to select/deselect which tokens to mask. Selected tokens will be highlighted with a dashed border.

Selected Tokens:

No tokens selected

4 Attribution Graph Visualization

Locked

🔧 Graph Configuration

Layer Selection:

Backprop Mode: Strategy:

Pruning Mode:

Top-P Mass (0-1):

✅ Analysis Complete!

You have completed the full step-by-step analysis pipeline. Review the results above or export your findings.

🔍 Debug-XAI

📁 Load from Trace (Auto-fills all fields)

✍️ Manual Input

🤖 Model Configuration

🎯 Target Token Localization & Text Truncation

📊 Voting Results:

✂️ Truncated Text:

💡 Explanation:

🔄 Contrast Token Selection

Top-K Token Predictions:

✓ Contrast Token Pair Identified:

⚙️ Input Attribution Configuration

🎨 Input Attribution Visualization

Attribution Map:

Positive Relevance Tokens (Sorted by Relevance, Descending)

Corresponding Tokens (for verification)

🧪 Perturbation Evaluation (Ground-Truth Comparison)

Perturbation Results:

🖱️ Manual Token Perturbation

🔧 Graph Configuration

📊 Circuit Visualization

✅ Analysis Complete!