µKE: Matryoshka Unstructured Knowledge Editing of Large Language Models

Large language models (LLMs) have emerged as powerful knowledge bases yet are limited by static training data, leading to issues such as hallucinations and safety risks. Editing a model's internal knowledge through the locate-and-edit paradigm has proven a cost-effective alternative to retraining, though current unstructured approaches—especially window-based autoregressive methods—often disrupt the causal dependency between early memory updates and later output tokens.

In this work, we first theoretically analyze these limitations and then introduce Matryoshka Unstructured Knowledge Editing (µKE), a novel memory update mechanism that preserves such dependencies via a Matryoshka-style objective and adaptive loss coefficients. Empirical evaluations on two models across five benchmarks demonstrate that µKE improves edit efficacy by up to 12.33% over state-of-the-art methods, and remains robust when applied to diverse formatted edits, underscoring its potential for effective unstructured knowledge editing in LLMs.

Figure 1: Comparison between different unstructured editing paradigms. (a) One-for-All updates one memory for the entire edit target. (b) Window-by-Window splits the target into windows but overlooks memory dependencies. (c) Our Matryoshka approach maintains proper dependency while benefiting from multiple memory shifts.

Large Language Models increasingly power diverse applications but suffer from static training data limitations. Model editing has emerged as a cost-effective method for updating internal knowledge without extensive retraining. While early methods focused on structured knowledge triplets, recent approaches like AnyEdit extend to unstructured editing through window-based strategies.

However, these window-based autoregressive methods disrupt causal dependencies between early memory updates and later output tokens. µKE addresses this limitation through a novel Matryoshka-style memory update mechanism that preserves these critical dependencies.

Experimental Setup

We comprehensively evaluate µKE on long-form knowledge editing across multiple dimensions:

Key Findings

• µKE improves edit efficacy by up to 12.33% in BLEU score over state-of-the-art AnyEdit
• Achieves near-perfect scores (>99.9%) on multiple benchmarks with the UnKE-based variant (µKE*)
• Maintains robust performance across diverse editing formats including math, code, poetry, news, and chemistry
• Preserves model's general capabilities on MMLU and IFEval benchmarks
• Shows superior generalization to paraphrased questions compared to baseline methods
• Performance remains stable across different target lengths unlike window-based methods

Performance Analysis

Figure 5: µKE performance with various window sizes. Performance is relatively insensitive to window size, with variations typically within 3% across all metrics. The optimal window size varies by dataset (best at 80 tokens for UnKEBench, 60 for MQuAKE).

Figure 6: Batch editing performance. µKE consistently outperforms AnyEdit for different batch sizes, highlighting the significance of maintaining memory dependency for unstructured batch editing across UnKEBench, CounterFact, and MQuAKE benchmarks.

If you find our work useful, please cite:

@inproceedings{su2025muke,
    title={µKE: Matryoshka Unstructured Knowledge Editing of Large Language Models},
    author={Su, Zian and Huang, Ziyang and Zhang, Kaiyuan and Zhang, Xiangyu},
    booktitle={Conference on Language Modeling (COLM)},
    year={2025}
}