Elucidating the Design Space of
LM Data Attribution

We fine-tune a model on made-up facts and trace an attribution query back to the training document that taught it. Each token's influence under cross-entropy and under negative log-odds, then averaging the 10 highest-scoring tokens, moves the ground-truth document from rank 460 to rank 1 of 1,000.

TL;DR. Data attribution benchmarks disagree on which methods work, and since each uses different training data, queries and tasks, it is hard to tell why. We fine-tune an LLM on fictitious facts, so we know exactly which documents taught each answer, and vary one property of the training data at a time. Existing methods lose track of these documents when a fact is repeated across many documents or when look-alike distractors are added. The cause is at the token level: the evidence sits in a few tokens per document, but standard methods average it with every other token, and cross-entropy further suppresses the tokens the model is most confident about. Scoring each document by its top-10 tokens and using the negative log-odds loss sharply raises recall, and the loss change also helps on FTRACE‑TREX.