Personal Information Parroting in Language Models

Modern language models (LM) are trained on large scrapes of the Web, containing millions of personal information (PI) instances, many of which LMs memorize, increasing privacy risks. In this work, we develop the regexes and rules (R&R) detector suite to detect email addresses, phone numbers, and IP addresses, which outperforms the best regex-based PI detectors. On a manually curated set of 483 instances of PI, we measure memorization: finding that 13.6% are parroted verbatim by the Pythia-6.9b model, i.e., when the model is prompted with the tokens that precede the PI in the original document, greedy decoding generates the entire PI span exactly. We expand this analysis to study models of varying sizes (160M-6.9B) and pretraining time steps (70k-143k iterations) in the Pythia model suite and find that both model size and amount of pretraining are positively correlated with memorization. Even the smallest model, Pythia-160m, parrots 2.7% of the instances exactly. Consequently, we strongly recommend that pretraining datasets be aggressively filtered and anonymized to minimize PI parroting.

Key Contributions

R&R detector suite (regexes and rules) for identifying emails, phone numbers, and IP addresses that outperforms prior regex-based PII detectors
Empirical measurement of verbatim PI parroting on 483 manually curated instances, finding 13.6% exact reproduction rate in Pythia-6.9B via greedy decoding from preceding context
Scaling analysis across Pythia model sizes (160M–6.9B) and pretraining steps (70k–143k), demonstrating both positively correlate with PI memorization

🛡️ Threat Analysis

Model Inversion Attack

Core contribution is measuring training data extraction: when prompted with tokens preceding a PI span in the original pretraining document, the model reproduces personal information verbatim — a direct instantiation of training data reconstruction from a language model.