How to Make Causal Inferences Using Texts

New text as data techniques offer a great promise: the ability to inductively discover measures that are useful for testing social science theories of interest from large collections of text. We introduce a conceptual framework for making causal inferences with discovered measures as a treatment or outcome. Our framework enables researchers to discover high-dimensional textual interventions and estimate the ways that observed treatments affect text-based outcomes. We argue that nearly all text-based causal inferences depend upon a latent representation of the text and we provide a framework to learn the latent representation. But estimating this latent representation, we show, creates new risks: we may introduce an identification problem or overfit. To address these risks we describe a split-sample framework and apply it to estimate causal effects from an experiment on immigration attitudes and a study on bureaucratic response. Our work provides a rigorous foundation for textbased causal inferences. ∗We thank Edo Airoldi, Peter Aronow, Matt Blackwell, Sarah Bouchat, Chris Felton, Mark Handcock, Erin Hartman, Rebecca Johnson, Gary King, Ian Lundberg, Rich Nielsen, Thomas Richardson, Matt Salganik, Melissa Sands, Fredrik Sävje, Arthur Spirling, Alex Tahk, Endre Tvinnereim, Hannah Waight, Hanna Wallach, Simone Zhang and numerous seminar participants for useful discussions about making causal inference with texts. We also thank Dustin Tingley for early conversations about potential SUTVA concerns with respect to STM and sequential experiments as a possible way to combat it. †Ph.D. Candidate, Department of Politics, Princeton University, [email protected] ‡Ph.D. Candidate, Graduate School of Business, Stanford University, [email protected] §Associate Professor, Department of Political Science, University of Chicago, JustinGrimmer.org, [email protected]. ¶Assistant Professor, Department of Political Science, University of California San Diego, [email protected] ‖Assistant Professor, Department of Sociology, Princeton University, brandonstewart.org, [email protected]