From Bullshit to Cognition: Computing Within the Epistemic Crisis of Large Language Models in Systematic Literature Review

We stipulate that we are in a crisis of epistemology, and that large language models (LLMs) are a central aspect of that crisis. To the end of computing within this crisis, we situate an in-situ experiment with LLMs in a specific, specialized form of research, namely systematic literature review. Contrary to other studies in this domain, we argue that the question of LLM integration in systematic reviews cannot be fully addressed by measuring how well LLMs replicate the labor of human researchers. It is crucial to examine the epistemological implications of how LLMs are integrated into the process. To explore this, we conducted a systematic literature review, situated within the humanities and social sciences, using two different LLM products: a general-purpose model (GPT-4o) and a purpose-specific tool (Elicit). We analyze how our two exemplar implementations influence the review process and what this reveals about the cognitive and epistemological effects of LLMs in research, drawing on a conceptual vocabulary of focusing on notions of bullshit and nonconscious cognition.