ABSTRACT
All politics relies on rhetorical appeals, and the ability to make arguments is considered perhaps uniquely human. But as recent times have seen successful large language model (LLM) applications to similar endeavours, we explore whether these approaches can out-compete humans in making appeals for/against various positions in US politics. We curate responses from crowdsourced workers and an LLM and place them in competition with one another. Human (crowd) judges make decisions about the relative strength of their (human v machine) efforts. We have several empirical ‘possibility’ results. First, LLMs can produce novel arguments that convince independent judges at least on a par with human efforts. Yet when informed about an orator’s true identity, judges show a preference for human over LLM arguments. This may suggest voters view such models as potentially dangerous; we think politicians should be aware of related ‘liar’s dividend’ concerns.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Supplemental data
Supplemental data for this article can be accessed online at https://doi.org/10.1080/00323187.2024.2335471.
Notes
1. See e.g. Spirling (Citation2023) for discussion as to why open source LLMs may be generally preferable to proprietary efforts.
2. For example, Gallup has historically asked “In your view, should immigration be kept at its present level, increased or decreased?”.
3. For 4 prompts, a large run of 300 arguments was generated. We did this to get a general sense of performance, in terms of how many ‘usable’ arguments we should expect: see Supporting Information (SI) C.
4. SI D gives more information on the curation process.
5. The task was fielded through MTurk with a random treatment assignment. Through random chance, the control group was slightly larger than the treatment. We also had several more people in the treatment group vs. the control fail to complete the task adequately to be included in the analysis.
Additional information
Notes on contributors
Alexis Palmer
Alexis Palmer is a PhD candidate at New York University.
Arthur Spirling
Arthur Spirling is the Class of 1987 Professor of Politics at Princeton University.