Social Science
The political preferences of LLMs
Abstract
I report here a comprehensive analysis about the political preferences embedded in Large Language Models (LLMs). Namely, I administer 11 political orientation tests, designed to identify the political preferences of the test taker, to 24 state-of-the-art conversational LLMs, both closed and open source. When probed with questions/statements with political connotations, most conversational LLMs tend to generate responses that are diagnosed by most political test instruments as manifesting preferences for left-of-center viewpoints. This does not appear to be the case for five additional base (i.e. foundation) models upon which LLMs optimized for conversation with humans are built. However, the weak performance of the base models at coherently answering the tests’ questions makes this subset of results inconclusive. Finally, I demonstrate that LLMs can be steered towards specific locations in the political spectrum through Supervised Fine-Tuning (SFT) with only modest amounts of politically aligned data, suggesting SFT’s potential to embed political orientation in LLMs. With LLMs beginning to partially displace traditional information sources like search engines and Wikipedia, the societal implications of political biases embedded in LLMs are substantial.
Key Questions
1. Do conversational LLMs exhibit political biases?
Yes, the study found that most conversational LLMs tend to generate responses that align with left-of-center viewpoints when assessed using various political orientation tests.
2. How were the political preferences of LLMs evaluated in the study?
The researcher administered 11 political orientation tests to 24 state-of-the-art conversational LLMs, both closed and open source, to assess their political leanings.
3. Do base (foundation) LLMs without fine-tuning exhibit political biases?
The study observed that base models did not consistently exhibit the same left-of-center biases as conversational LLMs. However, their weak performance in coherently answering the test questions made these results inconclusive.
4. Can LLMs be fine-tuned to specific political orientations?
Yes, the study demonstrated that LLMs could be steered towards specific locations in the political spectrum through Supervised Fine-Tuning (SFT) with modest amounts of politically aligned data, indicating SFT's potential to embed particular political orientations in LLMs.
5. What are the societal implications of political biases in LLMs?
As LLMs begin to partially displace traditional information sources like search engines and Wikipedia, the embedded political biases could significantly influence public opinion and information dissemination, underscoring the importance of addressing these biases.
