Main

The heated debate among political scientists over whether ideological polarization has intensified in America is still ongoing1,2,3. Ideological polarization refers to individuals having divergent beliefs on ideological issues, such as abortion or affirmative action4. Some works conclude that Americans have moved towards the most extreme ideological positions of the political spectrum in recent decades5,6. Social media have been pointed out as an underlying root for this increasing disagreement7,8. Recommendation algorithms, in particular, are suspected to reinforce ideological segregation9, reducing engagement with information from opposing viewpoints, a phenomenon known as echo chambers10,11,12,13. Very recently, however, this hypothesis has been disputed, after observing that changing Facebook’s feed algorithm to reduce exposure to like-minded content does not seem to reduce the political polarization of users14,15,16.

Other researchers instead argue that the perception of widespread ideological polarization in American politics is exaggerated and primarily driven by the behaviour of political elites and the media17. They point to partisan–ideological sorting as a cause for the increasing divide between Democrats and Republicans, by which people sort into the ‘correct’ combination of party and ideology18. Over the past 40 years, there has been a substantial increase in the relationship between party identification and ideological and social identification19,20,21, and in the relationship between party identification and positions on a wide range of specific policy issues22. Partisan sorting contributes to shaping ideological consistency of the electorate, together with the correlations between their attitudes towards political issues23.

In the field of social psychology, researchers have argued that partisan sorting has been responsible for increased levels of partisanship and polarized behaviour, including partisan bias, activism and anger24. They claim that partisan–ideological sorting has increased affective polarization in the USA25 and other countries26, while ideological polarization has not changed much. Within this framework, groups with strong social identities, such as racial, religious or ideological identities, firmly align with parties in recent decades27. For instance, the average correlations between different issue attitudes and party identification (Democratic or Republican) between 1972 and 2004 increased by 5 percentage points per decade28. Other studies found correlations with race and religion: Black and atheist Americans aligned with the Democratic Party, while White and Christian groups with the Republican Party29. Likewise, Americans with higher incomes tend to hold more conservative preferences on economic policy but more liberal stances on social policy, and vice versa30. Interestingly, online interactions on social media are segregated by demographic factors31 and contribute to the rise of affective polarization through partisan sorting32.

To discern these underlying factors, researchers from different disciplines, ranging from political and social science to physics and computer science, have recently embarked on the endeavour of quantifying ideological polarization33,34,35. On the one hand, they leverage data from social media to define new methodologies to measure the extent of polarization36,37,38,39. However, social media users are usually not representative of the whole population40. For instance, Facebook and Twitter users are demographically biased, being younger, more liberal, and better educated, on average41. On the other hand, several measures of ideological polarization and partisan sorting have been proposed on offline data42,43,44. However, these methods are mainly based on self-reported scores with respect to the degree of partisanship or the ideological identity strength, often including substantial measurement error due to social desirability bias45. Similarly, partisanship measured by self-reported scores might be biased, even by the structure of the questionnaire itself46. Furthermore, most of these works encompass single topics, while ideological polarization embraces multiple issues at the same time47, thus requiring a multidimensional modelling framework48,49. When dealing with several topics simultaneously, the discrete nature of the Likert scale—commonly used to assess opinions in research questionnaires—makes it even harder to interpolate ideological polarization.

In this study, we propose a methodology to quantify partisan sorting and the evolution of ideological polarization simultaneously, with respect to different topics. To this aim, we focus on the respondents of the American National Election Studies (ANES, https://electionstudies.org/), encompassing opinions regarding a large number of political, social and economic issues. We take into account the political leaning (Democratic or Republican parties) and five demographic attributes (race, gender, age, affluence and education) of the ANES respondents, and embed their opinions into a two-dimensional ideological space. There we can gauge ideological polarization by simply measuring the Euclidean distance between opposite political and demographic groups. We can measure partisan sorting by the heterogeneity of the opinion distribution in the ideological space: ideologically more consistent parties will occupy smaller regions50, a sign of high ideological coherence in matching political leaning with issue preferences28. Furthermore, we chart the identification of socioeconomic groups into parties in the ideological space, depicted as the alignment between parties and different demographic groups.

Our results show that racial and income groups are aligned into parties, but the ideological distance between these demographic groups did not increase in the past 30 years. Instead, we observe that Democrats and Republicans have become ideologically more distant over time. Furthermore, Democratic voters became more heterogeneous after 2010: a part occupied a novel region of the ideological space, connected with opinions regarding minority rights. These observations directly contradict the hypothesis of sorted, more homogeneous parties in the past decade.

Results

We focus on the ANES surveys conducted in the years 1992, 2000, 2008, 2016 and 2020. The last five US Presidents were elected these years: Clinton (Democratic), Bush (Republican), Obama (Democratic), Trump (Republican) and Biden (Democratic). We preprocessed data to select questions related to different topics such as the state of the economy, government spending, religion or minority rights. By picking common questions across years, we can study the temporal evolution of the respondents’ opinions. We refrained from selectively choosing questions explicitly associated with ideological polarization, instead we included all questions that met quantitative criteria concerning scales and missing values (see ‘The ANES dataset’ section for details). Our working dataset consists of N = 11,614 respondents answering 29 different questions. This dataset includes questions related to policy (for example, ‘Should federal spending on aid to poor people be increased, decreased or kept about the same?’), social identity (‘Which of these statements comes closest to describing your feelings about the Bible?’) and personal beliefs or attitudes of the respondents (‘Would this country be better off if we worried less about how equal people are?’), see Supplementary Table 2 for a general overview. The questions explore the respondents’ values regarding government intervention, social equality, racial justice and personal responsibility. In this way, we aim to build a broader ideological profile based not only on direct policy preferences, but also on cultural beliefs and moral values that might indirectly influence opinions concerning political or governmental actions. While we believe that this enriched representation can better capture the ideological spectrum of the respondents, we also check that our results are consistent by selecting only a subset of questions specifically related to policy.

To compare the opinions of different groups of individuals, we leveraged the rich metadata in the ANES survey to identify five demographic attributes (race, gender, age, affluence and education) and the political leaning of the respondents. We select two opposite groups per attribute, reported in Table 1, to highlight the differences in their opinions. For instance, we consider Low-Income and High-Income groups if the respondent belongs to a percentile lower than 34% or higher than 66%, respectively, of the American income distribution. Individuals in the middle-income group were excluded. With respect to the political leaning, we considered Democrats and Republicans (as per self-attribution in ANES data), excluding individuals classified as Independents. Supplementary Fig. 3 shows the proportion of each group for the different attributes over the years.

Table 1 Classification of the respondents by attributes according to two opposite groups
Full size table

In the following, we will first describe the opinion embedding into the ideological space and show how parties and demographic groups occupy different regions. We can thus quantify the degree of identification of demographic groups into parties and the ideological distances between them. Furthermore, we will study the evolution of trajectories over time, showing how groups move in the ideological space over the years. Finally, we will show how partisan sorting decreased in the past decade, as Democratic voters became less ideologically homogeneous, occupying a novel region of the ideological space.

Embedding opinions into an ideological space

A possible data representation is a multidimensional space containing 11,614 points (respondents) in 29 dimensions (questions). Each data point corresponds to a vector in the space of the 29 questions selected (forming the axes), representing the opinion of a single respondent. However, this high-dimensional representation is sparse: as the number of dimensions increases, the volume of available space grows exponentially and every data point becomes increasingly separate from the rest51. As a consequence, distance metrics such as Euclidean distance lose their usefulness in high-dimensional spaces52.

Therefore, we reduced the dimensionality of the data representation by embedding the opinions of ANES respondents into a two-dimensional ideological space. This opinion embedding allows us to chart and quantify ideological polarization by using a simple Euclidean distance. Furthermore, it allows us to interpolate the discrete nature of the original data, where opinions are expressed in the Likert scale, into a continuous ideological space. In the past years, researchers have applied different machine learning techniques to estimate the structure of preferences within American political institutions53. Similar dimensionality reduction methods have been used to map the opinion similarity and ideological leaning of internet users54,55, as well as the scientific knowledge landscape into a low-dimensional spatial representation56,57. By studying the embedding trajectories of users in this space one can, for instance, predict their future evolution in time58.

In this work, we implement Isomap59, a nonlinear, quasi-isometric, low-dimensional embedding algorithm (see ‘Dimensionality reduction’ section for more details), to obtain a two-dimensional data representation. Unlike other embedding methods, Isomap is designed to preserve large-scale structures of the original data60, making it valuable for examining the global properties of the ideological space59,61. Since we selected the same set of 29 questions for each year, we embedded all opinions across years into the same ideological space, which allows us to study the temporal evolution of opinions. Note that the spatial orientation of the ideological space is arbitrary, with the two axes representing a (nonlinear) combination of the 29 questions selected. In this setting, the Euclidean distance between two points is the only meaningful information, measuring the ideological distance between the corresponding individuals: the larger the distance, the less akin their opinions. Supplementary Fig. 4 shows the density of respondents in the ideological space. Within this framework, partisan sorting can be quantified by the heterogeneity of the opinion distribution: more ideologically consistent and homogeneous parties will occupy smaller regions. Furthermore, we can measure the alignment of the opinions of different political and demographic groups: groups with similar stances on multiple topics will occupy the same regions in the ideological space.

As a first step, we checked that the opinion embedding does not lose part of the information contained in the original data by comparing the accuracy of a classification algorithm on original and embedded data. We predicted the political leaning of respondents (Republican or Democratic) by using logistic regression informed by (1) their responses to the full set of 29 questions and (2) their coordinates in the ideological space. The tenfold cross-validated balanced accuracy reads 0.76 ± 0.06 for original data and 0.75 ± 0.06 for embedded data (see ‘Dimensionality reduction’ section for details). Furthermore, we test that our findings (shown in the following) do not depend on the specific embedding representation by using a different value of the Isomap hyperparameter K and by implementing other embedding algorithms, namely principal component analysis (PCA)62, t-distributed stochastic neighbour embedding (t-SNE)63 and uniform manifold approximation and projection (UMAP)64 (Supplementary Information). Finally, we also check that our results hold by embedding opinions into a higher dimensional ideological space, formed, for example, by three or four dimensions (Supplementary Information). The cross-validated accuracy in predicting the political leaning of respondents in three- and four-dimensional embeddings is the same as in the two-dimensional case.

Ideological distance between demographic groups

Figure 1 shows the density distribution of opposite groups in the ideological space for different attributes. For each attribute, we assign a binary variable {+1, −1} to opposite groups, for example, a value +1 to Republicans and −1 to Democrats. We then divide the ideological space in a lattice of small cells and compute the average of the binary variable in each cell. Cells with a large majority of one group are coloured by darker colours, while cells in lighter colours are populated by roughly the same number of individuals of the two groups. For instance, in Fig. 1a, dark red (or blue) cells are mostly populated by Republicans (or Democrats).

Fig. 1: Opinion distribution of respondents across socioeconomic groups.
figure 1

a–f, The density distribution of opposite groups in the ideological space for the following attributes: Democrats (D) and Republicans (R) (a), Black (B) and White (W) (b), Female (F) and Male (M) (c), aged 17–34 and 55+ years (d), Low Income (LI) and High Income (HI) (e) and No College degree (NC) and College degree (C) (f). Darker cells have a strong prevalence of one group with respect to the other. Cells with no respondents are coloured in grey. We bin the ideological space into 34 × 29 cells.

Full size image

One can see that the opinions of different groups are not randomly distributed in the space, but rather ordered for most attributes. One group (for instance, Democrats) tends to occupy a well-defined region of the ideological space, while the opposite group (Republicans) is more present in the opposite region. The attributes of party, race, affluence and education show the most ordered distributions, with opposite groups clearly separated in the ideological space, while the opinion distributions with respect to gender and age are much more uniform. This indicates that Republicans and Democrats, for instance, have different and often opposite opinions with respect to the 29 questions selected. While this observation can be checked at the level of the single question, our method allows us a bird’s-eye view of ideological polarization with respect to many different topics at the same time.

Furthermore, opposite groups are not only segregated in the ideological space, but they are also ordered along a certain direction. The Black, Low-Income and No-College groups mostly populate the bottom region of the embedding, whereas the White, High-Income and College groups are mostly present at the top of it. This observation can be quantified by computing the gradient of the different opinion distributions throughout the ideological space. For each cell of Fig. 1, we computed the direction and rate of the fastest increase of the average opinion. By means of the ‘gradient’ function from the open-source Python library numpy65, we used finite differences to approximate the derivatives of the gradient with high accuracy. We defined the gradient of the opinion distribution as the vector sum of the gradient values of each cell. A longer (shorter) gradient vector is obtained, thus, if the opinion distribution is more (less) ordered along a certain direction.

Figure 2 shows the resulting gradients for each attribute, indicating that groups whose opinions are more polarized are Democrats versus Republicans, Low Income versus High-Income, Black versus White and No College versus College, in this order. The attributes of gender and age show little polarization, as indicated by short gradient vectors. In Fig. 2, we colour each vector by the party colour map, with Republicans in dark red and Democrats in dark blue. The remaining vectors are coloured according to the Republicans–Democrats axis. As a consequence, the education (College to No College) vector, almost orthogonal to the party one, is very lightly coloured.

Fig. 2: Party identification of socioeconomic groups.
figure 2

Gradient vectors for each attribute computed from their opinion distribution shown in Fig. 1. We discarded cells with less than ten respondents. We slightly smoothed data by applying a different Gaussian filter with a low standard deviation for each attribute. The blue–red colour map obtained for the party attribute is used, with Republicans in dark red and Democrats in dark blue. The vectors are coloured according to the Republicans–Democrats axis, that is, the colour of each point in the vectors is proportional to the radius multiplied by the cosine of their angle with the Republicans–Democrats axis.

Full size image

The gradient vectors are useful to quantify the identification of different demographic groups into parties. Indeed, one can compute the degree of partisan alignment by means of the cosine similarity, Sc, between vectors. Party and race gradient vectors point in similar directions with Sc = 0.94, indicating that the opinions of Black (White) individuals significantly overlap with those of Democrats (Republicans). The same observation partially holds for the affluence attribute (Sc = 0.72), opinions of High-Income (Low-Income) individuals are more similar to Republicans (Democrats). Conversely, the education vector is almost orthogonal to the party one (Sc = 0.17), indicating little correlation between the educational scale and partisan leaning. We remark that this observation is valid over time-aggregated data, that is, over a temporal horizon of 30 years. Recently, instead, well-educated individuals increasingly identified with or leaned towards the Democratic party66. Therefore, one could identify the party–education attributes as the two main axes of the ideological space, and express the likelihood of individuals to belong to the four groups defined by these axes. Individuals in the upper (lower) region of the ideological space are more likely to be Republicans (Democrats) with (without) a College degree. Looking at Fig. 1b,e, one can see that this upper (lower) region is mostly populated by White (Black) and High-Income (Low-Income) individuals.

The correlation between the opinions of different demographic groups observed so far can be quantified more precisely. In Supplementary Fig. 5 we compare the opinion distributions in the ideological space of the most polarized attributes (that is, party, affluence, race and education) by means of the two-dimensional Kolmogorov–Smirnov (KS) test67. In this way, we identify the demographic groups with similar opinion distribution, that is, showing a certain degree of ideological affinity68,69. Supplementary Fig. 6 shows that different socioeconomic groups can be ordered in terms of ideological affinity, from Republican, White and High-Income groups to Low-Income, No-College and Black groups.

Charting ideological trajectories in time

Next, we studied the temporal evolution of the opinions of different groups over 30 years. To this aim, we started by considering a coarse-grained quantity to capture the aggregate behaviour of different socioeconomic groups. We computed the centroid (or geometric centre) of each group in each year, defined as the arithmetic mean position of the opinions of the group, indicating the average group opinion. Figure 3 shows the Euclidean distance between the centroids of opposite groups, for each attribute across the years. We remark that we do not compare distances between different ideological spaces from each year but, since we select common questions across years, we can chart how different groups’ opinions evolve in the same ideological space. From Fig. 3, we see that the groups in party, race, affluence and education show larger distances, indicating that they are more polarized. This corroborates the results shown in Figs. 1 and 2 that the attributes with the largest centroid distances correspond to the most ordered opinion distributions with the longest gradient vectors. Furthermore, we check that the Euclidean distance between centroids is proportional to the KS distance between distributions, provided by the KS test (Supplementary Fig. 7). This observation confirms that the closer two groups are in the ideological space, the more similar their opinion distributions, a sign of ideological affinity.

Fig. 3: Evolution of the ideological distance across socioeconomic groups.
figure 3

The Euclidean distance between the average opinions (centroids) of the opposite groups of each attribute, for every year.

Full size image

Concerning the temporal evolution, we note that most centroid distances do not vary much over time, with the notable exception of the party groups. The disagreement between Democrats and Republicans has considerably increased over the years, overcoming the ideological separation between racial groups after 2008. Notably, this finding is also valid when considering separately policy and non-policy issues (Supplementary Fig. 8). We stress that one cannot directly compare absolute distances between different embeddings (for example, by comparing the distances reported in Fig. 3 and Supplementary Fig. 8), but should only compare distances between different groups in the same embedding. As such, we can observe a similar pattern for both policy-related and non-policy-related questions. The groups that show the greatest ideological distance are partisan and racial groups, but only partisan divisions have grown over time. The only important difference when splitting between policy and non-policy issues is the larger ideological distance between College and No-College groups for non-policy issues. We remark that political and demographic groups are not formed by different individuals, but the same individual belongs to many groups, for example, White, High Income and College. To check that the centroid distances are statistically significant, we built a null model assuming that opinions do not differ across groups, see ‘Null-hypothesis significance testing’ section for details. We find P values lower than 10−4 for all attributes on all years, with the exception of Gender in 2016 (P = 0.016). This indicates that most socioeconomic groups are characterized by different, and in some cases very distant, opinions, as also indicated by the KS test. Finally, we check that the ideological distances between opposite groups follow a very similar trend also when considering a three- or four-dimensional ideological space (Supplementary Fig. 9).

While the centroid distance provides relevant information regarding the relative displacement of the two groups within the ideological space, we are also interested in charting the trajectory of each group over time. With this aim, we studied the evolution of the position of the average opinion of the most polarized groups (party, race, affluence and education) across years in the ideological space. First, we note that the average opinion of the general population varies over time, especially in 2008 and 2020 (Supplementary Fig. 10). Therefore, to meaningfully compare the trajectory of each group, in Fig. 4 we discard such a drift by subtracting the average population opinion from each group, for every year. In this way, we can see if groups move closer or not to the average population opinion, located at \(\left(0,0\right)\) in Fig. 4.

Fig. 4: Ideological trajectories of socioeconomic groups.
figure 4

The average opinion (centroid) of the party, race, affluence and education groups across years in the ideological space. The centroids are linked by arrows in chronological sequence: 1992, 2000, 2008, 2016 and 2020. The average opinion of the population of each year is subtracted from each group, being located at \(\left(0,0\right)\) and marked with a star. The x and y axes are arbitrary combinations of the 29 questions selected.

Full size image

Note that we recover the spatial disposition of the groups in the ideological space found in Figs. 1 and 2. Republicans, White, High-Income and College groups are located in the upper region above the centre \(\left(0,0\right)\), whereas we find their counterparts at the bottom. While most groups orbit in the ideological space without moving much across years, we note two notable exceptions. First, the average opinions of the Black group and the general population were particularly similar in 2008. This could be the result of the ‘Obama effect’70, or how Barack Obama’s election in 2008 influenced the prejudice of White individuals against Black people. Between July 2008 and January 2009, racial prejudices were reduced by a rate that was at least five times faster than in the previous two decades71. The Obama effect could be reflected in the trajectories charted in the ideological space, with the Black group closer to the general population in 2008. Second, the increasing partisan polarization reported in Fig. 3 is due to both Republicans and Democrats moving away from the centre, especially in 2016 and 2020. We address this point more in detail in Fig. 5, showing the distance of Democrats and Republicans from the centre each year. We observe that Republicans are constantly further away from the centre than Democrats, as they steadily depart from it since 1992. In 2016 and 2020, however, Democrats almost caught up with the gap. Supplementary Fig. 11 shows that this finding holds when considering a three- or four-dimensional ideological space.

Fig. 5: Evolution of the Democrats’ and Republicans’ ideological distance from the centre.
figure 5

The Euclidean distance of the average opinions of Democrats and Republicans from the centre, for every year.

Full size image

It is important to bear in mind that the geometric centre only provides information about the average opinion of the group. One can further characterize the opinion of groups by taking into account the heterogeneity of their distribution. If a group populates a large region of the ideological space, their opinions are very heterogeneous, while groups localized within a small region are characterized by more homogeneous opinions. In Supplementary Fig. 12, we show the radius of gyration of the opinion distribution of all groups considered, for every year (see ‘Heterogeneity of the opinion distribution’ section for details). We observe that the opinion dispersion depends more on time than the specific group. The radius of gyration clearly increased for all groups in 2016 and particularly in 2020. In general, opinions within a certain group became more heterogeneous over time, with the exception of Black and Republicans, whose opinions remained more homogeneous.

This last observation indicates that not only have parties become more ideologically distant on crucial issues in the past decades, but they have also become less ideologically homogeneous. Figure 6 supports this intuition by showing the opinion distribution of Democrats and Republicans aggregated before 2010 (years 1992, 2000 and 2008) and after 2010 (years 2016 and 2020). One can see that, while the Republican distribution does not change much from the first 20 years to the last 10 years, the Democratic distribution has become more dispersed in recent years, extending over a large region of the ideological space. This observation indicates that partisan sorting has declined in the past decade.

Fig. 6: Opinion shift in Democratic voters.
figure 6

a–d, The probability density function (PDF) of Democrats (a and c) and Republicans (b and d) in the ideological space. The results are aggregated over 1992, 2000 and 2008 (a and b) and 2016 and 2020 (c and d). A brighter (darker) colour indicates areas with higher (lower) density of individuals. The crosses represent the centroid of each distribution. The axes are arbitrary combinations of the 29 questions selected.

Full size image

We quantify the difference observed in Fig. 6 by computing the KS distance between the opinion distributions of different groups. We find that the KS distance between Democratic opinions before and after 2010 (Fig. 6a,c) is 0.32, much higher than the one between Republican opinions in the same time span (Fig. 6b,d), equal to 0.13. For comparison, the KS distance between pre-2010 Democrats and post-2010 Republicans (Fig. 6a,d) is equal to 0.44. Note that the average opinion (centroid) of Democrats in the years 2016 and 2020, represented by a cross in Fig. 6c, is poorly representative of the whole distribution as it falls in the middle of the two most populated areas. Indeed, a group of respondents emerges in a new region of the space (on the right side of the plots), indicating a clear opinion shift in a share of Democratic voters. By comparing their opinions in the two periods, we discover that the observed shift is mainly due to stronger opinions regarding racial resentment and other personal attitudes about social justice (Supplementary Table 3). To address this point more precisely, we repeated the analysis by splitting the battery into policy-related and non-policy-related questions, finding that the opinion shift among Democrats emerges mainly when considering only questions primarily related to social identity and personal beliefs or attitudes (Supplementary Fig. 13).

Furthermore, while racial and income groups are aligned into parties, such an alignment remains roughly constant over time. For instance, the fraction of White individuals in the Republican party was 0.84 before 2010 and 0.85 after 2010, while in the Democratic party that fraction was 0.56 and 0.63, respectively. Likewise, the fraction of Low-Income individuals in the Democratic party was 0.38 before 2010 and 0.36 after 2010, while in the Republican party that fraction was 0.24 and 0.27, respectively. Therefore, parties became ideologically more distant and less homogeneous, while their alignment with respect to demographic groups remained roughly constant.

Discussion

The results presented above offer a clear answer to the main research question addressed in the paper, of whether ideological polarization actually increased in the past decades or voters simply sorted themselves into parties matching their ideology more closely. We propose an approach to quantify the evolution of ideological polarization across political and demographic groups over time. We found that the ideological distance about fundamental issues between Democrats and Republicans increased in the past 30 years: both parties progressively moved away from the centre, at different rates (Fig. 5). Moreover, we observed that parties, and especially Democrats after 2010, became more heterogeneous, with a part of Democrats shifting their opinions regarding several issues connected to racial resentment and other personal attitudes about social justice (Fig. 6). These findings directly contradict the hypothesis of partisan sorting as the root of the increasing political polarization in America3,17,28—parties did not become more ideologically distant, they became more homogeneous and coherent on ideological issues. Furthermore, we explored the partisan alignment of demographic groups, finding that, while some of them (especially racial and income groups) strongly identify with parties (Fig. 2), such identification does not grow across years. Likewise, the ideological distance between all demographic groups did not increase over time, at odds with the ideological distance between parties (Fig. 3). These findings also offer interesting insights regarding the hypothesis that parties built stronger social identities in the past decades, fuelling social polarization19,21,24.

On the methodological side, the two-dimensional ideological space we defined provides a more nuanced characterization of an individual’s ideology than the simple one-dimensional liberal/conservative axis commonly used in the literature. We remark that our results remain robust across different choices of Isomap hyperparameters, alternative embedding algorithms and higher dimensionalities of the ideological space. This modelling choice allows us to chart and easily visualize the evolution of ideological polarization and partisan sorting by computing the distance between opinion distributions and their heterogeneity, as well as the trajectories of political and socioeconomic groups in time. Such analysis would not be possible in the traditional one-dimensional representation, or much more cumbersome in higher dimensionalities. Moreover, we selected questions through only objective criteria from the ANES dataset, thus avoiding potential measurement errors resulting from social desirability bias45. For instance, we excluded all questions related to self-reported scores, such as the degree of partisanship or the ideological identity strength.

It is important to contextualize our findings within the historical period under examination, which witnessed a gradual, and subsequently rapid, realignment of parties along racial, religious and cultural lines20,72. It can be argued that the 1992 presidential election marked the inception of the Republican party’s shift towards confrontational politics73. Additionally, the 2008 critical election is often seen as the endpoint of the domination of conservatism in the USA since the late 1970s74. The Obama presidency contributed to the swift racialization of American politics75, expanding racial considerations to previously nonracialized issues. On the other hand, the 2016 and 2020 elections were characterized by the emergence of a rapidly coalescing anti-diversity coalition of Americans mobilized and organized by Trump76. The strong partisan divide reflected an unprecedented alignment of racial, religious and ideological issues, which paved the way for the presidency of Trump77. Within this temporal dynamic, it is worth stressing that our methodology captures the ideological transition around the crucial year of 2010, which saw significant shifts in attitudes towards race78.

Our work is not exempted from some limitations, regarding both the data used and the methodology proposed. Concerning the former, the number of respondents in the dataset is not the same every year, for example, it was particularly low in the year 2000. Therefore, we reduced the number of individuals from the more participated surveys, 2016 and 2020, to have a similar number of individuals across years (see ‘The ANES dataset’ section). Likewise, the number of respondents in opposite groups is similar but not equal: some groups are more populated than their counterparts (Supplementary Fig. 3). The case of the race attribute is especially noteworthy, since the majority of ANES respondents are White. Furthermore, our methodology requires selecting the same set of questions across time. While this choice allows us to embed all individuals in the same ideological space and thus directly compare distances (not possible across different embeddings), it also limits the number and types of questions in the analysis. Tracking the same issues over time, indeed, overlooks significant political changes over the past 30 years. For instance, it would be important to include questions regarding transgender rights, especially relevant in the past decade. Our dataset includes a mix of policy (anti-gay discrimination law, federal spending and capital punishment), moral values (preference for equality and child rearing) and procedural (government organization) questions, thus avoiding selecting explicitly partisan-cheerleading questions. Instead, we used question-agnostic quantitative criteria that can be applied to other political survey datasets, intending to remove the human bias in hand-picking questions.

Regarding the methodology instead, a potential issue arises in quantifying ideological polarization based on the distance between average opinions of opposite groups, depicted by centroids in the ideological space. This definition may present challenges, particularly when centroids lose significance in highly heterogeneous distributions, such as the one reported in Fig. 6c. Respondents belonging to the same group could hold substantially different opinions, while the centre is located where the density of individuals is almost zero. Similarly, the centroid might be influenced by a few extreme opinions, despite the majority of individuals remaining moderate. To overcome this limitation, we implemented measures going beyond simple means79,80, such as the KS distance measuring overlap between distributions and the radius of gyration gauging heterogeneity. Furthermore, we obtained fully equivalent results by computing the distribution centroids by means of the median (Supplementary Fig. 14), which usually works better than the arithmetic mean in the presence of outliers. Finally, we remark that our findings are valid within the boundaries of the ANES dataset used and are limited to the questions selected to build the ideological space.

Despite this latter limitation, we believe the methodology proposed here can help political and social scientists to better quantify ideological polarization and social sorting. Our framework can be applied to any dataset including multiple topics and demographic features of individuals, encompassing a single temporal point or several ones. In this latter case, our approach proved particularly useful, allowing us to chart the trajectories of different demographic and political groups into a single ideological space. We remark that our methodology does not choose any a priori social or political dimension for the embedding, rather we observed the emergence of two orthogonal axes in the ideological space, corresponding to partisan orientation and education. In the future, we hope the ANES will continue to collect data regarding the same (or very similar) political, social and economic topics, to allow tracking the temporal evolution of demographic groups. Assembling datasets rich in the temporal dimension is indeed pivotal for a longitudinal assessment of political and ideological polarization among the electorate5,6,81. Likewise, with similar datasets it would be possible to gauge more precisely the evolution of social sorting in time. One example is the General Social Survey (https://gss.norc.org/), which for 50 years, has been collecting the opinions of Americans on a wide range of topics such as confidence in government, abortion or gun rights, together with the demographic information of respondents. Furthermore, our methods could be applied to countries different from the USA, where both political and societal polarization are threatening governance and democratic norms82. Finally, while our results are purely observational, future work is needed to assess causal connections between ideological polarization and social sorting.

Methods

Here we describe the ANES dataset, the robustness of the dimensionality reduction, the null model testing the significance of the centroids and the way we compute the heterogeneity of opinion distributions. Further details can be found in the Supplementary Information.

The ANES dataset

ANES (https://electionstudies.org/) is a continuation of a series of academically run surveys, asking questions to a representative sample of citizens in the USA about their opinions on a range of topics. The 2022 release includes the answers of 68,225 respondents throughout a sample of 32 years from 1948 to 2020. The dataset encompasses 1,030 variables in total: 161 variables are related to survey information (year, language used, interview mode and so on) and to information about both interviewer and respondent (gender, race, age and so on), while the remaining variables consist of the different questions collected by the surveys. Although it includes information about both race and ethnicity, in our analysis we only use information on race.

Our objective is to study the ideology of the American population about general topics, thus we narrow down the question list as follows. First, we excluded questions about political parties or election candidates (including presidential candidates). Then, we excluded binary questions (with only two possible answers), which most of the time refer to single events and not general topics such as ‘Do you read a daily newspaper?’. Since not all the questions have the same number of options to answer, we normalized all the scales to the range (0, 1), with negative and positive extreme answers close to 0 and 1, respectively. The ANES dataset is reduced to a total of 99 questions collecting opinions on different topics such as the state of the economy, government spending, religion or minority rights.

The questions vary over the years, according to the particular socioeconomic situation. For example, the disease AIDS was not discovered until the beginning of the 1980s, so the question ‘Should federal spending on AIDS research be increased?’ was only included after 1988. To compare the opinions collected throughout different years, we can only take into account shared questions. Since the number of shared questions between pairs of years is maximum after 1990 (Supplementary Fig. 1), we focus on the last 30 years. Within this time window, we selected the years 1992, 2000, 2008, 2016 and 2020. These years correspond to the election years of the last five US presidents, and are also the ones with the largest number of collected questions. This leaves us with a total of 41 questions (see Supplementary Table 1 for other combinations of years). After discarding the ones with a rate of missing answers greater than 20% (Supplementary Fig. 2), the number of shared questions is reduced to 29, which are reported in Supplementary Table 2. Moreover, since the number of respondents in the years 2016 and 2020 is much greater than in 1992, 2000 and 2008, we reduced the former by choosing a random number of respondents to have a similar number of respondents across years. We checked that this random selection of respondents does not alter our results. In chronological sequence, each year finally contains 2,485, 1,807, 2,322, 2,500 and 2,500 respondents. This set with 11,614 individuals in total forms the final dataset used in the paper.

Dimensionality reduction

To perform the dimensionality reduction of the dataset, we applied the Isomap algorithm59. Isomap considers the distribution of the K neighbouring data points by attempting to preserve pairwise geodesic (or curvilinear) distances83. Isomap estimates the geodesic distance between data points with the shortest path using Dijkstra’s algorithm84. The only hyperparameter of the algorithm is then the number K of nearest neighbours. All our findings were obtained by setting K = 10. Supplementary Figs. 15–17 show that a different choice of the hyperparameter K leads to fully equivalent results. Our Isomap implementation uses Isomap from the open-source machine learning library scikit-learn85.

Regarding the classification task, we relied on cross-validation for the evaluation of the classification performance, and we used logistic regression as the classification algorithm. As a strategy to split the data into training and testing sets, we used a Stratified K-Fold with ten splitting iterations. Our computational implementation uses cross_val_score and LogisticRegression from the library scikit-learn85.

Furthermore, we tested the robustness of our findings by implementing other embedding algorithms. PCA62 is a linear dimensionality reduction technique, that is, the new lower dimensions are a linear combination of the original dimensions. In essence, the low-dimensional representation describes as much of the variance in the high-dimensional data as possible. On the other hand, t-SNE63 and UMAP64 are nonlinear techniques like Isomap. Both methods are closely related, the low-dimensional space is constructed according to the statistical similarity between original points in the high-dimensional space. Supplementary Figs. 15–17 show that PCA, t-SNE and UMAP embeddings lead to similar results. We used PCA and TSNE from scikit-learn85, and the Python library umap64.

Null-hypothesis significance testing

We tested the statistical significance of the average opinions (centroids) of different demographic groups. Our null hypothesis is that opinions do not differ across groups. To test this, we ran a bootstrap analysis for each attribute by randomly assigning each respondent to one of the two opposite groups while preserving their proportion. We repeated this process 105 times for each attribute, obtaining the centroids of the demographic groups in every iteration. If each of the resulting 105 centroids per group is given by the real random vector \({\bf{X}}=\left(X,Y\;\right)\), we assume that it follows a bivariate normal distribution

$$f\left({\bf{x}}\right)=\frac{1}{2\uppi \sqrt{\left\vert \varSigma \right\vert }}\exp \left[-\frac{1}{2}\left({\bf{x}}-{\mathbf{\upmu }}\right){\varSigma }^{-1}{\left({\bf{x}}-{\mathbf{\upmu }}\right)}^{\mathrm{T}}\right]{,}$$
(1)

where Σ is the covariance matrix, \(\left\vert \varSigma \right\vert \equiv \det \left(\varSigma \right)\) and \({\mathbf{\upmu }}=\left({\mu }_{x},{\mu }_{y}\right)\) is the mean vector. The most general form of Σ is the symmetric and positive definite matrix

$$\varSigma =\left(\begin{array}{cc}{\sigma }_{x}^{2} & \rho \, {\sigma }_{x}{\sigma }_{y}\\ \rho \, {\sigma }_{x}{\sigma }_{y}&{\sigma }_{y}^{2}\end{array}\right),$$
(2)

where σx, σy are the standard deviations of x and y, respectively, and ρ is the Pearson correlation coefficient between both coordinates.

By definition, the distance of a point \({\bf{x}}=\left(x,y\right)\) to the two-dimensional distribution given by equation (1) reads as

$$r\left({\bf{x}}\right)=\sqrt{\left({\bf{x}}-{\mathbf{\upmu }}\right){\varSigma }^{-1}{\left({\bf{x}}-{\mathbf{\upmu }}\right)}^{\mathrm{T}}},$$
(3)

which is known as the Mahalanobis distance. If we keep it constant, then the location of x draws an ellipse centred in μ with semi-major and semi-minor axes given by the greatest and smallest eigenvalues of Σ as \(r\sqrt{{\lambda }_{1}}\) and \(r\sqrt{{\lambda }_{2}}\), respectively86. In a polar coordinate system \(\left(r,\theta \right)\), with r given by the Mahalanobis distance defined in equation (3), a parametric representation of x can be found as a function of both polar coordinates. Computing analytically the eigenvalues λ1 and λ2, we write

$$\begin{array}{rcl}x&=&r\,{\sigma }_{x}\cos \left(\theta \right)+{\mu }_{x},\\ y&=&r\,{\sigma }_{y}\left[\rho \cos \left(\theta \right)+\sqrt{1-{\rho }^{2}}\sin \left(\theta \right)\right]+{\mu }_{y}.\end{array}$$
(4)

Thus, its Jacobian determinant is \({{{J}}}_{{\bf{x}}}=r{\sigma }_{x}{\sigma }_{y}\sqrt{1-{\rho }^{2}}=r\sqrt{\left\vert \varSigma \right\vert }\).

In a null hypothesis statistical test, under the assumption that the null hypothesis is true, the P value is defined as the probability of obtaining results equally or more extreme than the result actually observed87. In other words, it measures the probability of obtaining a centroid distribution given by equation (1) compatible with the observed centroid. Following this definition, we compute the P value of a demographic group with opinion centroid at \({\mathbf{x}}={\mathbf{x}}^{* }\) as

$$\begin{array}{rcl}P\left(R\right)&=&\displaystyle\frac{1}{2\uppi \sqrt{\left\vert \varSigma \right\vert }}\mathop{\displaystyle\int}\nolimits_{0}^{2\uppi }\mathrm{d}\theta \mathop{\displaystyle\int}\nolimits_{R}^{\infty }{{{J}}}_{{\bf{x}}}\,\mathrm{e}^{-\frac{{r}^{2}}{2}}\mathrm{d}r\\ &=&\mathop{\displaystyle\int}\nolimits_{R}^{\infty }r\,\mathrm{e}^{-\frac{{r}^{2}}{2}}\mathrm{d}r=\mathop{\displaystyle\int}\nolimits_{{R}^{2}/2}^{\infty }\mathrm{e}^{-r}\mathrm{d}r=\mathrm{e}^{-{R}^{2}/2},\end{array}$$
(5)

where \(R\equiv r\left({{\bf{x}}}^{* }\right)\). Our null hypothesis stands that opinions do not differ across groups, which would mean that each observed centroid results from a random distribution, that is, it is not statistically significant. As usual, we reject the null hypothesis if the P value is less than or equal to the predefined threshold value of 0.05 (ref. 88).

Heterogeneity of the opinion distribution

Each respondent opinion embedded in the low-dimensional space can be written as a data point \({\bf{x}}=\left(x,y\right)\). How heterogeneous the distribution of opinions is can be computed by means of the covariance matrix given by equation (2). This square matrix is also known as the radius of gyration tensor, since its greatest and smallest eigenvalues λ1 and λ2, respectively, quantify the mean extension of the distribution in space86,89. Commonly, therefore, the radius of gyration is used to describe the average dispersion or heterogeneity, and it is defined as \(R_{g}=\sqrt{\lambda_{1}+\lambda_{2}}=\sqrt{{\rm{tr}}(\Sigma)}=\sqrt{\sigma_{x}^{2}+\sigma_{y}^{2}}\).

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.