Max Stearns
I previously posted twice about the Media Bias Chart, produced by Vanessa Otero, a Colorado patent attorney and founder of Ad Fontes Media. Ms. Otero originally created the now-viral chart to provide herself and her friends a means by which to assess media sources in an age of increasingly prevalent and influential fake news during the 2016 presidential campaign. Her chart ranks news sources along two analytical dimensions: (1) a north-south (or y) axis assessing media reliability, and (2) and an east-west (or x) axis assessing media bias. The former corresponds to media quality, and the latter to a conventional left-right ideological framing as commonly employed within the United States.
My first post corresponded to Otero’s version 3.0, and my second post corresponded to version 4.0. This post focuses version 5.0. As compared with version 3.0, version 4.0 had modest changes. The graphic was a bit more compact, and although it retained Gaussian shape overall, its defining features were a bit less clearly pronounced. Version 5.0 exhibits a distinct change from Gaussian form to an inverted and somewhat flattened “V.”
Version 5.0 and those that will follow derive the underlying data using a different process. Rather than having Otero rank sources using her own intuition according to clearly defined criteria, for this version she hired a cohort of analysts that she trained to ensure evaluative consistency. These analysts were hired from an applicant pool in which each person self-identified across the political spectrum, with one liberal, one moderate, and one conservative evaluating several thousand stories as needed to derive the overall ratings reflected in the new graphic.
Explaining the new, multiple-analyst-generated graphic builds upon these earlier accounts. As a result, the post that follows operates in three parts, the first two providing synopses of the earlier posts, explaining the logic and features of the prior graphics, and the third culminating in a comparative analysis of version 5.0 with what came before. (Those familiar with the earlier posts might wish to skip ahead to part III). Part III will provide insights into why version 5.0 possesses its distinct features and will also reflect on what we might expect in the future.
Part I: Commitment Bonding and Dimensionality
In my first post, I set out my initial thesis concerning the logic of the graphic, including, most notably, its apparent Gaussian shape, based, in large part, on the series of seventeen evaluative factors that Vanessa Otero had employed to explain version 3.0. My primary insight and claim was that we could distill most of Otero’s evaluative criteria based on the presence or absence of features corresponding to credible commitment bonding for quality and accuracy, and the remainder to ideological bias. By commitment bonding, I mean costly infrastructures, such as print media and distribution channels, and journalistic staffing, which together convey a strong interest in ensuring and maintaining a reputation for accurate and reliable news coverage, along with credible analysis and opinion. The north-south axis was primarily associated with the presence of absence of such costly and credible commitment bonds.
The primary claim was that media sources that have committed to costly journalistic enterprises are less likely to risk compromising established reputations for credible and accurate quality reporting in exchange for whatever short-term benefits arise from sensationalistic or salacious stories. Although such stories might attract an immediate and temporary spike in readership, they simultaneously risk compromising the costly investments in reputation for accuracy and journalistic integrity. In the long term, the most credible media outlets will tend to confirm and disconfirm ideological biases on both sides of the conventional left-right spectrum approximately half the time, that is, assuming we can imagine an ideological location approximating the center.
The sources widely viewed as highly ranked for quality and accuracy, such as the New York Times, the Wall Street Journal, and the Washington Post, have made notable and costly financial investments that represent such commitment bonds. These include large investitive journalistic staffs well placed around the globe, and robust physical infrastructures associated, for example, with printing daily publications and effective distribution channels. In the social media age, with notable sources not including print media, comparable investments entail large journalistic staffs able to cover the scope of domestic and international events, political and otherwise. Another feature that characterizes such sources is a figurative (sometimes literal) wall separating the news from opinion divisions.
These features likewise correspond to maintaining high quality and accurate timely coverage of news, feature stories, and cultural events. Major media outlets thus invest massive resources in developing original news content, in addition to editorial staffs capable of providing meaningful commentary and analysis. One of Otero’s initial evaluative criteria, not surprisingly, included present non-print media outlets that had historically offered print media. This implies that now or in the past, the most credible sources invested in considerable start-up costs that can only be justified if, in the case of print sources, the medium has a vast readership, or in the digital context, a sufficient on-line readership to justify the investment and corresponding maintenance costs capable of deriving supportive advertising revenue.
Because such heavy financial investments form a credible commitment bond, sources issuing those bonds will generally be disinclined to allow the temporary attention associated with salacious coverage or sensationalistic stories that will appeal to potentially large numbers of readers on one or the other end of the ideological spectrum to gain that benefit at the risk of sacrificing more generally its overall reputation for rigor, detachment, and fair news coverage. Such short-term gains cannot justify the longer-term reputational hits given those costly investments.
Even media sources unable to generate original news reporting can nonetheless issue softer commitments corresponding to quality, such as careful opinion and analysis that avoids sensationalism in much the same way. And although opinion and analysis are not always entirely distinct (even news coverage sometimes blends into these categories), such enterprises try to identify opinion as such. At the outer extremes, sources unwilling or unable to offer even these forms of commitment bonding can compete along the dimension of attracting an ideologically motivated cohort of readers. This strategy corresponds to a greater willingness, indeed a desire, to appeal with sensationalistic, and sometimes salacious, coverage as needed to attract attention, often including outrageous attention-grabbing headlines. The consequence is generally to cater to strongly ideological readers, sometimes with unsupported claims, even bordering on the conspiratorial or containing outright falsehoods. In the original graphic through version 3.0, the resulting curve assumed an approximate Gaussian form, or the shape of a general probability distribution (PDF) function, even though the composition was not generated by population density data corresponding to the points along the x axis.
Finally, a note on dimensionality. In my initial post, I further responded to potential concerns that the left-right, or x, axis might not adequately account for dimensional complexity, especially in the dynamic context of the Trump campaign and eventual victory. I further discussed the special problem of libertarians who, nominally at least, claim to eschew conventional left-right ideology. Drawing upon an earlier post titled “Ideological Blindspots (Part IV): The Dimensionality of Trumpism” I explained that, although the left-right axis might shift over time, and although the Trump (and also Bernie Sanders) populist campaigns might shift the end points along that dimension, however the election settles, we will inevitable experience settled endpoints. Those endpoints will form the basis for newer anchors, and thus understandings, of the far left and far right positions, onto which a general ideological axis will form. There is much to be said about this in the aftermath of the Trump victory and his ascendant merging, and sometimes disrupting, conventional understandings of conservatism with elements of populism, but for now suffice to say that by my lights, history has large operated consistently with this prediction.
Part II: The Gaussian Curve as a Series of Readership- or Revenue-Maximizing Equilibria
In the second related post, I provided a more comprehensive analysis of the media graphic’s Gaussian shape, which, once more, might be likened to a standard probability distribution function (PDF). Because the graphic does not arise from population density plotting, the question arises what accounts for the curve's familiar, and appealing, presentation.
In the second post, I observed that the curve’s shape is best understood as a series of equilibrium points across Otero’s two identified, and relevant, dimensions: reliability (y axis) and ideological bias (x axis). The theses in these two posts are mutually reinforcing. This analysis helps to explain the factors that correlate to what we might call the maximand for published (print or on-line) media sources, namely readership and advertising revenue. Those sources capable of maximizing these highly valued inputs will do so by exhibiting relatively strong commitments to journalistic accuracy and quality. And they will do so in a manner that corresponds to the mechanisms that I distilled within first post as a series of media commitment bonds. Such sources, once more, will avoid the risk of dissipating their costly journalistic reputation for short-term gains that would result from catering to either extreme along the east-west ideological spectrum, or x axis, with the consequence that the most successful media sources equilibrate at or near the ideological center.
At the opposite extreme, sources unable to issue credible media-commitment bonds can gain large, if highly biased, readerships with sensationalistic or salacious coverage of already generated news stories and analysis provided initially by other, more reliable, sources. Such stories compete, along either side of the ideological spectrum, with other like-biased sources. The combined result pushes further and further in the direction of ideological bias at the expense of markedly diminished reliability.
Between these extreme poles, and descending along each side from the apex of best news sources toward least credible and most sensationalist sources, are sources exhibiting a range of quality respecting opinion and analysis. Within that grouping are variably biased media outlets that will seek to commit to, and maintain reputations for, consistently high quality reporting and analysis. As with sources that commit to original news reporting, these sources will likewise tend to eschew sensationalistic or salacious coverage for short term gains that arise at the expense of long-term reputations for careful news coverage and analysis.
Once more, this implies that this last group of sources will try to avoid drifting universally to the extreme right or extreme left so as to be predictably viewed seriously by a readership seeking careful analysis and opinion that is generally regarded as fair, even if not entirely impartial. The fairness of such reportage correlates, once more, to a reputation for reliability. If we accept the starting points of the extremes, namely highest quality and least biased, in the center, to most biased and lowest quality, at the far left and far right, these in-between data points combine to form what appears as a Gaussian curve, or PDF. In the second post, I posited that the points along that curve represent, for each source, a maximization point, with the curve as whole depicting the combined equilibria associated with maximal readership or revenue, however derived, along the chart’s two dimensions.
Part III: A Tale of Two Analyst Cohorts*
Version 5.0 of the chart followed a different evaluative process. Whereas the prior versions were all Vanessa Otero’s personal creation, her goal, and the goal of Ad Fontes Media, is to create a mechanism by which to ensure to her own readers and followers that the graphic itself is unbiased or minimally biased. At the risk of infinite regress, her goal is to commit across the x axis to being ideologically centrist, and along the y axis to being maximally reliable. Accomplishing this means ensuring that the data she has generated prove replicable in large numbers even if individual readers might disagree with specific media source placements.
In the second post, I speculated about some possible approaches to replicability. Beginning with this newest version, which also includes the interactive version 5.0, Otero pursued what seems the most tractable method, namely having a hired team of committed news analysts that she trained according to her specific ratings methodology using the two dimensional graphic.
Version 5.1 conveys the same data as 5.0, but it does so with fewer overlayed media sources on each page. This allow educators, and other users, access to more specific information without the clutter associated with the dense plotting of sources on 5.0. The customization function on the interactive chart further allows users to visualize individual article ratings attributed to each selected media source.
Thus far, version 5.0 exhibits a notable visual change. Whereas the earlier graphics evince a Gaussian shape, these charts more obviously reflect the flattened shape of an inverted “V.” The flattening arises from some north-south scoring compression as compared with a broader spread along the vertical axis for the earlier versions. Because the compression of versions 5.0 and also 4.0 are partly the product of somewhat arbitrary numerical valuations, there is less to say about it here. The more important aspect is the change from Gaussian to inverted “V,” and this gives rise to the following questions:
(1) Why has the team of Ad Fontes trained analysts altered the shape from Gaussian to an inverted “V”?;
(2) Which shape, if either, more accurately reflects the overall set of relationships among the listed media sources (or, put differently, is one or the other shape, more likely to be “correct”); and
(3) Is engaging a broader cohort of trained analysts, or a smaller cohort of professional analysists, likely to restore the inverted “V” to a more Gaussian shape?
Answering these questions provides the basis for greater insight into the larger Ad Fontes Media project of ranking news sources across the two dimensions of reliability and bias. Before proceeding, it is important to acknowledge that it might not be possible to provide definitive answers to these questions. Between versions 4.0 and 5.0, more than a single variable has been changed. As a result, the best I can offer are hypotheses coupled with explanations. As this project further develops, newer data might confirm or falsify these hypotheses, thereby providing the basis for greater insight and analysis.
The current ratings project entailed identifying qualified reviewers, after an on-line fundraising campaign and advertising for hiring the analysts themselves. Those hired came with a range of backgrounds and professions, and each analyst self-identified as conservative, moderate, and liberal. Originally, Otero hired twenty analysts, and presently Ad Fontes Media continues to employ nine. Although the search terms required some college, most analysts in the original cohort held a four-year degree, with over half also holding a masters, JD, or MBA. Of the nine analysts who remain, two hold a four-year degree, six also hold a masters, and one holds both a JD and masters.
For each article of the thousands reviewed, a cohort comprising one from each ideological category evaluated and ranked the source. Otero provided several finely grained grading criteria, and an ultimate criterion, along the y and x axes. When evaluating each article, the analyst made the determination as to the effect of the separate grading considerations on the overall awarded grade. A question for future rankings is whether the finely grained evaluative criteria, if automatically transformed into a total score, would vary those results in a meaningful way. From these data it might also become possible to ascertain which of the more finely grained criteria, if weighted in a particular manner, could potentially restore the graphic’s Gaussian shape. This is helpful, especially if, as explained below, neither that or the inverted “V” shape can be claimed as necessarily more accurate.
From the ranked evaluations, using the evaluators’ own ultimate rankings along each dimension, Otero retained the results unless they represented extreme outliers, which she defined as three or more standard deviations from the mean. The resulting graphic changed some specific media locations. Examples include moving Huffpost upward, and National Review downward, on reliability. By separating out CNN into TV and on-line, the rankings moved CNN TV upward and slightly left, and CNN on-line simply upward.
Before offering a hypothesis on the change to an inverted “V,” it is worth observing that the revised shape largely corroborates the revealed analytical relationships demonstrated in Otero’s earlier-generation presentations. The overall relationship of commitment bonding represented along the y axis to the ideological placement represented along the x axis, remains, even as the detail as between these two visual presentations of the graphic has changed. The data are thus far consistent with the earlier hypotheses concerning the tradeoff between reliability and ideological bias and with the depiction of the resulting locations along each dimension as representing equilibria respecting one or more of the identified maximands.
With respect to version 5.0, there are two possible explanations for the transformation from a Gaussian curve to an inverted “V.” My essential thesis is that as you gain expertise in what Ms. Otero aptly describes as lateral reading, meaning deliberately reading for any given news story across the ideological media landscape, coupled with professional level news comprehension, you are more apt to give greater credit for veracity to the best sources opposite your ideological location even as those sources exhibit relatively modest bias. Conversely, even with media-bias training, those with relatively less lateral reading expertise are more apt to marginally discount the veracity or reliability of sources opposite their ideological location that exhibit even relatively modest bias. For any three-reader cohort—liberal, moderate, conservative—it is reasonable to assume that two will occupy, in general for a given topic, one or the other ideological position. Moderate analysts are not necessarily moderate on each issue, but rather they are apt to hold a mix of relatively liberal and conservative positions, encouraging them to self-identify as moderate, rather than as liberal or conservative overall. (For a related blogpost explaining that moderate does not imply unprincipled, see here.)
As a result, for any three-person cohort, two thirds will tend to exhibit a bias in the same general direction. One distinct feature of the independent ratings is that the more finely-grained the inquiry, the less bias the readership tends to exhibit. This, once more, raises the question whether the aggregate independent scoring, as opposed to an imposed cumulative scoring based on the separate component grades, will effect a more Gaussian shape. It also implicates whether simply having a larger analyst cohort is apt to have a similar benign effect by replicating, more generally, reviewer gradations along the ideological, or x, axis. A larger reviewer cohort might produce more centrist reviewing overall, thereby increasing the likelihood of replicating the shape imposed by an expert lateral reader, such as Ms. Otero.
A possible consequence of this newer process might be that as you expand the base of analysists below the top-most news-proficient analysists, or with a mix of skill levels, you are apt to see more analysts on each side of the ideological spectrum, left and right (including moderates who appear left or right on an issue-specific basis), pulling down the opposing side’s sources between the apex of maximal bonded commitment and extreme outlier junk news. This might not be true of all hired analysts, but if it is true of most, the effect on the curve could be significant. If this thesis is accurate, then a team of trained analysts who are strong, but not universally expert, in lateral reading, are apt to pull the Gaussian shape into the shape of an inverted “V.” Conversely, a team of increasingly professional news analysts, meaning a group that has spent a large part of their professional careers, or who have historically internalized a norm of lateral reading, would be more inclined to flesh out the inverted “V” shape toward a Gaussian shape.
This thesis, if sound, implies, per the second query, neither the Gaussian nor “V” shape can be identified as more accurate. Different reviewers routinely generate different ratings results. Consider, for example, reading Netflix movie rankings, on-line book reviews, or Rotten Tomatoes movie scores. In each instance, professional reviewer scores and general audience scores routinely diverge. And yet, they generally fail to do so dramatically. In general, my impression is that these different reviewer cohorts will still tend to rank the same or closely overlapping books or movies as best or worst. The variation is apt to be most pronounced for those the ratings in-between. And in the context of the media bias chart, it is the in-between ratings that effect the difference between a Gaussian shape versus an inverted “V.”
Although thus far I lack empirical data to corroborate this insight, it is possible that future generations of the media bias chart might provide two independent sets of helpful data. The original graphic was the product of Vanessa Otero’s subjective impressions concerning reliability and bias, which she then worked toward rendering more scientific with specific valuations as the chart project progressed. Beginning with version 5.0, Otero has to translate those intuitions into an algorithm that transformed a series independent analyst scores on each dimension into a chart location. Those valuations are likewise new to this version of the chart, and those are apt to be adjusted going forward. Although the cohort of trained analysts, most of whom lack Otero’s expertise in lateral reading, might affect the shape of the curve, so too the scoring in the algorithm can have this effect. This includes, but is not limited to, the difference between having each reviewer give an ultimate ranking versus having the more finely grained sub-rankings automatically transformed into each reviewer’s final ranking.
Another helpful data set will likely arise from a separate, related project. As this has all been proceeding, Ms. Otero has also developed a terrific body of educational software that will allow teachers at various instructional and age levels to teach students to engage with their own news-rankings analysis. The resulting data has the potential to test this post’s tale-of-two-cohorts thesis. Some faculty, and some students, are apt to be intense lateral readers, whereas others will be dipping into these waters only occasionally, while focusing principally on other academic pursuits. The latter will be trained lateral readers, whereas the former might be closer to expert. And once more, the comparative data might, over time, might provide helpful information concerning the relationships that give rise to the shape of the curve.
Otero’s version 5.1 interactive chart allows users to identify the particular sources of interest to them and the relationships among those sources on the chart. Eventually, Otero hopes to offer increasingly fine-grained analysis, per story, as opposed to offering only rankings for media sources as a whole.
The Ad Fontes Media project is a long game. Over time it might become possible to gain more finely grained data that draws out differing shapes based on reviewers drawn from increasingly specific reviewing cohorts, or cohorts of different sizes. Such data might inform whether the earlier, Gaussian, distributions better represent “expert” analyses, individually or aggregated, and whether, conversely, the flatter descents from the unbiased, or minimally biased, apex better represent well educated and trained novice reviewers. As such information becomes available in the years or decades ahead, we might also learn which of these presentations, or reviewer cohorts, provides more meaningful data to particular audiences for the media bias chart. There is much to look forward to as this project further develops.
I welcome your comments.
Final notes:
The author is a member of the Ad Fontes Media Advisory Board. Special thanks to Vanessa Otero for her input in an earlier draft of this post. Images are reproduced with the permission of Ad Fontes Media.
*Credit also goes to Charles Dickens for inspiring the Part III subtitle.