Centre for Science and Technology Studies Centre for Science and Technology Studies 2333AL Leiden Zuid Holland 31715273909
  • Blog
  • The end of gender disparities in science? If only it were true...

Blog archive

This is the archive of our old blog. Please visit our new Leiden Madtrics blog.

The end of gender disparities in science? If only it were true...

Gender, again?

There has been a resurgence of interest in the last few years regarding gender disparities and bias in science. Recent research has examined, inter alia, lower rates of participation of women in the scientific workforce, the lower rates of female production of scientific work, and lower scientific impact—measured by citations—of female-authored work. Contextual effects—such as effects of funding, environment, and other responsibilities of women—have been used to explain the observed disparities. Bias has been investigated through clever experimental designs. For example, provided with applications identical except for explicitly male/female names, directors (both male and female) of laboratories were more likely to hire the supposed male for laboratory manager positions and offered the male applicants higher salaries (Moss-Racusin et al., 2012). Similarly, an analysis of the contributions of 1.5 million computer programmers publishing on the GitHub platform showed that, when the first name of authors provided an indication of gender, codes produced by women were rejected more by their peers, whereas the opposite was observed when only the initials of the first name of the authors were presented (Terrell et al., 2016).

Encouraging results, but...

Recently, Elsevier—the world's largest commercial publishing house and owner of the Scopus bibliometric database—contributed to the debate with a report entitled “Gender in the Global Research Landscape” (Elsevier, 2017), in which it emphasizes the evolution of the place of women in research over the past two decades. Some results it contains are very encouraging: in each of the countries analyzed, the proportion of women active in research—defined as those, based on Scopus Authors’ Profiles, who contributed to at least one scientific article indexed in the Scopus database—increased between 1996-2000 and 2011-2015, and reached parity in some countries such as Brazil and Portugal. It also presents data on citations, concluding that parity has been achieved: globally, female papers are cited at nearly equal rates to men and exceed male rates in countries such as the United States. While this may sound like good news—and it would be! —this demonstrated parity should be received with a healthy dose of skepticism, given the number of studies showing the opposite. This short analysis aims, on the one hand, to explain the methodological artefact behind the surprising Elsevier results and, in addition, to provide evidence of the gap that persists in terms of citations received.

What is an article "written" by a woman?

The Elsevier findings hinge on a deceptively simple question: What is an article written by a woman? Or by a man? The answer to this question was obvious when authorship was mainly an individual matter, but has been complicated by the ubiquity of collaboration. Figure 1, based on data from the Web of Science for the period 2008-2015 to which we applied our first name-based disambiguation algorithm (N = 8,142,048) (Larivière et al., 2013), illustrates this difficulty. Indeed, we note that nearly 60% of articles have at least one female author and one male author. More than a third of the articles have only male authors, whereas about 6% of the articles have only female authors. Also noteworthy are the differences in terms of citations (ARC = Average of relative citations, a citation indicator that normalizes scores by discipline and year of publication): while articles containing both male and female authors have the highest citation rate (1.11), those written exclusively by women are cited lower than those written exclusively by men (0.86 vs. 1.05). However, when considering articles with at least one man (94% of articles) and those with at least one woman (65%), scientific impact is identical (1.08).

Figure 1. Number of articles and citation rates of the various cases associated with the assignment of an article to a woman or a man, 2008-2015

The mirage

These results demonstrate the problem underlying the Elsevier report. The report relies on "full counting", in which authors are credited for each article on which they are an author. The consequence is that there is high overlap between the sets of “male” and “female” papers, with the overwhelming majority of articles to which women contribute also having men among their authors. In order to overcome this limitation—and considering the dominant role of authors documented in the literature (Larivière et al., 2016)—we compiled, with the dataset used in Figure 1, the differences in citation rates for first and last authors, by gender. The results reinforce previous research on gender disparities: fewer citations are received for articles with women at the beginning of the author list (0.98 vs 1.09) and at the end (1.00 vs 1.08). The gender gap still exists in terms of scientific impact.

A glass ceiling on citations?

Despite its well-known limitations, the journal impact factor is perceived as an indicator of the reputation and visibility of a journal. This is due, among other factors, to the relationship between impact factors and acceptance rates, with high impact factors journals having significantly lower acceptance rates. In light of the citation disparities depicted above, one might expect, therefore, the gender gap in terms of impact factor and citations to be similar. Yet our results on the differences in male and female citations and impact factors show that, for both first and last authored papers (Figure 2), the average impact factor for men and women are much closer to parity than citations and that, systematically, the gap in citations is much greater than the gap in impact factors, and always in favor of men. Moreover, in some disciplines—such as Earth and Space, Biology, Engineering, and Social Sciences—the two indicators show opposite trends: while women publish in journals with higher impact factors than men, they obtain lower citation rates. The inset of Figure 2 shows the pattern observed at the level of all disciplines combined: while in both author positions women’s papers normalized citation scores are lower than those of their impact factors, the opposite is observed for men’s papers.

Figure 2. Male / female differences in normalized impact factor and average of the relative citations for articles written as first and last author, by discipline, and mean values obtained all disciplines combined (inset), 2008-2015

To better understand this phenomenon—similar to the Matilda effect documented by Margaret Rossiter nearly 25 years ago, where women are less recognized than their male counterparts (Rossiter, 1993)—we have grouped articles by women and men according to nine classes of standardized impact factors, and compared the citation rates of the articles in each of these classes. The results demonstrate that the largest citation disparity occurs in those journals of the highest impact factor (Figure 3). This shows that, even when women publish their works in high impact factor journals, this work is cited lower, both in absolute and relative terms.

Figure 3. Average of the relative citations of the articles, according to the impact factor class, for articles written as first and last author, 2008-2015

The prevailing winds

Despite the progress towards equality depicted in the Elsevier report, there remains a bias in the reception of female work, particularly in some of the most prestigious scientific journals. Even when subject to the same process of peer review—seen as the gold standard for evaluating quality—the work of women is cited at a lower rate, both in the position of first and last author. It may be argued that this is the result of differences in topics or other properties of the paper. However, studies have shown that, when controlling for all the properties of a paper, women’s work is grossly undercited (Caplar et al., 2016). The mounting evidence reinforces the relative and absolute neglect of women’s work, despite progress towards equity in science.

Caplar, N., Tacchella, S., Birrer, S. (2016). Quantitative evaluation of gender bias in astronomical publications from citation counts. Retrieved from: https://arxiv.org/abs/1610.08984

Elsevier. (2017). Gender in the global research landscape. Retrieved from: https://www.elsevier.com/__data/assets/pdf_file/0008/265661/ElsevierGenderReport_final_for-web.pdf

Larivière, V., Desrochers, N., Macaluso, B., Mongeon, P., Paul-Hus, A., & Sugimoto, C. R. (2016). Contributorship and division of labor in knowledge production. Social Studies of Science, 46(3), 417-435.

Larivière, V., Ni, C., Gingras, Y., Cronin, B., & Sugimoto, C. R. (2013). Global gender disparities in science. Nature, 504(7479), 211-213.

Moss-Racusin, C. A., Dovidio, J. F., Brescoll, V. L., Graham, M. J., & Handelsman, J. (2012). Science faculty’s subtle gender biases favor male students. Proceedings of the National Academy of Sciences, 109(41), 16474-16479.

Rossiter, M. W. (1993). The Matthew Matilda effect in science. Social Studies of Science, 23(2), 325-341.

Terrell, J., Kofink, A., Middleton, J., Rainear, C., Murphy-Hill, E., Parnin, C., & Stallings, J. (2016). Gender differences and bias in open source: Pull request acceptance of women versus men (No. e1733v2). PeerJ Preprints.


About Vincent Lariviere

Professor at Université de Montréal and visiting professor at CWTS. At Université de Montréal, Vincent holds the Canada Research Chair on the Transformations of Scholarly Communication and is associate vice-president for planning and communications.

About Cassidy Sugimoto

Associate professor at Indiana University Bloomington. Cassidy’s research focuses on scholarly communication and scientometrics.

1 comment

Mandatory fields
  • Sarah Huggett March 30th, 2017 5:36 am
    We appreciate the interest in Elsevier’s new Gender in the Global Research Landscape report and the analyses therein. Elsevier is a global information analytics company working to advance science and healthcare, so an important goal of our report was to apply data and analytics to address issues of gender disparity and encourage others to examine and discuss these critical issues via a data-driven approach as well.
    In developing the Elsevier report, we considered how best to examine issues related to contributions to scholarly work. Contributorship is a complex issue: unless each author’s roles(s) is/are clearly stated (e.g., using a schema such as Project CRediT), it is impossible to establish who did what on a paper or how much work each researcher contributed by simply examining the authorship byline. We chose to use the full-counting method because we consider it the most transparent way to attribute scholarly work; however we do acknowledge within the report the limitations to this approach as well as other studies using different methodological approaches that have produced different findings. Other methodologies have drawbacks as well. For instance, attributing papers only to first or last authors overlooks the contribution and impact of all other collaborating authors. The approach is not applicable across all fields either, notably those in which author order is traditionally alphabetical (e.g., mathematics and economics). And while there have been several attempts to propose various fractional counting schemes (e.g., equal weight attributed to each author vs. custom weights attributed to each author depending on their position on the paper), none has yet emerged as an established best practice. Furthermore, by their very nature, fractional counting and first/last authorship-only analyses may provide disincentive to collaboration. For the former, more credit goes to each author the fewer authors there are; for the latter, there is no recognition for middle authors. In the long term, widespread use of these techniques in research evaluation could discourage collaboration, which could negatively impact the advancement of team science.
    As the analysis by Drs. Larivière and Sugimoto notes, when we apply a full counting method to scholarly output counts for women and men researchers, we find there is little difference between the impact of women’s and men’s output. In terms of citations, our report does not claim that parity has been achieved between women and men researchers; rather, we conclude that the impact of men’s and women’s scholarly papers is similar. The two statements are subtly but importantly different. We also do not perform analyses on ‘female papers’ or ‘male papers.’ As noted by Drs. Larivière and Sugimoto, most papers have both men and women among their authors, so such analysis would have been misleading. Citation analysis is only one aspect of our report: in many of the other analyses included, the findings do reveal disparities between the genders, particularly in terms of international collaboration, international mobility, and the representation of women among researchers in certain scientific disciplines.
    Our report analyses reveal that among subjects areas in which one gender is underrepresented in author bylines, that same gender is also underrepresented in authorship positions that tend to signify a leadership role (first and/or corresponding author). The analyses of citation rate disparity by gender among first and last authors by Drs. Larivière and Sugimoto are intriguing, especially when examined by journal impact level. Their analyses find that the gaps in citation performance appear to be smaller in those fields with higher representation of women. It is clear that both impact and leadership are topics ripe for further investigation.
    All together, the analyses in the Gender in the Global Research Landscape report provide a broad picture of research performance through a gender lens and highlight that there have been gains for women in the global research landscape over time, albeit incrementally and unevenly. The report also calls attention to the fact that there remain important questions regarding persistent disparities and that establishing a robust framework for addressing gender issues through evidence-based analyses is critical to advancing policy to address them.
    Sarah Huggett
    Analytical Services Product Manager
    Elsevier | Research Intelligence
    On behalf of the Gender in the Global Research Landscape report team
Build on Applepie CMS by Waltman Development