Centre for Science and Technology Studies Centre for Science and Technology Studies 2333AL Leiden Zuid Holland 31715273909
  • Blog
  • Monitoring how science finds its way into society: measuring societal impact through area-based connectedness (ABC)

Blog archive

This is the archive of our old blog. Please visit our new Leiden Madtrics blog.

Monitoring how science finds its way into society: measuring societal impact through area-based connectedness (ABC)



The impact of scientific research on society has many dimensions. These cannot be captured by a single measure. In addition, research is a collective effort and depends on the collaboration of many actors. The impact research has on society should therefore be measured at the level of this collective effort.

In this post I propose a new approach to frame and to quantitatively monitor societal impact. I refer to this approach as ABC (Area-Based Connectedness) to society. The core idea of this approach is that actors should be assessed on societal impact through their contribution to societally connected research areas.

Societal impact as science-society connectedness

If we want to quantify the relation between societal issues and research, we should look at existing signals between these two: signals of connectedness. A research area is more connected to society than another if these signals are stronger or denser.  Connectedness to society then describes the extent to which research finds its way into society or the extent to which research is mobilized as societally relevant.

Research is a collective process

The way research develops depends on the collective effort and collaboration of many actors. The impact research has on society should therefore be measured at the level of this collective effort. In order to define this effort, we need to identify coherent research areas. Such an area is not necessarily represented by a community of people. After all, researchers may be involved in a variety of areas, for instance because of their position (e.g., a group leader or professor) or expertise. An area in this context would rather be represented by a set of publications on a specific topic.

Research areas

Recent research at CWTS (Waltman & van Eck, 2012) has demonstrated that scientific output can be algorithmically structured into groups of coherent clusters of publications using direct citations, i.e., one publication citing another. Such clusters are independent from journals in which papers are published or any other superimposed classification. Such a structure is self-organized around topics, themes, methods, and so on. I will interpret such clusters as (research) areas.

 

picture1 2

The above map illustrates the landscape of science by around 4000 areas positioned on the basis of their citation traffic. If two areas cite each other often, they are in each other’s vicinity. If there is hardly any or no traffic, they are plotted on a distance. The size of a circle represents the volume (number of publications). The colors represent a clustering of areas into broad fields.

This structure will be our starting point to monitor societal connectedness. The areas represent what researchers do and, overall, I consider this a way to describe how research is organized. This provides the lens through which we look at signals of connectedness with society.

What signals of connectedness can we distinguish?

The signals we may pick up to monitor connectedness between science and society differ from one area to the other. I suggest a few but there will be more. I propose in a first approach the following signals retrieved from Web of Science, PATSTAT or Altmetric.com:

  • A publication (co-)authored by non-academic organizations (companies, peripheral hospitals, social actors), retrieved from Web of Science;
  • A publication cited by at least one patent (PATSTAT);
  • A publication mentioned in at least one policy document (Altmetric.com);
  • A publication mentioned in a newspaper/news item (Altmetric.com);
  • A publication mentioned on Twitter (Altmetric.com).

In the next step of the analysis these signals are implemented. For instance, I will discuss how a paper being mentioned in a policy document can be considered as a signal of connectedness.

From signals to indicators of connectedness with society

Areas differ in volume (number of publications on a topic), so we need a relative measure: the proportion of papers in an area with some signal. For instance: the proportion of papers mentioned in a policy document. Within an area (a set of publications on a topic), we calculate the proportion or share of papers being mentioned. This share varies from one area to the other. We consider one area more connected to society than another through this signal if the share is bigger. In this way, each area will be characterized by the strength of this signal, which is a measure of connectedness.

ABC to society

After having obtained such measures for each area, we can use them to characterize an actor’s output. The ABC to society of an actor by a specific signal will be based on the distribution of the actor’s output over the different areas combined with each area’s connectedness to society.  For instance, the connectedness of an actor regarding (public) policy will be based on the number of publications of the actor in different areas and the strength of this signal, i.e. mentions in policy documents, in those areas. It should be noted that it will not be measured by the share of papers by this actor mentioned in policy documents. There are at least two reasons to define this indicator indirectly through the area and not directly through the actor’s share of the output itself. Firstly, in the example of references in policy documents, I consider the choice of a particular publication to be arbitrary. Choosing publication X to represent the background for a public issue, is after all a choice made by the author of a policy document without him/her being involved in the scientific discourse itself. Secondly, some actors may be more visible for policy or available for industry. In some cases, the fact that they are mentioned represents their visibility or social skills and not their contribution to the research. I am not saying that this skill should not be valued but it is a different contribution than the one I am measuring. The ABC-to-society signal I propose to measure the policy connectedness should be insensitive to such variations. Furthermore, the overall ABC to society needs a baseline to determine the strength of the calculated ABC. In the box shown below, I provide an example of the calculation of ABC to society.

rekenvoorbeeld PNG

A case of three Dutch organizations

As an exemplary case study, I implemented the approach for three Dutch mission-oriented organizations, using five different signals. I used the CWTS publication level classification containing over 4000 clusters to create a profile for the Netherlands Organization for Applied Scientific Research (TNO), the National Institute for Public Health and the Environment (RIVM), and the Netherlands Cancer Institute (NKI). TNO operates on the interface between science and industry to boost innovation. RIVM supports government and society in improving health and the environment. NKI is an international center of cancer research and clinical expertise, development and training. The chart below depicts the extent to which research at the three organizations is connected to society through five ABC to society dimensions (2014-2017).

 

picture2

Figure 0‑1Five ABC to society scores for three Dutch organizations 2014-2017

All three organizations have a clear societal dimension by mission, which is well represented by the ABC scores overall but with different dimensions standing out. TNO’s research connects to society most prominently through policy and industry. NKI’s research is connected through almost all dimensions but stands out through patent citations, most likely in the context of pharmaceuticals. Finally, RIVM’s research is highly connected to society by policy, most likely regarding health issues.

Is assessing societal impact as simple as ABC?

In summary, I propose an alternative way to measure societal impact of scientific actors. The core message is that societal impact should not be assessed at the level of the single actor, because the impact is not depending on the effort of individual actors. Such impact is a result of the collective effort in research. The approach I propose measures the societal impact in terms of the connectedness at the level of research areas. Actors can then be assessed by their contribution to those areas, the "ABC to society". Assessing societal impact of science actors is challenging and far too complex to be established by a single indicator. ABC to society may guide the discussion into a more promising approach.

References

Waltman, L., & Van Eck, N.J. (2012). A new methodology for constructing a publication-level classification system of science. Journal of the American Society for Information Science and Technology, 63(12), 2378-2392. arXiv:1203.0532.


About Ed Noyons

Everard (Ed) Christiaan Marie Noyons is a senior researcher at the Centre for Science and Technology Studies (CWTS) at Leiden University. He is a member of the Engagement & Inclusion and Evaluation & Culture focal areas. He is also a board member of CWTS BV.


No comments

Mandatory fields
    Share on:
    Subscribe to:
    Build on Applepie CMS by Waltman Development