Research Performance of World Universities: An Exploratory Bibl

Save exceptions stated by law no part of this publication may be reproduced in any form, by print, photoprint, microfilm or other means, included a complete or partial transcription, without the prior written permission of the author.

Behoudens uitzondering door de Wet gesteld mag zonder schriftelijke toestemming van de recht-hebbende(n) op het auteursrecht niets uit deze uitgave worden vermenigvuldigd en/of openbaar gemaakt worden door middel van druk, fotokopie, microfilm of anderszins, hetgeen van toepassing is op de gehele of gedeeltelijke bewerking.

Table of Contents

Executive Summary

1. Introduction

2. Data and methodology

2.1 Assignment of articles to universities; accuracy

2.2 Universities analysed in this paper

2.3 Indicators calculated

2.4 Research disciplines and their coverage by the WoS

3. Results

3.1 General patterns

3.2 Differences between European and US universities

3.3 Differences across European national academic systems

3.4 A university’s disciplinary specialisation versus its citation impact

3.5 University disciplinary profiles

4. Discussion and conclusions

4.1 What is a research university?

4.2 Under-representation of social sciences and humanities

4.3 US versus European universities

4.4 Differences in the structure of national academic systems

4.5 Citation impact of specialised versus general universities

4.6 University profiles and disciplinary classification systems

4.7 Network analysis

4.8 Further data collection and verification

Acknowledgements

References

Executive Summary

This paper analyses a series of bibliometric indicators of the research performance of universities, derived from the Web of Science, published by Thomson Scientific. It highlights important factors that should be taken into account in the interpretation of bibliometric rankings of universities, and presents general patterns in two partly overlapping sets: a set of the 386 most frequently publishing world universities, and a set of 529 European universities. It describes data collection and data handling. It illustrates research policy questions that may be addressed in secondary analyses, and proposes further steps towards the creation of a reliable information system on world universities, useful in research management and policy at the institutional, national and supra-national level, and for the wider public. Key notions and findings presented in this paper are:

1.Introduction

As internationalization and globalization in academic research and teaching proceeds, more and more attempts are made to identify top research universities from a global perspective. Universities are more and more competing for research funds, research students and researchers in the global research area. Their reputation as research universities is a crucial factor in such a competitive system. Therefore, members of the international scientific community, officials responsible for institutional, national and supra-national science policies, and the wider public need ‘objective’, ‘reliable’ information about the research performance of universities.

Comparative analyses of the performance of universities at a national level, focusing on particular research fields or disciplines, have been carried out for many years. For instance, in 1995 The US National Research Council (NRC), the working arm of the National Academy of Science and the National Academy of Engineering, published a report presenting a quality rating of PhD programs at 274 US institutions in 41 fields, based on surveys sent to faculty (Goldberger et al., 1995).

The NRC report also presented bibliometric indicators based on publication and citation data extracted from the ISI Citation Indexes, but these indicators were not used by the NRC for ranking purposes. Diamond and Graham (2000) further analysed the NRC data and concluded that “reputational ratings showed a strong positive correlation with citation densities”, in the sense that the institutions appearing in the top of the former tended to be highly ranking on the latter as well. However, younger and smaller “challenging” institutions tended to have higher positions in the citation impact rankings than in the reputational rankings.

A recent phenomenon is the compilation of rankings of universities from a supra-national or global perspective. For instance, the European Commission published in the recent European Science Indicators Reports listings of European universities presenting their bibliometric scores. Global rankings of universities were published by the Jiao Tong University in Shanghai (SJTU 2005) and by Times Higher Education Supplement (THES 2005). The former was to a large extent based upon bibliometric indicators, and partly upon counts of prizes and awards. In compiling the THES rankings, expert opinions collected from surveys constituted the most important indicator, while bibliometric indicators played a less important role. For a thorough review of these two rankings, the reader is referred to van Raan (2005).

The study presented in this paper provides a series of bibliometric indicators of the research performance of universities, derived from the Web of Science, published by Thomson Scientific. Research universities produce knowledge, contribute to the advancement of scientific/scholarly knowledge. These contributions are normally embodied in research articles, published in the open, serial literature and subjected to criticism of colleagues. A base assumption underlying a bibliometric approach is that one can learn about scientific activity and performance by analyzing the scientific literature.

This paper aims at contributing to a public information system on universities, particularly research universities, useful in research management and policy at the institutional and national level, and for the wider public.

Outcomes of citation analysis are often presented to the ‘outside world’ in the form of rankings of entities such as individual scholars, research departments or institutions. Rankings are appealing, because they look simple, and are compiled in many other domains of human activity as well (e.g., in sports, popularity polls, etc). Although the need for policy makers and the wider public to obtain insight into the scholarly quality of research activities in the various universities is legitimate, scholarly research quality is not as straightforwardly measured and ranked as performances in many other societal domains.

Rankings are in a sense one-dimensional: entities are ordered by descending score on one particular statistic. Of course one can design some compound statistic, taking into account scores on several aspects rather than one. But then the problem is: how should one weight the importance of the various aspects? Moreover, rankings disregard how the performance of one entity depends upon that of others. In this paper three bibliometric indicators play a key role, measuring article production, disciplinary specialisation, and citation impact, respectively. It is assumed that all three aspects are significant aspects of research performance, but no attempt was made to attach weights to them.

The structure of this paper is as follows. Chapter 2 deals with data collection and methodological issues. It describes how research articles extracted from the Web of Science were assigned to universities on the basis of information about the institutional affiliations of the authors. It specifies the two sets of universities analysed in this paper, and makes a distinction between general and specialised universities. Next, it defines the five indicators that were calculated in the study. Finally, it describes the classification of articles into 15 research disciplines applied in this paper.

Chapter 3 presents the outcomes of a number of analyses based on the two sets of universities. It focuses on differences between US and European universities and national academic systems. It examines statistical relationships between concentration or specialisation of research activities on the one hand, and research performance measured by citation impact on the other. It also shows typical bibliometric, disciplinary profiles of universities, both from an internal university and from a global viewpoint. Chapter 4 discusses the outcomes and draws conclusions.

2. Data and methodology

2.1 Assignment of articles to universities; accuracy

For European universities the Membership Directory of the European University Association (EUA) was used as a starting point. Since this list did not include all European universities, it was expanded during the project. The data collection process aimed at defining the article output of European universities publishing at least some 500 papers during the time period 1997-2004. For non-European universities the process identified the articles of the 200 most frequently publishing universities. Articles were assigned to universities on the basis of the information on the institutional affiliations of authors, included in the corporate address field. Two rounds were carried out.

In a first round, papers were selected with the name of a university (and its major departments) mentioned explicitly in the address. Name variations were taken into account. For instance, Ruprecht Karls University is a name variant of the University of Heidelberg, TUM of the Technical University München; and Université Paris 06 of Université Pierre et Marie Curie. For European universities, this round took into account all variations occurring 5 or more times. For non-European universities this threshold was set to 25.

In a second round, additional papers were selected from affiliated, teaching hospitals on the basis of an author analysis. This round added to a particular university’s article output selected in the first round papers from affiliated hospitals, published by authors who did not explicitly mention this university’s name in their institutional affiliation, but who showed strong collaboration links with that university, as its name appeared in the address lists of at least half of their papers. In this way, for instance, a part of the papers containing the address Addenbrookes Hospital was assigned to University Cambridge, and a part of the papers with the address Hospital La Pitié Salpetrière to University of Paris VI, and another part to University of Paris V.

Since the de-duplication and counting process of European universities took into account only name variants occurring 5 or more times, an overall accuracy rate for this group of universities is estimated to be about 95 per cent. It is somewhat higher for universities with a large number of published articles than it is for universities with smaller publication volumes. For non-European universities it is around 90 per cent. It is important to note that the data were not verified by representatives of institutions, except in the case of Flemish universities.

Data on Flemish universities were verified by institutions themselves on a paper by paper basis. There data were kindly provided by the Steunpunt O&O Statistieken at the Katholieke Universiteit Leuven. In this way, the article output of French and Dutch speaking universities with the same name when translated in English, could be disentangled (e.g., Free University Brussels may relate both to the Dutch speaking Vrije Universiteit Brussel and the French speaking Université Libre de Bruxelles). This task cannot be carried out properly merely on the basis of institutional names in corporate addresses.

The University of London was split into colleges (e.g., University College London) on the basis of an analysis of the names appearing in the department field. The same was done for other universities whenever appropriate, e.g., University of Wales, and National University of Ireland. Articles from the University of London and other compound main organizations that gave no information about particular colleges or sub-parts were assigned to sub-parts on the basis of an author analysis, similar to that applied in assigning papers from affiliated hospitals to universities. Data for US universities took into account the existence of university systems. For university systems the constituent universities were treated separately (e.g., University of California at Los Angeles; University of Texas at Austin).

2.2 Universities analysed in this paper

This paper analyses two sets of universities. The first and most important one is the set of universities that published more than 5,000 articles in WOS journals during 1997–2004, or on average more than 625 papers per year during this time period. It contained 386 universities, and is denoted as the global or world set, containing world universities. In view of the collaboration among institutions, resulting in co-publications by scientists from two or more institutions, it would be more precise to state that the universities contributed at least one author to more than 625 papers per year. Technically, this number is denoted as an integer count.

An alternative method for computing the article output of a university takes into account co-publications with other institutions by applying a fractional counting scheme. For instance, an article with 2 addresses from institution A and 1 from B is assigned to A for the fraction 2/3 and to B for 1/3. For universities in the global set, it was found that, on the average, the ratio of the number of fractional and integer counts of a university’s papers amounts to 0.55. As a result, universities studied in this paper have published at least about 350 fractionally counted papers per year.

It can be shown that during the past decades, the global scientific community published about 0.6 articles per unique active scientist per year (see Moed, 2005). Assuming that this figure is also valid for universities, it follows as a rough approximation that the number of active (i.e. publishing) researchers in a university in a year is slightly lower than the number of articles to which the university contributed (i.e., to its total number of published articles using an integer counting scheme). Roughly speaking, the universities in the global set at least some 600 active (i.e. publishing) researchers in a year.

A second set of universities analysed in this paper is a set of 529 European universities publishing at least 500 articles during 1997-2004, or on average 65 articles per year. The number of active (publishing) researchers is at least about 60. There is an overlap between the European and the global set: 172 European universities are included in both sets.

It is useful to distinguish between general and specialized universities. General universities cover a wide range of scientific/scholarly disciplines. A typical example is a university that offers courses and carries out research in virtually all domains of science and scholarship. Specialized universities are mainly active in a limited number of disciplines. Often – but definitely not in all cases – their name reveals the disciplines on which they focus. Typical examples are technical, medical, and agricultural universities.

Although general universities show less concentration of research activities among disciplines than specialized universities, they do not necessarily have the same level of activity in all disciplines. They may be more active in some disciplines than in others, and their research profile may reveal a certain specialization, though not as pronounced as in specialised universities. In practice, it is very difficult to draw a sharp borderline between general universities with a certain specialization on the one hand, and specialized universities on the other. The transition from the first to the second group is fluent. Section 2.3 describes a disciplinary specialization index that was calculated for all universities involved in this study.

2.3 Indicators calculated

The bibliometric measures calculated in the Shanghai ranking study (SJTU 2005) were the number of highly cited researchers in a field, the number of articles published in the journals Nature and Science, and the number of articles in journals that were processed by Thomson Scientific for the Web of Science. The THES ranking study (THES 2005) calculated citations per faculty.

An overview of the pros and cons of a number of bibliometric indicators are presented in Table 1. The indicators calculated in this paper are summarized in Table 2. The first indicator, denoted as article output, is defined as the number of articles published during a particular time period in journals processed for the WoS. Article types included in the counts are full articles, letters and reviews. Other types, such as editorials, discussion papers and meeting abstracts are not included.

A disciplinary specialisation Index for a particular university is calculated in the following manner. First, the number of a university’s papers is determined in each discipline. Next, the distribution of papers across disciplines is compared to the ‘world’ distribution, generated jointly by all universities included in the study. In each discipline a publication activity index (PAI) is calculated as follows. If F denotes the proportion of a university’s papers in discipline D, and  the world share of papers in that discipline (i.e., the proportion of papers published in D by all universities included in the study), the university’s publication activity index (PAI) in D is defined as (F/)/((1–F)/(1–)), following a proposition by Bookstein and Yitzahki (2001).

Next, the PAI values of for all disciplines are summed up, and for each discipline a new, normalised index is defined as the percentage share of its PAI value relative to this sum. In this way, for each university the values of its (normalised) PAI over all disciplines sum up to one hundred. Finally, Pratt’s Index is calculated of a university’s distribution of normalised publication activity across disciplines. Pratt Index ranges between 0 (no specialization at all) and 1 (extremely strong specialization). For further details on this index the reader is referred to Egghe and Rousseau (1990).

The Normalised Citation Impact is defined as the average number of citations per article published from a university, relative to the world citation average in the subfields in which it is active. It is also denoted below as ‘citation impact’ or ‘impact per paper’. Details can be found in Moed, de Bruin and van Leeuwen (1995) or in van Raan (1996). A value of 1.0 indicates a citation impact equal to the world citation average.

2.4 Research disciplines and their coverage by the WoS

In this paper, a classification of research articles into 15 disciplines is used, based upon a grouping of journal categories available in the Web of Science. It is presented in Table 3 below. Although journal categories were primarily categorised according to their cognitive contents, an analysis of reference patterns in the papers in journals assigned to a category played an important role as well. Journal categories that revealed similar cited reference characteristics were grouped, particularly those with similar percentages of references to papers published in journals covered by the Web of Science.

In this paper an attempt was made to separate the more clinically oriented biomedical specialties from the more basic biological sciences, and to roughly discriminate in the latter between biological sciences primarily related to humans and those predominantly dealing with animals and plants. A group of journals dealing with applied research in physics and chemistry was taken as a separate discipline, apart from the more basic oriented disciplines physics and chemistry.

In social sciences, psychology and psychiatry constitute a separate discipline. Other social science journals primarily dealing which medical or health–related research were grouped into a field other social sciences primarily related to medicine and health, including public environment and occupational health, nursing, sport sciences, rehabilitation, substance abuse, family studies, geriatrics, health policy and several other journal categories. Economics constitutes a separate discipline as well. The field other social sciences includes sociology, education, political sciences, anthropology, geography, internal relations and several smaller journal categories.

Humanities and arts includes the field of law. Other major journal categories in this main field are literature, history, art, classics, language and linguistics, philosophy, archeology, poetry, dance, and music. The group ‘Other’ comprises the journal category ‘multi–disciplinary’. For more details, the reader is referred to Moed (2005, p. 187–189).

The Web of Science (WoS), published by Thomson Scientific (formerly the Institute of Scientific Information (ISI)) does not claim to have complete journal coverage, but rather to include the most important journals. Its founder, Eugene Garfield (1964, 1979), developed a powerful and unique criterion for expanding the database beyond the core of journals whose importance to a given field is obvious: the frequency at which journals are cited in those sources that are already included in the index.

The extent to which citation analysis based on the Web of Science can be validly applied in all domains of scholarship, particularly in applied and technical sciences, social sciences and humanities, is often debated. A thorough examination of differences in the structure of the written communication systems among the various domains of scholarship, and the extent to which these systems are covered by the Thomson/ISI Citation Indexes, provided the following conclusions (Moed, 2005).

WoS coverage tends to be excellent in physics, chemistry, molecular biology and biochemistry, biological sciences related to humans and clinical medicine; good, yet not excellent, in applied and engineering sciences, biological sciences related to animals and plants, geosciences, mathematics, psychology and other social sciences related to medicine and health; and moderate in other social sciences including sociology, political science, anthropology and educational sciences, and particularly in humanities.

A principal cause of non-excellent coverage is the importance of sources other than international journals, such as books, national journals and conference proceedings. In fields with a moderate ISI coverage, language or national barriers play a much greater role than they do in other domains of science and scholarship. In addition, research activities may be fragmented into distinct schools of thought, each with their own ‘paradigms’. These factors to some extent hamper the creation of a comprehensive publication database primarily based on citation relationships among journals.

3. Results

3.1 General patterns

Figure 1 relates to the 386 universities publishing more than 5,000 papers during the time period 1997–2004. The horizontal axis gives the average number of articles published per year during this time period, and the vertical axis their normalized citation impact. Universities are categorized into three broad geographical regions: USA, Europe and all other countries.

Figure 1: Number of published articles per year and normalised citation impact for 386 world universities with at least 5,000 articles published during 1997–2004

Figure 1 shows that US universities are highly overrepresented in the top of the ranking based on normalised citation impact, and to a lesser extent, on the number of published articles per year. In fact, in the group of the 25 universities with the highest citation impact, all universities are from the USA, and in the group of 76 universities with a citation impact above 1.5, 67 (88 per cent) are located in the USA. Among the top 25 institutions with the highest number of published articles per year, 20 (80 per cent) are from the USA.

Mean, P25, P50, P75: the mean, 25th , 50th (i.e. the median) and 75th percentile of the distribution. Data relate to 386 world universities publishing at least 5000 papers during 1997-2004.

Table 4 presents statistics of the distribution of the five bibliometric indicators listed in Table 3 among the 386 world universities in the global set. It shows for instance that the mean of the number of papers these universities published per year amounts to 1,594. The median (P50) is lower (1,262), reflecting that the distribution is skewed to the right. 25 Per cent of universities have a normalised citation impact below 0.98 (i.e., the bottom 25 per cent), whereas for the top 25 per cent it is at least 1.43.

Table 5 gives Pearson correlation coefficients among these five variables. It shows that bigger universities in terms of numbers of articles published tend to generate per paper a higher normalised citation impact, and tend to be more general than smaller ones. Moreover, universities with a higher share of their papers published jointly with institutions from the private sector tend to publish more papers and to generate a higher citation impact per paper than institutions do that have less co-publications with the private sector. In the set of 386 world universities there is no significant correlation between normalised citation impact and disciplinary specialisation. This outcome is further discussed in Section 3.3.

3.2 Differences between European and US universities

In the set of 386 world universities, 172 are located in Europe, and 122 in the USA. Table 6 gives for each geographical region the mean and quartiles of the distribution of normalised citation impact among universities. The tables shows that US universities tend to have a higher normalised citation impact than European academic institutions: 1.55 versus 1.11. The 75th percentile of the distribution for Europe is lower than the 25th percentile for the USA. The third column shows that the 172 European universities account for around 72 per cent of the total European university output. The 122 US universities published about 83 per cent of the total US university output. This percentage is higher than the 72 per cent obtained for Europe, and indicates that there is a stronger concentration of published articles among US universities than there is among European institutions, in agreement with earlier analyses published by Matia et al. (2005).

Legend to Table 6: Mean, P25, P50, P75: The mean, 25th, 50th (i.e. the median) and 75th percentile of the distribution. % Papers from univs: A rough estimate of the percentage of the total university article output from a country/region published by the universities in the set of 386 world universities. Both percentages are rough estimates, as the number of articles published by the total collection of universities in Europe or the USA is not precisely known in this study.

In order to further characterize differences among European and US universities, an institution’s citation impact was analysed per discipline, using a classification of research articles into 15 disciplines presented in Section 2.4. For each institution the number of disciplines was determined in which it was ‘world leader’, i.e., ranked among the top 10 or top 25 per cent according to normalized citation impact in the set of 386 world universities described in Section 3.1 above. For each region, the number and percentage of universities was determined that was world leader in at least one discipline, and for these institutions the average number of such ‘top’ disciplines per university was computed. These indicators were calculated for all universities in the set, and also for the the ‘very best’ universities in their region, i.e., being among top 25 percent in their region on the basis of their overall normalised citation impact.

The results are presented in Table 7. The upper half of this table present the outcomes when the concept of ‘world leader’ in a discipline is defined as being among the top 10 per cent among all 386 world universities in that discipline. In the lower half, the criterion for being world leader is somewhat relaxed, and defined as belonging to the top 25 per cent in a discipline. Focusing on the upper half, the second and third column show that 26 per cent of the 172 European universities analysed in this section are world leaders (among the top 10 per cent) in one or more disciplines. For the 122 US universities this percentage is 81. The last two columns focus on the ‘very best’ universities in a region, and show percentages of 67 for European and 100 for US institutions. The average number of disciplines in which European universities are world leader is substantially lower than that for their US counterparts.

Analysing the appearance among the top 25 per cent rather than the top 10 percent in a discipline (lower half of Table 7), it can be seen that in the set of all 172 European universities 65 per cent is world leader in at least one discipline. In the set of the 43 very best European universities this percentage amounts to 100, equal to that for the US. However, the number of disciplines in which a European university is world leader is on average substantially lower than that calculated for US universities: 5.4 versus 12.3. It needs emphasizing that this average of 2 top disciplines for European universities is the mean of a rather skewed distribution. Several universities located in the UK are world leader in a number of disciplines similar to that for US universities.

3.3 Differences across European national academic systems

Table 8 presents indicators for all countries with at least one university publishing over 5,000 papers during 1997–2004. The last two columns give per country the number of universities with 5,000 papers, and the normalized citation impact of these papers. Large differences can be observed among countries with respect to both indicators. Countries generating the highest normalised citation impact are USA (1.50), Switzerland (1.38), UK (1.30), Netherlands (1.28), Canada (1.26), Denmark (1.23) and Finland (1.23).

Table 8 also presents more detailed results on European universities. Columns 3-6 relate to the set of European universities publishing at least 500 papers. For most countries the number of universities (column 4) is higher and their normalised citation impact (column 5) somewhat lower than that for their biggest universities given in columns 7 and 8. As outlined in Chapter 2, in the group of non-European countries and Russia, the study focused on the top 200 universities win terms of total article output. Therefore, for these countries Table 8 does not present any data as regards universities with more than 500 papers.

The third column of Table 8 provides per European country a rough estimate of the importance of a country's universities in the national research system, expressed as the percentage of articles published from universities, relative to a country's total article output. Large differences exist among countries. France has the lowest percentage of university papers, 51 per cent, and Sweden and Turkey the highest: 90 and 91 per cent, respectively. Netherlands, Belgium, Greece, Finland, Austria and Portugal show scores between 80 and 85 per cent. UK, Germany, Italy and Spain have percentages between 73 and 78. Switzerland, Denmark, Norway and Ireland have scores between 71 and 76 per cent, while Eastern European countries Poland, Czech Republic, Hungary, Croatia and Slovenia show relatively low percentages between 55 and 69.

Column 5 gives per European country Pratt’s Concentration Index of published articles among its universities, a measure that ranges between 0 (no concentration, .i.e., all universities publish the same amount of papers) and 1 (total concentration, all papers are published by one single university). Countries showing a relatively low Pratt Index are the Netherlands, Switzerland, Portugal and Hungary. The UK, Austria and Czech Republic show the highest value of this concentration index.

Figure 2 plots for major European countries a country’s Pratt Concentration Index (on the horizontal axis) against the normalized citation impact of the papers published by its universities with at least 500 published articles during 1997–2004 (on the vertical axis). This figure shows that there is apparently no linear correlation between these two variables. The Pearson correlation coefficient amounts to 0.06 (not significant at p=0.01).

Figure 2: Pratt Index versus normalised citation impact for major European countries (universities with > 500 papers during 1997-2004)

3.4 A university’s disciplinary specialisation versus its citation impact

This section analyses disciplinary specialisation within a university, i.e., the extent to which its research papers are evenly distributed among research disciplines, or whether there are particular disciplines on which a university focuses its research activities. Figure 3 plots for each of the 386 universities with at least 5,000 papers during 1997–2004, their disciplinary pecialisation index measuring the degree of concentration of their published articles among disciplines (on the horizontal axis), against the normalised citation impact of its papers (on the vertical axis). In order to obtain an impression of differences across countries, universities from the Netherlands, Germany, Sweden, UK and USA are indicated by special symbols.

Figure 3 reveals that there is in the total set of 386 universities no simple relationship between these two variables. The line drawn in this figure is the liniar regression line. The Pearson and Spearman rank correlation coefficients are –0.06 and –0.10, respectively, and are not significant at p=0.01. In this set of 386 world universities papers published by general and specialised universities tend to generate similar citation impacts per paper.

F
igure 3: 386 World Universities’ disciplinary specialisation index versus their normalized citation impact

The impact measure plotted in Figure 3 relates to a university’s total article output in all disciplines combined. More information can be obtained from an analysis by discipline. Do specialized universities in their specialized areas perform better than general universities do in the same areas? Specialisation is defined here from a disciplinary perspective, in terms of the distribution of a university’s research articles among 15 disciplines, listed in Table 3 in Chapter 2. It does not take into account specialisation within a discipline.

For each university, the normalised citation impact was calculated for all papers in each of the 15 disciplines separately. As illustrated in Table 8 in Section 3.3, large differences exist in universities’ normalised citation impact across countries. In order to correct for these differences, a further normalisation of the citation impact indicator was carried out, by calculating per discipline the ratio of the citation impact of a university from a particular country and the mean citation impact across all universities in that country. This ‘double’-normalised impact indicator was correlated with the publication activity index (PAI, see Section 2.3), expressing the institution’s specialisation in a discipline, based upon the distribution of its papers among disciplines compared to the world distribution. Only universities with at least 50 papers in a discipline were included in the correlation analysis for that discipline.

Table 9 presents the outcomes of this analysis. In four disciplines a significant correlation was found between citation impact per paper and degree of specialisation (publication activity index): in biological sciences primarily related to humans, clinical medicine, molecular biology and in physics, with Pearson coefficients of 0.24, 0.23, 0.41 and 0.17, respectively. In all other disciplines the correlations were not significant at p=0.01.

Table 9: Pearson correlation coefficients between a university’s normalised citation impact in a discipline and its publication activity index in that discipline

3.5 University disciplinary profiles

The distribution of a university’s article output and citation impact per paper across disciplines may provide a view of its disciplinary profile, in terms of the degree of concentration or specialisation. Figures 4a-b to 7a-b present bibliometric profiles of a general, an agricultural, a technical and a general, ‘top’ university, respectively. For each university 2 figures are given. The first figure (4a-7a) plots for each discipline the publication activity index (PAI) on the horizontal axis against the normalised citation impact on the vertical axis.

The second figure (4b-7b) plots for each discipline a university’s percentile rank in the distribution of published articles across universities (horizontal axis) against the percentile rank in the distribution of normalised citation impact across universities (vertical axis). Percentile rank is simply defined as the rank position obtained by a university in a ranking by descending score (number of published articles or normalised citation impact), divided by the number of universities active in the discipline. As a result, universities with the highest score appear in the top of the ranking and therefore have the lowest percentile rank.

In these figures disciplines are indicated by their acronyms. For the full names the reader is referred to Table 3 in Section 2.4. The acronyms for humanities and arts (A&H) and other social sciences (SOC) are displayed between brackets. This was done to indicate that the database does not cover these disciplines sufficiently well. Their scores and position in the plots may not properly reflect university’s activity and impact in those fields. The ‘field’ multidisciplinary is also placed between brackets, since it does not represent a field, but a collection of multidisciplinary journals, including Science, Nture and PNAS.

Figure 4a represents a general university. The publication activity indices are for all disciplines not far from 1.0. This university is one of the most ‘general’ universities in the set of 386 institutions analysed in this study. Figure 4b shows that this university’s papers rank among the top 25 per cent institutions with the highest normalised citation impact in engineering and in economics. In clinical medicine this university’s rank percentile is slightly below 25 per cent.

Figure 5a shows that the university has an extremely high publication activity index in biological sciences primarily related to animals and plants (PAI=10.2) and also a high value in geosciences. Figure 5b reveals that, in these two disciplines, this university ranks among the top 25 per cent of institutions, both in terms of number of published articles and of citation impact. There is little publication activity in physics, mathematics, psychology and humanities & arts. The technical university represented in Figures 6a and 6b shows the highest publication activity in applied physics and chemistry and in engineering. This institution has little activity in biomedical disciplines.

Figures 7a and 7b present profiles op a ‘top’ general university. This university is among the top 25 per cent institutions with the highest citation impact per paper in almost all disciplines.

4. Discussion and conclusions

4.1What is a research university?

A research university is normally defined as a university in which professors have the ius promovendi and train PhD students who carry out research and aim at attaining a doctoral degree. Thus, research universities are academic institutions conducting research at a certain scale. A base assumption underlying the work described in this paper is that, in order to assess the size of research activities within a university, it is appropriate to consider the volume of an institution’s article output. Hence, a research university could in principle be defined as an institution of which the article output exceeds a certain level. However, it is extremely difficult to define a precise quantitative threshold value..

The set of universities analysed in Chapter 3 of this paper consists of institutions publishing at least 5,000 particles during 1997–2004. All these institutions can be qualified as research universities. This threshold was largely determined by the fact mentioned in Section 2.1 that the study on which this paper is based, generated sufficiently accurate data for the top 200 Non-European universities only. Of universities not included in the global set of 386 universities one cannot conclude that they are no research universities or that they perform badly in research. A number of universities have a substantial article output and even a high citation impact per paper, but their total number of articles did not exceed the threshold applied in this paper. This is especially the case for more specialised universities. Chapter 3 revealed large differences among institutions with respect to the number of disciplines covered. For instance, there are general, technical, medical, agricultural and humanities universities. Since specialised universities carry out research activities in a limited number of disciplines, their total numbers of published papers tend to be lower than those of general universities.

The set of 529 European universities analysed in this paper contains universities with a much smaller article output than those in the global set, but it is questionable whether all these institutions can be labelled as research universities. The set includes a number of (poly-)technical schools and other higher education institutes mainly involved in teaching. It needs emphasising that differences may exist among countries with respect to the criteria that are applied in assigning the qualification ‘university’ to an institution of higher education.

4.2 Under-representation of social sciences and humanities

As explained in Section 2.3, the analyses presented in this paper relate to articles published in journals that were processed for Thomson Scientific’s Web of Science (WoS). The indicators calculated in this study may not properly reflect a university’s publication activity, specialisation and citation impact in those domains of scholarship, especially when it is located in a non-English speaking country. Universities specialising in these fields tend to publish a substantial fraction of their publication output in other sources than in the journals processed for the WoS, and therefore remain invisible in this analysis. The same may be true for universities with a strong technical orientation, publishing the majority of their article output in conference proceedings not covered by the WoS. It does not necessarily follow that these institutions do not perform well, but rather that the database used in this paper may not cover their fields of specialization sufficiently well.

4.3 US versus European universities

Figure 1 clearly showed that US universities are overrepresented in the top of the world ranking based on published article output, and particularly in that according to descending citation impact. Some analysts have sought an explanation for this phenomenon in an assumed US bias in referencing practices, in the sense that US authors tend to overcite papers from their own country. But more detailed empirical analyses did not find evidence for such an US bias, at least in the natural, technical and biomedical sciences (e.g., Moed, 2005), fields that are responsible for the overwhelming part of the country's total article output in the Web of Science.

In a recent report, Lambert and Butler (2006) analysed differences among continental European countries, the UK and the USA as regards the structure and performances of their national academic systems. They mentioned several structural factors that in their view are responsible for what they term as ‘mediocrity’ of (particularly continental) European universities. They argued that European universities are seriously under-funded compared to those in other regions, particularly in the USA; that there is a lack of concentration of funds among institutions; that they are far less selective in their admissions; and finally that governance systems tends to be more ‘top-down control’, and give less autonomy of European universities compared to those in the USA, although within Europe there are large differences as well.

A detailed discussion of the analyses and conclusions of Lambert and Butler falls beyond the scope of this paper. But one point needs to be clarified. The authors claim that European universities – with some exceptions, mainly in the UK – are ‘mediocre’. The citation impact analysis presented in Section 3.2 indeed revealed that the overall citation impact per paper of European universities tends to be lower than that of their US counterparts. One may question whether the term ‘mediocre’ is appropriate to qualify the position of European universities in the world rankings. But even if one adopts this qualification from Lambert and Butler, it needs emphasising that ‘mediocrity’ of a university does not necessarily imply that it is mediocre in all disciplines. A key finding from Section 3.2 is that the very best European universities are among the 25 per cent best in the world in at least one discipline, and 65 per cent of them even in the top 10 per cent in a field, but that the number of disciplines in which they are world leader is on average substantially lower than that found for US top universities.

One may distinguish two models for distributing ‘top’ research or excellence among universities. The first could be labelled as the concentration model: a relatively limited number of big research universities carries out research at a world top level in a wide range of research disciplines, if not in the entire spectrum of human science and scholarship. The other universities are mainly teaching universities. The size of their research activities and their quality tends to be substantially lower than that of the top research universities.

The second model, denoted as the distributed model, holds that research activity and excellence is more evenly distributed among universities in a national academic system, or even in the supra-national system such as the European research area. Universities tend to be at the world top level in a limited number of disciplines, but in order to maintain a strong link between teaching and research, they carry out research in other disciplines as well – especially those in which they have teaching duties – at a qualitatively high level, though not necessarily at the highest level from a global viewpoint.

More detailed empirical studies should be made of the structure of European, US and other (supra-) national academic systems, and focus on the extent to which they are structured according to one of the models described above. Therefore, they should also analyse the performance of other US universities than the 122 studied in this paper, and of academic institutions in other countries. A key question is which of the models is the most appropriate in the various countries, especially which model provides the most optimal conditions for ‘top’ research. This complex question awaits further study. The study presented in this paper provided some first results that are discussed in the next section.

4.4 Differences in the structure of national academic systems

Results for an individual university can only be interpreted properly when the structure of a national academic system is taken into account, and the particular role of a university therein. The following examples further may illustrate this. In some countries, a major part of research activities is carried out in separate research institutes rather than in universities, or in institutes that are physically located in universities but that are funded by external, – for instance, national, – research organisations. A typical example is France, in which CNRS plays a dominant role in research. Universities from such a national system tend to publish less papers – and generally obtain lower positions in rankings of world universities – than those from countries in which the research is mainly concentrated in universities. This is clearly reflected in Table 8, according to which France has the lowest percentage of university papers among European countries – i.e. the lowest percentage of papers with an university and/or affiliated hospital address, relative to the total national article output.

As a second example, a university’s disciplinary specialisation index depends upon the structure of a national academic system in which it is embedded, particularly upon how teaching and research activities are distributed among individual universities and between universities and other higher education institutes. As pointed out by Lambert and Butler, historical, political and cultural factors – including national or regional rivalry, different religious traditions or different concepts of academic education – may account for differences across national systems.

For instance, in Germany and in several other countries, the leading concept of academic education (‘bildung’) in the late 1800s did not allow the emerging technical disciplines to find a base within the university, and separate technical schools were founded, some of which grew into big technical universities. In other countries universities did incorporate research and teaching in technical disciplines from the beginning. Such institutions tend to be more ‘general’, and therefore obtain lower values of the disciplinary specialisation index than general universities do in countries with a system of technical universities. In addition, they will tend to publish more papers.

One of the recommendations made by Lambert and Butler (2006) is to establish more concentration of funding and research activities in a limited number of ‘top’ universities. This interesting proposal deserves more attention than can be given to it in this paper. However, the findings obtained in Section 3.3 show that the relationship between concentration and performance is complex. In Europe there is no clear tendency that national academic systems showing more concentration of research activities among its universities, generate – as a whole – a higher citation impact per paper than national systems in which the article output is more evenly distributed among academic institutions. Although this issue needs to be analysed in more detail, this outcome itself may be of interest in the debate of the effectiveness of national research policies aimed to establish more concentration of research activities among universities in a national academic system.

4.5 Citation impact of specialised versus general universities

Section 3.3 found no linear correlation between a university's degree of disciplinary specialisation and its overall citation impact per paper. In other words, general universities, showing a rather even distribution of research papers among disciplines, and specialised universities, having their article output concentrated in a limited number of disciplines (regardless which ones), show similar citation impacts.

A second analysis of the relationship between a university's degree of activity in a discipline and the average citation impact of its papers in that discipline revealed significant linear correlations between these two variables in four disciplines: biological sciences primarily related to humans, clinical medicine, molecular biology and in physics. In the other disciplines the correlation was in most cases negative, but in all cases not significant. These outcomes await further interpretation. The fields in which a significant, positive correlation was found, are typically 'big science' fields, and perhaps the outcomes show that the concept of 'critical mass' in research activity is more relevant in 'big science' than it is in other domains of human scholarship. It needs emphasising, however, that this analysis focuses on specialisation across rather broadly defined disciplines, and that it does not take into account specialisation within a discipline. A more detailed study could further analyse differences across countries and subject of specialisation.

4.6 University profiles and disciplinary classification systems

The disciplinary profiles presented in Section 3.4 are exemplars of a general, an agricultural, a technical, and a genuine general ‘top’ university, respectively. Such profiles provide a useful insight into a university's profile in terms of its specialisation and the average citation impact of its papers. More detailed data-analytical studies could further analyse patterns in the profiles of large numbers of universities, analysing their similarities, and group them into classes or 'clusters', possibly revealing profiles that are different from the 'standard' ones presented in this study. Such an approach would also be able to find for a particular university a number of institutions showing a similar profile, and thus to suggest appropriate benchmarks in a comparative research evaluation study.

The categorization of research activities into 15 disciplines is rather broad, and identifies to a large extent ‘classical’ disciplines. Although its application in this study provides an informative insight into a university’s cognitive orientation, it may not take fully into account recent developments in scientific research in general, and in university research in particular. Key elements are ‘relevance’ and ‘trans-disciplinarity’ (e.g., Rip, 2004). Relevance relates to the structuring of research activities, both of a fundamental (‘strategic’) and more applied nature, around finding solutions for practical, societal problems. Trans-disciplinarity relates to the tendency that research more and more crosses traditional cognitive-disciplinary and also institutional barriers. It is a great challenge to further develop operational classification systems of scientific-scholarly publications that take into account these developments. The inclusion of a field ‘social sciences primarily related to medicine and health’ in the disciplinary categorization used in this paper may be considered as a first step.

4.7 Network analysis

In order to analyse the structure of a national academic system, highlighting the position of individual universities, maps based on network analysis are particularly useful. The reader is referred to the interesting work of CWTS colleague Clara Calero. This work involves the development of a methodology that allows one to identify the best research universities in their national or regional environment, based on a series of bibliometric and network indicators (Calero, 2006). Institutions are not ranked on the basis of one single indicator. Instead, a network analysis is applied to represent relations between universities based on co-authorship, and to identify patterns of co-publication activity.

4.8 Further data collection and verification

The publication data for the universities analysed in this paper were not verified by representatives of the institutions, except in a few cases. A main future task will be to find ways to enable them to verify the data. The bibliometric data used in this study focus on the 'output' side of research. It should be combined with other publicly available, verified or certified information, reflecting aspects of the 'input' side, including per discipline at least the number of students and various categories of research staff, and the amount of public funding. Although these ‘input’ measures partly reflect ‘output’ categories such as research quality as well – for instance, ‘good’ institutions tend to attract more funding than less good ones – their use in statistical analyses is indispensable, and will enrich the comparative analysis of national academic systems.

It is essential that these data are not only available at the level of a university as a whole, but at least also by discipline, in order to relate ‘output’ to ‘input’ at the level of disciplines. Therefore, the mismatch that currently exists between disciplinary categorisations at the output and the input side needs to be solved (Luwel, 2004). In this way, a public information system on world research universities can be built, that is not only useful for the general public, but also constitutes a database for further research on research performance and its determinants.

Acknowledgements

The research presented in this paper is partly funded by the European Commission, within the framework of the ASSIST project (Analysis & Studies and Statistics & Indicators on Science and Technology), which is part of the European Commission (EC) Sixth Framework Programme for Research, Technology and Development. The author is grateful to Dr Wolfgang Glänzel and Dr Bart Thijs at the Steunpunt O&O Statistieken of the Katholieke Universiteit Leuven for providing him with accurate publication data on Flemish universities. He wishes to thank his CWTS colleague Clara Calero for her important contributions to the ASSIST project, and is grateful to his CWTS colleague Dr Marc Luwel for his comments on an earlier version of this paper and for stimulating discussions.

References

Bookstein, A., and Yitzahki, M. (1999). Own language preference: A new measure of “relative language self-citation”. Scientometrics, 46, 337–348.

Calero, C. and Moed, H.F. (2006). Depicting the landscape of research universities. Paper to be presented at the Ninth International Conference on Science and Technology Indicators, Leuven (Belgium), 7-9 September 2006.

Diamond, N., and Graham, H.D. (2000). How should we rate research universities? http://www.physics.northwestern.edu/graduate/Graham_Diamond.html.

Egghe, L. and Rousseau, R. (1990). Introduction to Informetrics. Amsterdam: Elsevier.

Garfield, E. (1964). The Citation Index – A new dimension in indexing. Science, 144, 649–654.

Garfield, E. (1979). Citation Indexing. Its theory and application in science, technology and humanities. New York: Wiley.

Goldberger, M.L, Maher, B.A, and Flatteau, P.E. (1995). Research-doctorate programs in the United States: Continuity and change. National Academy Press.

Lambert, L. and Butler, N. (2006). The Future of European Universities: Renaissance or Decay? London (UK): Centre for European Reform (CER), ISBN 1 901 229 67 X.

Luwel, M. (2004). The use of input data in the performance analysis of R&D systems. In: Moed, H.F., Glänzel, W., and Schmoch, U. (2004) (eds.). Handbook of quantitative science and technology research. The use of publication and patent statistics in studies of S&T systems. Dordrecht (the Netherlands): Kluwer Academic Publishers, 315–338.

Matia, K., Amaral, L.A.N., Luwel, M., Moed, H.F., Stanley, H.E. (2005). Scaling phenomena in the growth dynamics of scientific output. Journal of the American Society for Information Science and Technology 56, 893-902.

Moed, H.F., de Bruin, R.E., and van Leeuwen, Th.N. (1995). New bibliometric tools for the assessment of national research performance: database description, overview of indicators and first applications. Scientometrics, 33, 381–442.

Moed, H.F. (2005). Citation Analysis in Research Evaluation. Dordrecht (Netherlands): Springer. ISBN 1-4020-3713-9, 346 pp.

Rip, A.(2004). Strategic research, post-modern universities and research training. Higher Education Policy 17, 153-166.

SJTU (2005). Academic Ranking of World Universities -2005. Shanghai Jiao Tong University, Institute of Higher Education, available at

THES (2005). World University Rankings. Who is Number One? The Times Higher Education Supplement, February 24, 2005; April 29, 2005; May 2, 2005, available at http://www.thes.co.uk/worldrankings/

Van Raan, A.F.J. (1996). Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics, 36, 397–420.

Van Raan, A.F.J. (2005). Challenges in Ranking of Universities, available at http://www.cwts.nl/cwts/AvR-ShanghaiConf.pdf.