Centre for Science and Technology Studies Centre for Science and Technology Studies 2333AL Leiden Zuid Holland 31715273909
  • Blog
  • Let's move beyond too simplistic notions of 'misuse' and 'unintended effects' in debates on the JIF

Let's move beyond too simplistic notions of 'misuse' and 'unintended effects' in debates on the JIF



pecking orderResearchers in the field of scientometrics are quite often robust in their criticisms. Last week, the paper A simple proposal for the publication of journal citation distributions was published online. The contribution by Lariviere et al. (2016) takes a familiar normative stance regarding the 'misuses' and ‘unintended effects’ of indicators, and in particular the JIF. Arguments against the JIF often cite its technical shortcomings, for instance in claiming it is open to manipulation and misuse by editors and uncritical parties. Larivière et al. (2016) argue that the IF "is an inappropriate indicator for the evaluation of research or researchers" because the single numbers conceal "the full extent of the skew of distributions and variation in citations received by published papers that is characteristic of all scientific journals" and therefore "assume for themselves unwarranted precision and significance.” The authors hope that the method they propose for generating the citation distributions that underlie JIFs will "help to refocus attention on individual pieces of work and counter the inappropriate usage of JIFs during the process of research assessment."

Not merely for the sake of argument

Improving validity, reliability and transparency are obviously useful endeavors, but I will argue that it is more helpful if we do not take received assumptions of the JIF as given. Instead, we should position as a focus of intervention the multiple roles and influences of the JIF in actual research practices. My primary concern is this: by limiting solutions to ‘improper’ indicator uses to questions of validity or transparency, we assume that more transparency or better indicators will necessarily give rise to better evaluation practices. Though I applaud sincere, methodologically sophisticated calls for more transparency such as the one made by Larivière et al., I am afraid they do not suffice. The recourse we then take is towards an upstream solution, guided by an optimistic yet also slightly technocratic mode of 'implementation' (De Rijcke & Rushforth, 2015). If journals would indeed start to publish the citation distributions behind their JIFs, what exactly would this change on the shop-floor, in assessment situations, and in the daily work of doing research?

The JIF trickles back up

A general criticism that can be made about the well-known accounts against the JIF is their inattention towards the ‘folk theories’ of the JIF as applied by scientists and evaluators in actual practices (Rushforth & De Rijcke, 2015; see also Aksnes & Rip, 2009). “What characterizes folk theories is that they provide orientation for future action… They are a form of expectations, based in some experience, but not necessarily systematically checked. Their robustness derives from their being generally accepted, and thus part of a repertoire current in a group or in our culture more generally.” (Rip 2006, 349) In our research in biomedicine we for instance found that researchers use these ‘folk theories’ to navigate quite routine knowledge making activities, including selecting useful information from the overwhelming amounts of literature they could potentially read; settling discussions over whom to collaborate with and when; and how much - additional - time to spend in the laboratory producing that data. Given the extent of the embeddedness, I must say I feel ambivalent about statements that the JIF ‘misleads’. For one, not all of these embedded uses are grounded in naïve assumptions about the citation performance of individual papers in particular journals with a certain JIF. Secondly, who misleads whom? These different embedded uses of the JIF will ‘trickle back up’ into formal assessment procedures, because “[a]uditors are not aliens. They are versions of ourselves.” (Strathern 1997, 319)

An example

Let's consider the hypothetical situation of a formal assessment procedure in which a research group in oncology is looking to hire a new professor. The hiring committee has at its disposal the publication lists of the candidates. These lists also specify the JIF of the journals in which the candidates have published. Now suppose that the committee takes a look at the publication list of one of the candidates. The committee members start to compare the journals on the list by way of the JIFs. They see that this researcher mainly publishes in the top-tier journals in oncology. If we take the warning of Lariviere et al. to heart, we would advise the committee members not to conflate these numbers with the actual citation impact of the individual papers themselves. And rightly so. But does this mean that all uses of the JIF in this formal hiring procedure in this particular setting are off limits? Larivière et al. should answer this question with a 'yes', seeing that they disapprove of all uses of the JIF in the assessment of individual researchers. However, I think that in this case it is very well possible to come up with a reasonable motive for using JIFs to support the decision-making process about who (not) to hire. In some biomedical fields, different 'tiers' of journals with certain JIF-ranges can both denote a certain standing in a field and a particular type of scientific work (e.g. descriptive in the lower IF ranges versus causal in the higher IF ranges). So what a committee can do hypothetically in this field, on the basis of the JIFs, is assess whether a researcher mainly does descriptive work or primarily publishes about biological mechanism. In other words, the committee can make a substantive assessment of the type of work the candidate is involved in by looking at the JIF values of the journals in which she publishes (among other characteristics of the journal). And they can use this information, for instance, to deduce whether or not the candidate’s research lines would fit into the rest of the research team they are looking to hire for. The reader of this blogpost will understand that this is merely a hypothetical example. But I hope the point is clear: the JIF can acquire a range of different meanings in actual research and assessment practices.

Conclusion

Larivière et al. put forth a methodologically driven plea to focus not on the JIF but on individual papers and their actual citation impact. Though commendable, I think this strategy obscures a much more fundamental issue about effects of the JIF on the daily work of researchers and evaluators. JIF-considerations have a tendency to either move to the background other measures of scientific quality (e.g. originality, long‐term scientific progress, societal relevance), or to allow them to become redefined through their relations to the JIF and other quantitative performance indicators. In my opinion this insight leads to a crucial shift in perspective. For truly successful interventions into indicator-based assessment practices to happen, I think we need to move beyond too simplistic entry points to the debate of ‘misuse’ and ‘unintended effects’. My hypothesis is that researchers (and evaluators) continue to use the JIF in assessment contexts - despite the technical shortcomings – for the complicated reason that the indicator is already so engrained in different knowledge producing activities in different fields. Our research findings suggest that in calling for researchers and evaluators to ‘drop’ the JIF, people are actually calling for quite fundamental transformations in how scientific knowledge is currently manufactured in certain fields. This transformation is the primary, and also the quite daunting, task.


I would like to thank Alex Rushforth for collaborating with me on the project in biomedicine that I draw on extensively above. The text is partly based on our joint articles that came out of the project.

I would like to thank Paul Wouters and Ludo Waltman for valuable discussions that informed the preparations for this blogpost.

See also the blogpost by Ludo Waltman in response to the same article.

 

References

  • Aksnes, D. W., and A. Rip. (2009). Researchers' perceptions of citations. Research Policy, 38 (6), 895-905.
  • Rip, A. (2006). Folk theories of nanotechnologists. Science as Culture, 15 (4), 349-365.
  • Rushforth, A.D. & de Rijcke, S. (2015).  Accounting for impact? The Journal Impact Factor and the making of biomedical research in the Netherlands. Minerva, 53, 117-139.
  • de Rijcke, S. & Rushforth, A.D. (2015). To intervene, or not to intervene, is that the question? On the role of scientometrics in research evaluation. Journal of the Association for Information Science and Technology, 66 (9), 1954-1958.

 


About Sarah de Rijcke

Professor of Science and Evaluation Studies and deputy director of CWTS, and coordinator of the Science and Evaluation Studies research group. Her research interests include academic valuation and evaluation processes, changing research cultures, knowledge infrastructures, and roles of research in and for society.


30 comments

Mandatory fields
  • fghrtrt November 14th, 2018 2:26 am
    http://www.google.com/
    https://mirvba.ru/
    http://www.itdox.ru/
    http://replicatrainers.medex52.ru/
    http://www.gokstroy.ru/
    http://www.notary-krsk.ru/
    http://www.isa-mgsu.ru/
    http://tilzit-fishing.ru/
    http://veraslovo.ru/
    http://www.hyundai-minusinsk.ru/
    http://www.amurpak.ru/
    http://mini-horse-fx.ru/
    http://www.enfagroupp.ru/
    http://www.volgguvd.ru/
    http://opfrspb.ru/
    http://artbeauty29.ru/
    http://www.nitroneopunk.ru/
    http://molotok-tomsk.ru/
    http://www.apu-cheb.ru/
    http://www.vobler-club.ru/
    http://www.sdv-ski.ru/
    http://www.teplomurom.ru/
    http://www.vesti-respubliki.ru/
    http://www.antenn-proizvoditel.ru/
    http://www.muzkulatu.ru/
    http://www.verobuddismo.ru/
    http://www.chatnervi.ru/
    http://www.autoinstrument44.ru/
    http://www.dvdmebel.ru/
    http://www.loveissochi.ru/
    http://www.dmscruise.ru/
    http://www.allstanok.ru/
    http://www.roerich-belogorie.ru/
    http://www.sibro-etty.ru/
    http://www.tver-nashgorod.ru/
    http://www.admturmr.ru/
    http://www.termodom56.ru/
    http://www.school-ongudai.ru/
    http://www.xxspektr.ru/
    http://www.feesmg.ru/
    http://www.hellosamara.ru/
    http://www.blytuz.ru/
    http://www.kupi-kedi.ru/
    http://www.zaozak.ru/
    http://www.cubajam.ru/
    https://www.szn-krs-sv.ru/
    Reply
  • candid22n October 10th, 2018 4:57 pm
    https://pauio.com/trum/ - 트럼프카지노 /
    https://pauio.com/woori/ - 우리카지노 /
    https://pauio.com/mca/ - 엠카지노 /
    https://pauio.com/gatsby/ - 개츠비카지노 /
    https://pauio.com/theking/ - 더킹카지노 /
    Reply
  • jangs5635 October 10th, 2018 4:57 pm
    https://dancesweb.com/trump/ - 트럼프카지노 /
    https://dancesweb.com/33casino/ - 33카지노 /
    https://dancesweb.com/gatsby/ - 개츠비카지노 /
    Reply
  • wjddud6666 October 10th, 2018 4:56 pm
    https://stylebet77.com/obama/ - 오바마카지노 /
    https://stylebet77.com/yescasino/ - YES카지노 /
    Reply
  • jsazzang6 October 10th, 2018 4:56 pm
    https://como79.com/first/ - 퍼스트카지노 /
    https://como79.com/super/ - 슈퍼카지노 /
    https://como79.com/trump/ - 트럼프카지노 /
    Reply
  • game6352y October 10th, 2018 4:55 pm
    https://custory.com/gatsby/ - 개츠비카지노 /
    https://custory.com/bacara/ - 바카라사이트 /
    https://custory.com/cass/ - 카지노사이트 /
    https://custory.com/trump/ - 트럼프카지노 /
    Reply
  • nenetv24 October 2nd, 2018 7:47 pm
    nba중계

    스포츠중계

    해외야구중계

    느바중계

    농구중계

    프리미어리그중계

    프리메라리가중계

    세리에a중계

    분데스리가중계

    무료스포츠중계

    에레디비지에중계

    ufc중계

    lol중계

    롤중계

    테니스중계

    nfl중계

    미식축구중계

    nfl중계

    http://nenetv24.com/
    먹튀검증-http://nenetv24.com/
    무료스포츠중계-http://nenetv24.com/
    스포츠중계-http://nenetv24.com/
    실시간스포츠중계-http://nenetv24.com/
    스포츠중계사이트-http://nenetv24.com/
    nba중계-http://nenetv24.com/
    ufc중계-http://nenetv24.com/

    먹튀검증-http://nenetv02.com/
    무료스포츠중계-http://nenetv02.com/
    스포츠중계-http://nenetv02.com/
    실시간스포츠중계-http://nenetv02.com/
    스포츠중계사이트-http://nenetv02.com/
    nba중계-http://nenetv02.com/
    ufc중계-http://nenetv02.com/
    먹튀검증-http://nenetv03.com/
    무료스포츠중계-http://nenetv03.com/
    스포츠중계-http://nenetv03.com/
    실시간스포츠중계-http://nenetv03.com/
    스포츠중계사이트-http://nenetv03.com/
    nba중계-http://nenetv03.com/
    ufc중계-http://nenetv03.com/
    먹튀검증-http://nenetv24.com/
    무료스포츠중계-http://nenetv24.com/
    스포츠중계-http://nenetv24.com/
    실시간스포츠중계-http://nenetv24.com/
    스포츠중계사이트-http://nenetv24.com/
    nba중계-http://nenetv24.com/
    ufc중계-http://nenetv24.com/

    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//





    해외농구중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    해외축구중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    무료실시간스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    실시간스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    느바중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    리그앙분석-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    리그앙-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    분데스리가중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    프리미어리그분석-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    epl분석-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    레이커스분석-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    골스분석-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    믈브분석-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    nba분석-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    느바분석-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    nfl중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//

    테니스중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    롤중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    lol중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    ufc중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    에레디비지에중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    무료스포츠중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    분데스리가중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    세리에a중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    프리메라리가중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    프리미어리그중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    해외야구중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//
    nba중계-http://nenetv03.com/-http://nenetv24.com/-http://nenetv02.com//


    www.nenetv24.com

    먹튀검증-www.nenetv24.com/
    무료스포츠중계-www.nenetv24.com/
    스포츠중계-www.nenetv24.com/
    실시간스포츠중계-www.nenetv24.com/
    스포츠중계사이트-www.nenetv24.com/
    nba중계-www.nenetv24.com/
    ufc중계-www.nenetv24.com/

    먹튀검증-www.nenetv02.com/
    무료스포츠중계-www.nenetv02.com/
    스포츠중계-www.nenetv02.com/
    실시간스포츠중계-nenetv02.com/
    스포츠중계사이트-nenetv02.com/
    nba중계-www.nenetv02.com/
    ufc중계-www.nenetv02.com/
    먹튀검증-www.nenetv03.com/
    무료스포츠중계-www.nenetv03.com/
    스포츠중계-www.nenetv03.com/
    실시간스포츠중계-www.nenetv03.com/
    스포츠중계사이트-www.-nenetv03.com/
    nba중계-www.nenetv03.com/
    ufc중계-www.nenetv03.com/
    먹튀검증-www.nenetv24.com/
    무료스포츠중계-www.nenetv24.com/
    스포츠중계-www.nenetv24.com/
    실시간스포츠중계-www.nenetv24.com/
    스포츠중계사이트-www.nenetv24.com/
    nba중계-www.nenetv24.com/
    ufc중계-www.nenetv24.com/
    유튜브-https://www.youtube.com/watch?v=7E-bB5CinM0
    youtube-https://www.youtube.com/watch?v=7E-bB5CinM0
    먹튀검증-https://www.youtube.com/watch?v=7E-bB5CinM0
    무료스포츠중계-https://www.youtube.com/watch?v=7E-bB5CinM0
    스포츠중계-https://www.youtube.com/watch?v=7E-bB5CinM0
    실시간스포츠중계-https://www.youtube.com/watch?v=7E-bB5CinM0
    스포츠중계사이트-https://www.youtube.com/watch?v=7E-bB5CinM0
    nba중계-https://www.youtube.com/watch?v=7E-bB5CinM0
    ufc중계-https://www.youtube.com/watch?v=7E-bB5CinM0
    먹튀검증-https://www.youtube.com/watch?v=7E-bB5CinM0
    무료스포츠중계-https://www.youtube.com/watch?v=7E-bB5CinM0
    스포츠중계-https://www.youtube.com/watch?v=7E-bB5CinM0
    실시간스포츠중계-https://www.youtube.com/watch?v=7E-bB5CinM0
    스포츠중계사이트-https://www.youtube.com/watch?v=7E-bB5CinM0
    nba중계-https://www.youtube.com/watch?v=7E-bB5CinM0
    ufc중계-https://twitter.com/gscu4233
    먹튀검증-https://twitter.com/gscu4233
    무료스포츠중계-https://twitter.com/gscu4233
    스포츠중계-https://twitter.com/gscu4233
    실시간스포츠중계-https://twitter.com/gscu4233
    스포츠중계사이트-https://twitter.com/gscu4233
    nba중계-https://twitter.com/gscu4233
    ufc중계-https://twitter.com/gscu4233
    스포츠중계-https://www.youtube.com/watch?v=7E-bB5CinM0
    실시간스포츠중계-https://www.youtube.com/watch?v=7E-bB5CinM0
    스포츠중계사이트-https://www.youtube.com/watch?v=7E-bB5CinM0
    nba중계-https://www.youtube.com/watch?v=7E-bB5CinM0
    ufc중계-https://nenetv.xyz
    먹튀검증-https://nenetv.xyz
    무료스포츠중계-https://nenetv.xyz
    스포츠중계-https://nenetv.xyz
    실시간스포츠중계-https://nenetv.xyz
    스포츠중계사이트-https://nenetv.xyz
    nba중계-https://nenetv.xyz
    ufc중계-https://nenetv.xyz
    먹튀검증-https://nenetv.xyz
    무료스포츠중계-https://nenetv.xyz
    스포츠중계-https://nenetv.xyz
    실시간스포츠중계-https://nenetv.xyz
    스포츠중계사이트-https://nenetv.xyz
    nba중계-https://nenetv.xyz
    ufc중계-https://nenetv.xyz
    Reply
  • wangwanglolo October 2nd, 2018 10:17 am
    https://custory.com/theking/ - 더킹카지노 /
    https://custory.com/woori/ - 우리카지노 /
    https://custory.com/first/ - 퍼스트카지노 /
    https://custory.com/yes/ - 예스카지노 /
    Reply
  • Diet Leptosin September 26th, 2018 6:09 am
    for those of you who are looking for a good and safe diet for the body visit the following website;
    http://www.leptosinpusat.com/
    http://www.leptosinpusat.com/obat-diet-leptosin-yang-aman-dan-baik-untuk-tubuh-lapaknya-neng/
    https://infolangsingcepat.blogspot.com/
    https://fashioncompanyhan.blogspot.com/
    https://lapakleptosin.wordpress.com/
    Reply
  • Webroot Antivirus Support Phone Number September 24th, 2018 2:25 pm
    It is not that difficult to find the right “Guru” that you need. Filtering is done by area of expertise; many of these people are experts in Outlook Express, Linux, Windows, Microsoft Office and Excel. These are just to name a few of the areas that are covered by the experts when you call the technical support number hotline!
    https://www.advisorexpert.co.uk/antivirus-support-phone-number/
    Reply
  • AOL Support Phone Number September 24th, 2018 2:23 pm
    As mentioned, you are able to look through the list of all of the technical support staff and you will find that they all have great feedback and excellent ratings on how they have helped all of the customers who have ever called their technical support number.
    https://www.advisorexpert.co.uk/aol-support-phone-number/
    Reply
  • Office 365 Support Phone Number September 24th, 2018 2:21 pm
    At Tech support Number you are able to look through the database of the experts that they have hired to man their technical support number. You can be sure that all of the techs have been hand picked by the administration to give the customer maximum access to any answers that they might be needing.
    https://www.advisorexpert.co.uk/office-365-support-phone-number/
    Reply
  • Outlook Support Phone Number September 24th, 2018 2:19 pm
    Now Tech support Number is not like any other technical support number where you get connected with a ghost, nope, these guys are on their toes! You get connected instantly with a real, live operator who is courteous and knowledgeable to boot! A killer combination don’t you think? https://www.advisorexpert.co.uk/outlook-support-phone-number/
    Reply
  • Quickbooks Support Phone Number September 24th, 2018 2:17 pm
    Or even worse yet, you can hear someone else whispering to them. Yes folks, it has happened to us all! If you have answered yes to any one of these then look no further, this has been written just for you!
    https://www.advisorexpert.co.uk/quickbooks-support-phone-number/
    Reply
  • Sage Support Phone Number September 24th, 2018 2:16 pm
    Are you sick and tired of being put on hold when you dial a technical support number for a specific problem? Perhaps you not only get put on hold but the operator also gives you useless information while you hear him or her riffling through pages to find what you need?
    https://www.advisorexpert.co.uk/sage-support-phone-number/
    Reply
  • Arnav Nihaliya September 21st, 2018 11:24 am
    http://www.quickhelps.co/phone-number/quickbooks/
    http://www.quickhelps.co/uk/phone-number/quickbooks/
    http://www.quickhelps.co/phone-number/quicken/
    http://www.quickhelps.co/uk/phone-number/quicken/
    http://www.quickhelps.co/phone-number/sage/
    http://www.quickhelps.co/uk/phone-number/sage/
    http://www.quickhelps.co/phone-number/aol/
    http://www.quickhelps.co/uk/phone-number/aol/
    http://www.quickhelps.co/phone-number/outlook/
    http://www.quickhelps.co/uk/phone-number/outlook/
    Reply
  • Pengobatan Alternatif Gagal Ginjal September 15th, 2018 10:54 am
    Very interesting and useful information
    https://goo.gl/yKJ7gL || https://goo.gl/F6mUSf || https://goo.gl/4opyz2 || https://goo.gl/jaSQMK || https://goo.gl/HSoemD || https://goo.gl/03A3GR || https://goo.gl/2lY1tM || https://goo.gl/K8PfwH || https://goo.gl/MgRb5D || https://readymag.com/hanaalifian88/1161060/
    Reply
  • Obat Asma August 29th, 2018 9:41 am
    your information very interesting
    http://www.tempat.jellygamatqncmurah.com/obat-asma/
    http://obatgondoktradisional.jellygamatluxor.biz/obat-asam-urat/
    Reply
  • Obat Asam Urat August 18th, 2018 11:03 am
    Thank you for the good information you have provided
    http://obatgondoktradisional.jellygamatluxor.biz/obat-asam-urat/
    Reply
  • Cara Mengobati Kelenjar Getah Bening August 15th, 2018 4:37 am
    The information provided is very interesting and interesting to read.
    http://www.tempat.jellygamatqncmurah.com/cara-mengobati-kelenjar-getah-bening-tradisional-mujarab/
    http://rumahjellygamatgold.com/obat-penghancur-tumor-ginjal-tradisional-manjur/
    http://rumahjellygamatgold.com/cara-menyembuhkan-hepatitis-dengan-cepat/
    http://www.jual.jellygamatqncasli.com/obat-maag-kronis-tradisional-paling-ampuh/
    Reply
  • obat herbal alami July 30th, 2018 9:46 am
    Terimakasih Atas Informasinya sangat Bermanfaat
    Mampir ke Blogger Kami (Dehisna.com)
    http://www.dehisna.com/2018/07/obat-herbal-alami-penyakit-sipilis-pada.html
    http://www.dehisna.com/2018/07/khasiat-walatra-bersih-wanita.html
    http://www.dehisna.com/2018/07/walatra-bersih-wanita-obat-khusus.html
    http://www.dehisna.com/2018/07/manfaat-dan-khasiat-walatra-bersih.html
    http://www.dehisna.com/2018/07/walatra-bersih-wanita-mempunyak.html
    http://www.dehisna.com/2018/07/walatra-bersih-wanita-kapsul-obat.html
    http://www.dehisna.com/2018/07/solusi-untuk-mengatasi-keluhan.html
    http://www.dehisna.com/2018/07/walatra-bersih-wanita-obat-herbal-alami.html
    http://www.dehisna.com/2018/07/cara-pemesana-dan-harga-walatra-bersih.html
    http://www.dehisna.com/2018/07/cara-membunuh-kuman-serta-jamur.html
    http://www.dehisna.com/2018/07/obat-untuk-menjaga-kesehatan-miss-v.html
    http://www.dehisna.com/2018/07/khasiat-walatra-bersih-wanita-obat.html
    http://www.dehisna.com/2018/07/obat-herbal-untuk-mengobati-menorrhagia.html
    http://www.dehisna.com/2018/07/obat-herbal-rahim-lemah-saat-hamil.html
    http://www.dehisna.com/2018/07/cara-mengatasi-mimo-dan-kista.html
    http://www.dehisna.com/2018/07/cara-membersihkan-rahim-setelah.html
    http://www.dehisna.com/2018/07/cara-mengembalikan-elastisitas-miss-v.html
    http://www.dehisna.com/2018/07/obat-herbal-kista-ovarium-walatra.html
    http://www.dehisna.com/2018/07/obat-herbal-rahim-lemah-saat-hamil.html
    http://www.dehisna.com/2018/07/obat-herbal-alami-keputihan-dan-gatal.html
    http://www.dehisna.com/2018/06/obat-herbal-bau-tak-sedap-pada.html
    http://www.dehisna.com/2018/06/obat-nyeri-haid-walatra-bersih-wanita.html
    http://www.dehisna.com/2018/06/solusi-untuk-kesuburan-wanita.html
    http://www.dehisna.com/2018/06/walatra-sehat-sendi-obat-herbal-nyeri.html
    http://www.dehisna.com/2018/06/walatra-sehat-sendi-obat-herbal-nyeri.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-penyakit-stroke.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-jantung-bengkak.html
    http://www.dehisna.com/2018/06/obat-herbal-luka-gangren-atau-luka.html
    http://www.dehisna.com/2018/06/obat-herbal-penyakit-gondok-100-ampuh.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-campak-pada-anak.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-gagal-ginjal.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-maag-kronis.html
    http://www.dehisna.com/2018/06/obat-herbal-untuk-nyeri-sendi.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-penyakit-jantung.html
    http://www.dehisna.com/2018/06/obat-herbal-hipetensitekanan-darah.html
    http://www.dehisna.com/2018/06/obat-herbal-untuk-menghilangkan-bekas.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-batu-empedupenyakit.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-flek-hitam-pada-paru.html
    http://www.dehisna.com/2018/06/obat-herbal-ginjal-bocor.html
    http://www.dehisna.com/2018/06/obat-herbal-penyakit-liver-atau-hati.html
    http://www.dehisna.com/2018/06/obat-herbal-penyakit-kanker-payudara.html
    http://www.dehisna.com/2018/05/obat-mujarab-untuk-herpes.html
    http://www.dehisna.com/2018/05/obat-herbal-penyakit-hernia-hiatus.html
    Reply
  • obat herbal alami July 17th, 2018 10:15 am
    Mampir ke Blogger Kami (Dehisna.com)
    http://www.dehisna.com/2018/07/walatra-bersih-wanita-mempunyak.html
    http://www.dehisna.com/2018/07/walatra-bersih-wanita-kapsul-obat.html
    http://www.dehisna.com/2018/07/solusi-untuk-mengatasi-keluhan.html
    http://www.dehisna.com/2018/07/walatra-bersih-wanita-obat-herbal-alami.html
    http://www.dehisna.com/2018/07/cara-pemesana-dan-harga-walatra-bersih.html
    http://www.dehisna.com/2018/07/cara-membunuh-kuman-serta-jamur.html
    http://www.dehisna.com/2018/07/obat-untuk-menjaga-kesehatan-miss-v.html
    http://www.dehisna.com/2018/07/khasiat-walatra-bersih-wanita-obat.html
    http://www.dehisna.com/2018/07/obat-herbal-untuk-mengobati-menorrhagia.html
    http://www.dehisna.com/2018/07/obat-herbal-cara-melancarkan-haid-yang.html
    http://www.dehisna.com/2018/07/cara-mengatasi-mimo-dan-kista.html
    http://www.dehisna.com/2018/07/cara-membersihkan-rahim-setelah.html
    http://www.dehisna.com/2018/07/cara-mengembalikan-elastisitas-miss-v.html
    http://www.dehisna.com/2018/07/obat-herbal-kista-ovarium-walatra.html
    http://www.dehisna.com/2018/07/obat-herbal-rahim-lemah-saat-hamil.html
    http://www.dehisna.com/2018/07/obat-herbal-alami-keputihan-dan-gatal.html
    http://www.dehisna.com/2018/06/obat-herbal-bau-tak-sedap-pada.html
    http://www.dehisna.com/2018/06/obat-nyeri-haid-walatra-bersih-wanita.html
    http://www.dehisna.com/2018/06/solusi-untuk-kesuburan-wanita.html
    http://www.dehisna.com/2018/06/walatra-sehat-sendi-obat-herbal-nyeri.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-penyakit-usus-buntu.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-penyakit-stroke.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-jantung-bengkak.html
    http://www.dehisna.com/2018/06/obat-herbal-luka-gangren-atau-luka.html
    http://www.dehisna.com/2018/06/obat-herbal-penyakit-gondok-100-ampuh.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-campak-pada-anak.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-gagal-ginjal.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-maag-kronis.html
    Reply
  • obat herbal July 3rd, 2018 6:45 am
    http://klikgamat.com
    http://gamatori.com/2018/06/30/obat-nyeri-haid-yang-paling-ampuh/
    http://gamatori.com/2018/07/02/obat-keputihan-gatal-dan-perih-yang-terbukti-ampuh/
    http://gamatori.com/2018/07/03/obat-alami-cepat-hamil-paling-ampuh/
    http://www.klikgamat.com/2018/05/obat-mujarab-luka-diabetes_19.html
    http://www.klikgamat.com/2018/05/obat-alami-stroke-kronis.html
    http://www.klikgamat.com/2018/05/obat-alami-ambeien.html
    http://www.klikgamat.com/2018/05/obat-herbal-asma-untuk-anak-ampuh.html
    http://www.klikgamat.com/2018/05/obat-herbal-maag-akut-maupun-maag-kronis.html
    http://www.klikgamat.com/2018/05/obat-herpes-ampuh.html
    http://www.klikgamat.com/2018/06/obat-penyakit-anemia-manjur.html
    http://www.klikgamat.com/2018/06/obat-herbal-anti-kanker-obat-herbal.html
    http://www.klikgamat.com/2018/06/obat-alami-kista-tanpa-operasi-paling.html
    http://www.klikgamat.com/2018/06/obat-alami-sembuhkan-cacar-air-pada-anak.html
    http://www.klikgamat.com/2018/06/obat-jantung-koroner-alami.html
    http://www.klikgamat.com/2018/06/obat-turun-berok-paling-manjur.html
    http://www.klikgamat.com/2018/06/obat-jerawat-paling-ampuh-dan-terbukti.html
    http://www.klikgamat.com/2018/06/obat-asma-paling-ampuh-dari-bahan.html
    http://www.klikgamat.com/2018/06/obat-alami-asam-urat.html
    http://www.klikgamat.com/2018/06/obat-alami-kolesterol-paling-manjur.html
    http://www.klikgamat.com/2018/06/obat-alami-asam-lambung.html
    http://www.klikgamat.com/2018/06/obat-alami-radang-tenggorokan.html
    http://www.klikgamat.com/2018/06/obat-alami-sakit-gigi-paling-manjur.html
    http://www.klikgamat.com/2018/06/obat-alami-darah-tinggi.html
    http://www.klikgamat.com/2018/06/obat-sakit-pinggang-tradisional-paling.html
    http://www.klikgamat.com/2018/06/obat-herbal-tbc-paru-paling-mujarab.html
    http://www.klikgamat.com/2018/06/obat-panu-paling-ampuh.html
    http://www.klikgamat.com/2018/06/obat-bronkitis-paling-manjur.html
    http://www.klikgamat.com/2018/06/obat-mujarab-penyakit-kuning.html
    http://www.klikgamat.com/2018/06/obat-sipilis-alami-yang-paling-ampuh.html
    http://www.klikgamat.com/2018/06/obat-sinusitis-paling-ampuh-dari-bahan.html
    Reply
  • walatra bersih wanita June 28th, 2018 11:04 am
    https://caramengobatipenyakitasamlambungg.blogspot.com/p/walatra-bersih-wanita.html
    Reply
  • obat herbal alami June 23rd, 2018 6:16 am
    http://www.dehisna.com/2018/06/obat-herbal-alami-gagal-ginjal.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-maag-kronis.html
    http://www.dehisna.com/2018/06/obat-herbal-untuk-nyeri-sendi.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-penyakit-jantung.html
    http://www.dehisna.com/2018/06/obat-herbal-hipetensitekanan-darah.html
    http://www.dehisna.com/2018/06/obat-herbal-untuk-menghilangkan-bekas.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-batu-empedupenyakit.html
    http://www.dehisna.com/2018/06/obat-herbal-alami-flek-hitam-pada-paru.html
    http://www.dehisna.com/2018/06/obat-herbal-ginjal-bocor.html
    http://www.dehisna.com/2018/06/obat-herbal-penyakit-liver-atau-hati.html
    http://www.dehisna.com/2018/06/obat-herbal-penyakit-kanker-payudara.html
    http://www.dehisna.com/2018/05/obat-herbal-penyakit-hernia-hiatus.html
    http://www.dehisna.com/2018/05/obat-mujarab-untuk-herpes.html
    Reply
  • barokahherbal7 June 8th, 2018 4:38 am
    http://obatdarahtinggi.my.id/
    http://qncjellygamat.my.id/
    http://agenqncjellygamatbekasi.top/
    http://obatdarahtinggi.my.id/qnc-jelly-gamat/
    http://walatrasehatmata.biz.id/
    http://obatdarahtinggi.my.id/agen-qnc-jelly-gamat-jakarta-timur/
    http://obatdarahtinggi.my.id/walatra-zedoril-7/
    http://obatdarahtinggi.my.id/walatra-squalene-softgel/
    http://obatdarahtinggi.my.id/walatra-spirulina-plantesis/
    http://obatdarahtinggi.my.id/walatra-sehat-sendi/
    http://obatdarahtinggi.my.id/walatra-sarang-semut/
    http://obatdarahtinggi.my.id/walatra-madu-murni/
    http://obatdarahtinggi.my.id/obat-lemak-darah-tinggi/
    http://obatdarahtinggi.my.id/obat-paru-paru-berair/
    http://obatdarahtinggi.my.id/obat-muntah-darah/
    http://obatdarahtinggi.my.id/obat-hipertiroid/
    http://obatdarahtinggi.my.id/obat-radang-prostat/
    http://obatdarahtinggi.my.id/cara-mengobati-turun-berok/
    http://obatdarahtinggi.my.id/obat-kista-bartholin/
    http://obatdarahtinggi.my.id/obat-asam-urat/
    http://obatdarahtinggi.my.id/obat-infeksi-mata/
    http://obatdarahtinggi.my.id/obat-tumor-payudara/
    http://obatdarahtinggi.my.id/obat-tbc-kelenjar/
    http://obatdarahtinggi.my.id/obat-mata-juling/
    http://obatdarahtinggi.my.id/obat-infeksi-telinga/
    http://walatrasehatmata.biz.id/
    http://obatdarahtinggi.my.id/walatra-lycopene-softgel/
    http://obatdarahtinggi.my.id/obat-asam-lambung/
    http://obatdarahtinggi.my.id/walatra-hexabumin/
    http://obatdarahtinggi.my.id/obat-kencing-darah/
    http://obatdarahtinggi.my.id/obat-patah-tulang/
    http://obatdarahtinggi.my.id/walatra-habbaza-softgel/
    http://obatdarahtinggi.my.id/obat-diabetes/
    http://obatdarahtinggi.my.id/obat-kelenjar-getah-bening/
    http://obatdarahtinggi.my.id/obat-kanker-payudara/
    http://obatdarahtinggi.my.id/obat-kelenjar-tiroid/
    http://obatdarahtinggi.my.id/obat-paru-paru-basah/
    http://obatdarahtinggi.my.id/walatra-albumin/
    http://obatdarahtinggi.my.id/walatra-sehat-wasir/
    http://obatdarahtinggi.my.id/gamat-emas-kapsul/
    http://obatdarahtinggi.my.id/obat-jantung-koroner/
    http://obatdarahtinggi.my.id/obat-sakit-tulang-ekor/
    http://obatdarahtinggi.my.id/obat-infeksi-paru-paru/
    http://obatdarahtinggi.my.id/walatra-black-garlic/
    http://obatdarahtinggi.my.id/obat-herpes-zoster/
    http://obatdarahtinggi.my.id/obat-stroke-ringan/
    http://obatdarahtinggi.my.id/obat-usus-buntu/
    http://obatdarahtinggi.my.id/obat-infeksi-lambung/
    http://obatdarahtinggi.my.id/obat-tipes/
    http://obatdarahtinggi.my.id/obat-liver/
    http://obatdarahtinggi.my.id/obat-sesak-nafas/
    http://obatdarahtinggi.my.id/obat-kanker-darah/
    http://obatdarahtinggi.my.id/obat-mata-minus/
    http://obatdarahtinggi.my.id/obat-rematik-tulang/
    http://obatdarahtinggi.my.id/obat-tukak-lambung/
    http://obatdarahtinggi.my.id/obat-herpes-genitalis/
    http://obatdarahtinggi.my.id/obat-infeksi-saluran-kemih/
    http://obatdarahtinggi.my.id/walatra-jelly-gamat-original/
    http://obatdarahtinggi.my.id/obat-benjolan-diketiak/
    http://obatdarahtinggi.my.id/obat-batu-empedu/
    http://obatdarahtinggi.my.id/obat-stroke/
    http://obatdarahtinggi.my.id/obat-kista/
    http://obatdarahtinggi.my.id/walatra-bersih-wanita/
    http://obatdarahtinggi.my.id/obat-epilepsi/
    http://obatdarahtinggi.my.id/obat-gagal-ginjal/
    http://obatdarahtinggi.my.id/qnc-jelly-gamat/
    http://obatdarahtinggi.my.id/obat-gondok/
    http://obatdarahtinggi.my.id/obat-eksim/
    http://obatdarahtinggi.my.id/obat-batu-ginjal/
    http://obatdarahtinggi.my.id/obat-kolesterol/
    http://obatdarahtinggi.my.id/obat-ambeien/
    http://obatdarahtinggi.my.id/walatra-brain-nutrition/
    http://obatdarahtinggi.my.id/obat-bronkitis/
    http://obatdarahtinggi.my.id/obat-hipertensi/
    http://obatdarahtinggi.my.id/walatra-green-coffe/
    http://obatdarahtinggi.my.id/obat-rematik/
    http://obatdarahtinggi.my.id/obat-eksim-kering/
    http://obatdarahtinggi.my.id/obat-herpes-simplex/
    http://obatdarahtinggi.my.id/obat-kencing-manis/
    http://obatdarahtinggi.my.id/obat-radang-sendi/
    http://obatdarahtinggi.my.id/agen-qnc-jelly-gamat-bekasi/
    http://obatdarahtinggi.my.id/obat-maag/
    http://obatdarahtinggi.my.id/obat-pelangsing/
    http://obatdarahtinggi.my.id/obat-kaki-gajah/
    http://obatdarahtinggi.my.id/walatra-berry-jus/
    http://obatdarahtinggi.my.id/obat-hernia/
    http://obatdarahtinggi.my.id/obat-penambah-nafsu-makan-anak/
    http://obatdarahtinggi.my.id/obat-asma/
    http://obatdarahtinggi.my.id/obat-diet/
    http://obatdarahtinggi.my.id/obat-hepatitis/
    http://obatdarahtinggi.my.id/obat-rematik/
    http://obatdarahtinggi.my.id/obat-nyeri-haid/
    http://obatdarahtinggi.my.id/obat-leukimia/
    http://obatdarahtinggi.my.id/obat-rabun-jauh/
    http://obatdarahtinggi.my.id/obat-luka-operasi/
    http://obatdarahtinggi.my.id/obat-amandel/
    http://obatdarahtinggi.my.id/obat-pemutih-wajah/
    http://obatdarahtinggi.my.id/obat-lupus/
    http://obatdarahtinggi.my.id/susu-kambing-etawa/
    http://obatdarahtinggi.my.id/walatra-propolis/
    http://obatdarahtinggi.my.id/obat-penggemuk-badan/
    http://obatdarahtinggi.my.id/madu-anak/
    http://obatdarahtinggi.my.id/cara-menghilangkan-bopeng/
    http://obatdarahtinggi.my.id/obat-penghilang-jerawat/
    http://obatdarahtinggi.my.id/obat-keputihan/
    http://obatdarahtinggi.my.id/obat-demam-berdarah/
    http://obatdarahtinggi.my.id/obat-radang-usus/
    http://obatdarahtinggi.my.id/obat-cacar-air/
    http://obatdarahtinggi.my.id/obat-insomnia/
    http://obatdarahtinggi.my.id/obat-kesemutan/
    http://obatdarahtinggi.my.id/obat-kurap/
    http://obatdarahtinggi.my.id/obat-kudis/
    http://obatdarahtinggi.my.id/masker-pemutih-wajah/
    http://obatdarahtinggi.my.id/madu-penambah-berat-badan/
    http://obatdarahtinggi.my.id/obat-vagina-gatal-dan-bau/
    http://obatdarahtinggi.my.id/obat-rabun-dekat/
    http://obatdarahtinggi.my.id/obat-sakit-vagina/
    http://obatdarahtinggi.my.id/obat-chikungunya/
    http://obatdarahtinggi.my.id/obat-presbiopi/
    http://obatdarahtinggi.my.id/propolis-brazil/
    http://obatdarahtinggi.my.id/cabang-resmi-alwiherbal-net-bekasi-timur/
    http://obatdarahtinggi.my.id/obat-tuberkulosis/
    http://obatdarahtinggi.my.id/madu-penambah-nafsu-makan/
    http://obatdarahtinggi.my.id/obat-korengan/
    http://walatrasehatmata.biz.id/2017/10/13/obat-mata-katarak/
    http://walatrasehatmata.biz.id/2017/10/13/obat-mata-glaukoma/
    http://walatrasehatmata.biz.id/2017/11/17/obat-mata-minus/
    http://walatrasehatmata.biz.id/obat-mata-plus/
    http://walatrasehatmata.biz.id/obat-mata-juling/
    http://walatrasehatmata.biz.id/obat-ablasio-retina/
    http://walatrasehatmata.biz.id/cara-mengecilkan-perut-buncit/
    http://walatrasehatmata.biz.id/obat-rambut-rontok/
    http://walatrasehatmata.biz.id/obat-gusi-bengkak/
    http://walatrasehatmata.biz.id/agen-qnc-jelly-gamat-bekasi-utara/
    http://walatrasehatmata.biz.id/obat-darah-tinggi/
    http://walatrasehatmata.biz.id/obat-ambeien/
    http://walatrasehatmata.biz.id/obat-paru-paru-basah/
    http://walatrasehatmata.biz.id/obat-asam-lambung/
    http://walatrasehatmata.biz.id/obat-kolesterol/
    http://walatrasehatmata.biz.id/obat-jantung-koroner/
    http://walatrasehatmata.biz.id/obat-asam-urat/
    http://walatrasehatmata.biz.id/obat-stroke/
    http://walatrasehatmata.biz.id/qnc-jelly-gamat/
    http://walatrasehatmata.biz.id/obat-tipes/
    http://walatrasehatmata.biz.id/obat-turun-berok/
    http://walatrasehatmata.biz.id/obat-batu-empedu/
    http://walatrasehatmata.biz.id/obat-gagal-ginjal/
    http://walatrasehatmata.biz.id/obat-kelenjar-getah-bening/
    http://walatrasehatmata.biz.id/obat-infeksi-telinga/
    http://walatrasehatmata.biz.id/obat-hipertiroid/
    http://walatrasehatmata.biz.id/obat-tbc-kelenjar/
    http://walatrasehatmata.biz.id/obat-rabun-jauh/
    http://walatrasehatmata.biz.id/obat-rabun-dekat/
    http://walatrasehatmata.biz.id/obat-kuat/
    http://qncjellygamat.my.id/
    http://qncjellygamat.my.id/obat-kencing-batu/
    http://qncjellygamat.my.id/obat-stroke/
    http://qncjellygamat.my.id/obat-radang-tenggorokan/
    http://qncjellygamat.my.id/obat-kencing-manis-kering/
    http://qncjellygamat.my.id/obat-asma/
    http://qncjellygamat.my.id/obat-eksim/
    http://qncjellygamat.my.id/obat-lupus/
    http://qncjellygamat.my.id/obat-kanker-prostat/
    http://qncjellygamat.my.id/cara-menghilangkan-bopeng/
    http://qncjellygamat.my.id/madu-anak/
    http://qncjellygamat.my.id/obat-nyeri-sendi/
    http://qncjellygamat.my.id/obat-migrain/
    http://qncjellygamat.my.id/obat-mata-silinder/
    http://qncjellygamat.my.id/obat-luka-bakar/
    http://qncjellygamat.my.id/obat-leukemia/
    http://qncjellygamat.my.id/obat-kanker-mulut/
    http://qncjellygamat.my.id/obat-kurap/
    Reply
  • Alex Rushforth July 20th, 2016 11:08 am
    Sorry I'm a little late to this debate - the system lost my original text. So here goes again...
    The argument of Larivière et al’s paper which started this debate seems to rest on a kind of path dependency- account – the scientific system is locked-in around an inferior tool when there are better alternatives available. Although I would agree with this, I also think path dependency accounts are not the only way forward in these kinds of debates –for my money they can be a bit too restrictive, as by suggesting salvation lies in adopting a superior tool they close-off a lot of what is going on in the research system which we should pay closer attention to. What I think is more important is to understand the kinds of conditions under which something like the impact factor can come to have such an big effect on the way research is conducted and governed – in interviews Sarah and I conducted with scientists this revolved around issues like scarcity of resources, excessive quantities of scientific literature from which to choose what to read (and cite), fiercely competitive job markets, the fact researchers are incentivized to write and not to read etc. It is confronting these sorts of issues (which studying the uses of the impact factor points us toward) more than technical limitations of the impact factor per se which I think would help produce better systems of evaluation and ultimately better science. I fear that getting journals to publish distributions alongside JIFs will not loosen the grip of the impact factor – it’s not that researchers we spoke to aren’t aware of limitations in how the indicator is calculated – it’s more they recognize it’s the de facto standard against which their prospect for external grants or a job interview will depend. It’s these latter kinds of issues which will need to change first if the impact factor is to go away.
    Reply
  • Sabrina Petersohn July 18th, 2016 2:13 pm
    Sarah, in line with our recent conversations I take it this is an illustration of how to "embrace" the JIF and h-index in the form of an extended research agenda. Two thoughts pop up: Studying the way the JIF has become embedded in research and assessment practices opens up a perspective on the positions of researchers as active users of metrics and not simply as evaluated subjects. This means probably stating the obvious here but it is a very interesting position to consider in terms of the relationships between indicator developers and users. Also the issue of the dependency on and adequacy in specific usage contexts of indicators comes up. Thanks for the food for thought, Sabrina
    Reply
  • Mark Patterson July 18th, 2016 11:15 am
    As another author on the paper in question, I’d like to add that I agree completely with your point towards the end that the JIF is "so engrained in different knowledge producing activities in different fields”. Many people have been *talking* about this thorny problem for years now.
    I was pleased to participate in this collaboration because I think it sends a signal that journals are now prepared to do something about this as well. Recently the ASM Journals went further by announcing that they would no longer promote impact factors on their websites (https://www.asm.org/index.php/asm-newsroom2/press-releases/94299-asm-media-advisory-asm-no-longer-supports-impact-factors-for-its-journals). Neither do eLife or PLOS.
    We’re not going to change things overnight, but for journals that do promote impact factors, sharing the citation distribution underpinning the impact factor is another way to point out that using the impact factor is a poor way to evaluate any individual article or scientist. And to reinforce Stephen’s point, we are not arguing that we should move to a situation where we only look at citations to articles instead. Practices in research evaluation have to change. There are some examples on the DORA website - http://www.ascb.org/inspiration-and-good-practices/.
    Reply
  • Stephen Curry July 14th, 2016 11:44 am
    It’s interesting – and stimulating – to see how different people read the same work. Let me give a brief initial reaction (because I am still digesting) to a couple of points.
    First I find your hypothetical situation a little odd (and as with Ludo Walkman’s scenarios), not very congruent with my own experience. To be honest I am astonished by the assertion in regard to biomedical research that: “what a committee can do hypothetically in this field, on the basis of the JIFs, is assess whether a researcher mainly does descriptive work or primarily publishes about biological mechanism.” This strikes me as an indefensible use of JIFs.
    Second, perhaps I need to re-read my own paper, but I did not intend to shift the focus to the “citation impact” of individual papers, though clearly we wanted people to move beyond simplistic interpretations of JIFs and focus on the content of papers. Indeed our closing paragraph states, “However, we hope that the broader message is clear: research assessment needs to focus on papers rather than journals, keeping in mind that downloads and citation counts cannot be considered as reliable proxies of the quality of an individual piece of research.”
    To my mind the aim of our work was not to eliminate impact factors but to increase awareness of the data that underlies them (and the real variation within it). There may well be nuances that we have overlooked or perhaps underplayed – but I shall continue to reflect.
    Reply
Share on:
Subscribe to:
Build on Applepie CMS by Waltman Development