INTRODUCTION
In June 2011 Thomson-Reuters' Journal Citation Reports® — Science Edition service published the Impact Factors of all peer reviewed academic journals. Fourteen of the thousands of publications analysed are now grouped in a single subject area called Primary Health Care.1 With a 2010 Impact Factor (IF) of 2.070 the British Journal of General Practice (BJGP) remains the world's second most highly-cited journal of general practice and primary health care.
The 2010 IF is a figure representing the mean number of times peer-reviewed papers published in a journal in 2008 and 2009 are cited in other journals, or are self-cited in the journal of origin, during 2010. The total number of research papers published in the BJGP in 2008–2009 was 185 and the total number of cites in 2010 was 383, giving the IF of 2.070. In common with a number of other general medical and primary care journals, the BJGP's IF has fallen since last year. Table 1 also shows the 5-year IF, which is calculated in the same way for the 2010 citations of papers published in the preceding 5 years (2.664).
In this article we examine the significance of IFs and other bibliometric indices for authors, publishers, and research funders and have tried to identify the characteristics of highly-cited articles. We are particularly interested in the possible effects on bibliometrics of our recently-introduced paper short:web long publishing strategy and re-organised journal content.
BIBLIOMETRICS
Journal Citation Reports
The Journal Citation Reports (JCR)2 provide a set of bibliometric indices, including:
The previous year's and 5-year journal IFs, calculated as described above.
The Immediacy Index, calculated by dividing the number of cites to 2010 articles (184) by the number of articles published in 2010 (99) = 1.859 for the BJGP.
The Journal Cited Half-Life, which is the median age of the BJGP's items cited in the current JCR year (7.2 years), and the cited journal graph which shows the year-by-year distribution of citations to the journal. Cited information is most relevant to our IF; that is, how our journal content is cited by others and self-cited by us and our authors.
The Journal Citing Half-Life, which is the median age of items cited by the BJGP in the current JCR year (5.7 years), and the citing journal graph shows the year-by-year distribution of citations by the journal. Citing information tells us about the currency of references that we're publishing in our own journal.
Eigenfactor
Thomson Reuters (Scientific) also support the Eigenfactor™ (http://www.eigenfactor.org). The Eigenfactor score is ‘a measure of the overall value provided by all of the articles published in a given journal in a year’ and the Article Influence™ score is ‘a measure of a journal's prestige based on per article citations and is comparable to IF’.3 The BJGP ranking in this system is similar to its position in the IF tables. The BJGP Eigenfactor and Article Influence scores for 2010 were 0.00872645 and 0.797143 respectively. The site also allows graphical ‘mapping’ of journal citations to the subject areas of citing journals.
SCImago Journal Ranking
An alternative set of journal metrics is published by SCImago, using information contained in Elsevier's Scopus® database (http://www.scimagojr.com). The SCImago Journal Rank (SJR) is an indicator ‘that expresses the number of connections (that is, citations) that a journal receives through the citation of its documents, divided between the total of documents published in the year selected by the publication, weighted according to the amount of incoming and outgoing connections of the sources’.4 In other words, the citation assessment is weighted according to the influence, based on their citation rates, of the citing journals.
The SCImago website also provides information about the extent of international collaboration in the published articles and the H (Hirsch) index, which quantifies both the scientific productivity and scientific impact of the journal. Online comparison across the 22 primary care and family medicine journals is more difficult to do using the SCImago site because the Family Practice category does not contain all the primary health care/general practice journals.
Google Scholar
Google Scholar (http://scholar.google.com/) can be used to capture citation rates and to calculate impact factors, which are reasonably well-correlated with those published in other bibliometric sites.5
PUBLISHING IN THE BJGP
The BJGP receives around 500 original articles for consideration of publication as original research annually, and this number rises every year. Publishing around 120 original papers annually means that the BJGP's acceptance rate is around 24%. About a quarter of papers are rejected on editorial screening on the basis of relevance, quality and interest, and the remainder are submitted to an open peer review system in which at least two external reviewers are involved. Further reviews, often from statistical advisers, and opinions from members of the BJGP Editorial Board, are often requested. Most accepted papers require revision in response to the reviewers' and the editor's comments and authors are asked to accompany the revised full version of the paper, which is published online, by an 800-word summary of the paper which will appear as a two-page article in the print journal. Because of this paper short:web long publishing strategy the number of pages in the re-designed print journal has been reduced from 80 to 56, while the online version runs to an average of 120 pages. We have recently analysed the papers published in the BJGP in 2005–2010. The total number of articles was 1989, of which 552 (28%) were original research papers. Publications with UK lead authors accounted for 69%; 14% were from the Netherlands and the remainder were from a further 19 countries. Quantitative studies accounted for 80% of original research papers and 12.5% were randomised controlled trials. Of those with a clear clinical focus, papers on mental health, cardiovascular, respiratory and musculoskeletal disorders, diabetes, child health, cancer and neurology accounted for half. When we looked at the research approach taken in these papers, 70% could be classified as studies of service evaluation, diagnosis, prescribing, epidemiology, and patient perspectives.
We were interested to know which factors contribute to a higher citation rate. Adjusted regression models showed that, of the various research methods used in published studies, systematic reviews and meta-analyses were most highly cited (RR [relative risk] 1.98, P<0.001), and among subject areas, papers on communication (RR 1.65, P = 0.03) and diagnosis (RR 1.32, P = 0.02). The only clinical topic that emerged as more highly cited than a ‘non-specific’ reference group was terminal illness (RR 1.58, P = 0.01); care of older people (RR 0.66, P = 0.03) and, counter-intuitively, infection (RR 0.52, P = 0.02) were less cited among clinical topics. In the last 5 years, 48 BJGP papers received more than 10 citations, and these were on diagnosis and disease detection, chronic disease management, musculoskeletal and other painful conditions, depression, and diabetes, with some overlap between these. We also wondered whether editorials in the BJGP were likely to be more or less highly cited than research papers: about 12% of editorials published between 2005–2010 were cited four or more times, compared with 36% of original papers having four or more citations.
ONLINE IMPACT
We have reported the high level of online readership and numbers of article downloads of BJGP papers.6 Recent data from PubMed Central® (PMC) and ingentaconnect™ — the two platforms which host the online journal archive and current content — show a continuing upward trend in online ‘retrievals’ of BJGP articles (PDFs, full text downloads and other page views). The figure currently stands at about 160 000 per month from PMC (just under 2 million per year), well over double the figure for 2006, with Ingenta page views running at an additional 42 000 per month (500 000 per year). The relationships between download rates and article citations, and also with some of the prestige indicators mentioned above, is likely to be complex, but it appears that the journal has a significant online presence which is widely accessed.
DISCUSSION
A number of methods are available for determining the impact of a scientific journal on its subject area, ranging from the JCR IF, based on recent citations of peer-reviewed original papers, to broader assessments of the extent to which these and other material, editorials, analysis and review papers and so on, are accessed and used. Some of this impact is captured in the H Index, and some through measurements of online downloads. Perhaps looking at 5-year impact is more appropriate for publications at the ‘applied’ end of the research spectrum, because the 2-year figure tends to favour the faster turn-around subjects, generally in the basic sciences.7 The longer-term influences of individual papers or a series of papers in a cognate area is, of course, much more difficult to measure, and the process of getting research evidence into practice has long been a subject of intense, and often fruitless, study.8 Methods of approaching this might include seeking evidence of the use of published research data in clinical practice guidelines or clinical quality criteria, or of policy analyses in government or other professional publications. Establishing the influence of publications on the behaviour of individual clinicians, for example in adopting evidence-based practices or abandoning practices shown to be ineffective, is a longer term and more complex and challenging task.9
In the UK's 2008 Research Assessment Exercise, in which assessment panels used peer review of selected papers rather than bibliometrics, the primary care panel's judgement was that more than half the outputs submitted to them were ‘internationally excellent or world-leading’. The quality of the UK's primary care research, compared with data from Australia and other European and North American countries, has been confirmed in a more recent bibliometric analysis, in which citation metrics were the main outcome measure.10
These are critical times for general practice and primary care because many health systems, struggling to maintain quality of care in a difficult financial climate, are looking closely at the ways in which primary care can be re-shaped to take more of the strain of caring for an aging population with increasingly complex and expensive comorbidities. This makes research in and on primary care, aimed at improving methods of diagnosis, treatment and prevention more essential than ever, and the effective dissemination of the results of this research is clearly of great importance too. Health service reform must ensure that the funding and infrastructure required to sustain strong primary care research remain in place. While the IF scores provide a ready tool to make judgements about the ‘quality’ of research published in a journal, IFs alone are self-evidently a limited measure of the real impact of an article on professional behaviour and health outcomes. The recent report from the House of Commons Science and Technology Committee11 warns about the over-use of IFs in research assessment. Although desirable, capturing the broader ‘societal’ impact of research remains a considerable challenge.12 It will be interesting to see how ‘impact’ is operationalised in the UK's forthcoming Research Excellence Framework (REF), the successor to the Research Assessment Exercise, in the assessment of higher education institutions. Recent guidance indicates that REF sub-panels will assess the ‘reach’ and ‘significance’ of impacts on the economy, society, and/or culture and that impact will contribute 20% of the score for REF units of assessment.13
The increasing use of web technology is likely to have a major impact on the way in which scientific research is published and accessed in the future. Open access systems, with publishing costs built into research grants, will become more common and are likely to be associated with faster publication and probably higher citation rates. The place of print publication will be gradually eroded as tablets and apps become more widely used, and this will improve our ability to disseminate research wore widely and to make papers available in resource-poor locations. The volume of information generated by medical research is likely to go on increasing and there will be a continuing need for effective methods of pre- and perhaps post-publication review. Research into the effectiveness of review methodologies, the best approaches to dissemination and, critically, the assessment of the clinical and societal impacts of research outputs will continue to be essential.
Acknowledgments
We are grateful to Professor Roy Pounder for valuable suggestions about bibliometrics and to Professor Chris Salisbury and Dr Liam Smeeth for comments on an earlier draft of this paper.
Notes
Provenance
Freely submitted, externally reviewed.
Competing interests
All authors have an interesting in promoting the quality of research published in the BJGP. Catharine Hull, Erika Niesner and Roger Jones are employed by the Royal College of General Practitioners and Peter Schofield is a statistical advisor for the BJGP.
- © British Journal of General Practice 2012