This list is actually not compiled using Scopus data. No citation analysis was involved in producing this list. It is a listing of top downloaded articles on ScienceDirect.

Best,
Michael Habib
Product Manager, Scopus
habib@elsevier.com

==>

Computing giants launch free science metrics: Nature news

“New Google and Microsoft services promise to democratize citation data.”

http://www.nature.com/news/2011/110802/full/476018a.html

Just a quick clarification, the above ranking isn’t actually from Scopus, it is from ScienceDirect. It is only looking at raw usage counts of full-text available in ScienceDirect. Scopus data is not used in this lists compilation. Scopus citations were not considered when this list was calculated. I apologize if any messaging around the site was misleading..

Best,
Michael Habib
Product Manager, Scopus.

Microsoft Academic Search may not be there yet, but it is getting there.
And Google Scholar isn’t that bad – but either is better than Scopus.

This could be a direction of improvement for several of our CS domains. If replicability of experiments becomes a value of CS (as in Kuhn’s definition of a paradigm), this may impact on the way we evaluate research and researchers. However, note that values are seldom decided a priori; instead, they grow from the “milieu”.

Very interesting!

I wish here to provide some contributions to the discussion on the role of publications and how we as a community should seriously engage in a debate. Let me start by saying that I found Naughton’s points, which spurred this debate, very well taken. They are very much in line with a keynote I gave at ICSE 2009 (which can still be found online here http://www.ecss2011.polimi.it/). The spirit of the two presentations is strikingly very similar: start with reflection within an area and then generalize to question the way we do research and how we assess it.

The point I wish to make here is that in part we (academics) are all co-responsible for the very poor way we evaluate research (and, as a consequence, researchers) and it is in the first place our responsibility to change this. The obsession on papers and publications has been our fault and has been offered to others who are now evaluating us. And the obsession has mostly boiled down to “counting” (the number of publications, the number of citations, etc.). This obsession is now at all levels. Even governments tend to evaluate how productive we are (and how “useful” we are for society) by these metrics.

I believe it is our responsibility to say that publications per se are irrelevant and that we need to assess the more general “impact” of work. The notion of “impact” is not easy to condense in just one indicator, and certainly (as I discuss in my talk) impact has no relation with “impact factor”, h-index, or similar things. To move down to earth, I believe that we must recognized that “papers” increasingly less help in making progress (research that builds on other research) and transferring results to the world. Actually, they tend to slow down the process. We need to enlarge our view of “what are the results of research” and recognize that many other artifacts are equally or more important than papers, and we need to be able to reward people for producing them. In my talk I mentioned, I did an experiment in which I looked the papers published on a very respected journal, which mentioned the existence of a tool or demonstrator developed as part of the research described in the paper. In almost all cases either the tool did not run correctly, or wasn’t installable, or simply the tool did not exist. I have personally experienced the case in which, having read a paper that mentioned the existence os a tool or interesting data, I tried to get them, but almost invariably had to give up in the end. This story gives an example of what I mean by impact.

If we insist on evaluating papers as our primary object to measure and do not extend the notion of research result to include other artifacts (data sets, tools, etc.) we will be co-responsible for the situation in which we are.

Let me end here with an example of a positive move, which should be generalized in our community. The forthcoming ESEC/FSE conference is trying as an experiment to have an “artifact evaluation” phase in which papers accepted by the conference submit the “artifacts” developed as part of the research documented in the paper, and these are evaluated for replicability (http://2011.esec-fse.org/cfp-artifact-evaluation). This is just an initial step (and spurred heated debates when was proposed). In my opinion, instead, this should become common practice in our field. A check of replicability of results, usability of tools, … should be part of the standard acceptance process for contributions to conferences and journals.

Lat, but not least, notice that if we make the selection process of contributions to be inclusive of all artifacts that demonstrate the ability to really build on the reported research, we also slow down the submission rate and let reviewers concentrate on much fewer, but more crucial, things to assess.

This is an excellent suggestion.

If everybody would agree on putting a cap to no more than two papers a year when evaluating both researchers and projects, that would make each one consider carefully when writing a new paper whether that paper is indeed better than any of his own two most recent papers.

This will avoid the very common phenomenon of almost identical papers submitted to different conferences and journals. If publishing is digital, not Gutenberg-ian, any improvement to a result could be just simply added to a single publication, keeping a versioning system to allow distinguishing the evolution of the work.

It seems that question of publications is partial question of more generic problem.

Society needs science system as an economic force. Up to now universities do not have an intention to become the economic force and to act as the economic force. Universities require and receive resources as an economic force but act as art or culture bodies. Such contradiction can not last long. There are possible several scenarios:

1) “As is” – universities ignore society needs, society ignores universities needs, influence of universities will decrease, new player in a science system will appear to satisfy society needs;

2) Optimistic – universities will rethink their role in society and accept the target to become an economic force with well balanced education, research and innovation components. Such goal will be supported by adequate criteria of academic career for each individual. This means that instead of publications as the sole criteria for academic career universities will introduce assessment criteria with equal value of education, research and innovation achievements.

3) Pessimistic – status of universities will transform into art and culture institutions mainly humanitarian. Number of universities will decrease ten times. Research will be done mainly by structures like Google. Education will transform to the training to find knowledge instead of building capability to learn new knowledge. Pessimistic scenarios is quite similar to “as is” scenarios;

4) …

“Not only does the mechanism of peer review fail to protect us from disasters, in a certain way it guarantees mediocrity (…) I sooner blame someone for his publication list being too long than being too short.”

My idea for a paradigm-changing would be to subvert it by imposing a maximum number of papers by researcher/year. In this way we would be more available to do real research work (and, possibly, valorizing technical reports).

More and more people are in favour of de-emphasizing conference papers… As Naughton says “Why are we still behaving as if our proceedings are printed by overworked monks on a Gutenberg press?”

I think that returning to a situation where conferences are *primarily* a place where to meet people and do professional networking is an essential step in “rethinking the environment we have created and what it encourages”. And I think ECSS could play a role in this process…

Naughton concludes “may be the next big idea in DB research is … a new community model”: I think we can generalize “DB” to “CS” in his sentence…