One of the most actively debated postings on the blog has been the question that I asked 14 months ago- Peer-review. Is it outdated?
The overwhelming majority of the 46 comments were in favour of peer-review, despite its faults, but the most recent one, from Stephen Gay, of StephenLGay, Australia, expressed very serious concerns which I feel should be more widely debated.
Stephen does not mince his words. He feels that the peer review process is a “total shambles”. He says:
It is OK if you are using conventional approaches, but if your methods are new - particularly using advanced mathematics - it is almost impossible to get a fair review. Further, the editors tend to directly send your papers to your competitors. When it is obvious that the reviewer is incompetent the Editor tends to do nothing. That is the process is more important than the outcome. At this stage I generally don't bother sending papers to journals, as the 'leading academics' are so far intellectually behind it is impossible for them to understand simple concepts.
I had to take issue with Stephen on this. I acknowledge that the peer-review system certainly does have its faults but it is not a shambles. It was obvious to me that Stephen had had some bad experiences with journals, but I think many academics might find it a little insulting to be regarded intellectually inferior and incapable of reviewing innovative work using advanced mathematics. I also had to disagree with his generalisations on Editors!
Nevertheless I thank Stephen for his opinion and for bringing this important aspect of peer-review into the open, hopefully leading to further debate. What do you think, and have any of you had bad experiences in this respect?
Monday, 21 May 2012
Subscribe to:
Post Comments (Atom)
.jpg)
There are two sides here, and I have experienced both.
ReplyDelete1/ As a reviewer I often come across papers that just loads several equations after each other, without a clear line of thought, or just a repetition of what is found in the literature. Sometimes, though, there is a new concept introduced, and then it is in my role as reviewer to either deny reviewing, or to find some extra help in understanding what is written, if I cannot understand it.
2/ As an author I have to be careful when introducing new concepts to a traditional journal, and really build my case. Most important then is to suggest possible reviewers, who are sufficiently knowledgeable in the field. If you despite this, receive an (in your opinion) unfair review, you may send a letter to the editor and ask for a reexamination. However, first read the comments and try to understand what was unclear and badly presented (and take a deep breath or two).
I have never come across a bad review that tries to "sink you" out of perceived envy.
I am fully in support of the peer review process. A lot of nonsense gets published – in conference proceedings particularly where some of the papers are just there to fill space or even worse are long adverts by vendors.
ReplyDeleteMy only issue with advanced - or for that matter any mathematics - is the use of shorthand notation and the skipping of calculation steps. I was at university some 15 years and I don't always remember what curly brackets, dots and underscores mean. It would be helpful if editors and peer reviewers forced journal authors to show their workings in manners which practicing engineers can follow.
The peer review process is necessary. However, it has its problems and most researchers who publish had some bad and some good experiences. I think we should be very careful to place our bad experiences under the same umbrella of incompetence and incapable and intellectually inferior, as this is unfortunately not fair and totally unbalanced. I had bad experiences and after my first reaction of anger and similar feelings to that of Stephen I read the comments from the reviewers again and most of the time came to some sort of agreement with some of their comments, that allowed me to focussed on the ones I do not agree with and explain to the reviewers why I disagree and accept and change the ones where they are correct.
ReplyDeleteIf it is a really new innovative field, my opinion as a reviewer is that when two reviewers totally disagree on the content and quality, the editor usually send it to another 1 or 2 reviewers to ensure that the paper is reviewed effectively. I must state that not all journal do this but that is something the author can request if he or she is not happy with the reviewing process.
At the end of the day in all professional professions over many decades the only way to get you work evaluated and out in the public domain fairly is by peer review process with all its advantages and disadvantages.
As a reviewer, if we really feel we are "intellectually far behind", we should decline reviewing the paper and point towards more educated colleagues in the field. I don't think we will necessarily end up to the competitor... and honestly, I don't think scientists can be regarded as competitors.
ReplyDeleteIt is also the author's responsibility to send his paper to the most adequate journal.
If I take the example of advanced image analysis which I know best, I think that new tools and theories should be published in specialised journals (ex. Image Analysis and Stereology).
I do not expect the readership of Minerals Engineering to be interested by the mathematical developments but by the innovative applications. On the other hand, there must be some guarantee that the theory is sound and well recognized by knowledgeable experts.
Mr Pirard,
DeleteI can assure you that in the world of lucrative grants and other forms of research funding, the knives are often out and competitors very definitely exist amongst academics. And let's face it: A researcher's publication list pays his way in terms of reputation.
So, I suspect some elbowing going on from time to time.
Many readers will be aware of the recent boycott of Elsevier, mainly the issue is the financial model, forcing universities to buy journals in bundles, but it has certainly brought out the knives! (see http://thecostofknowledge.com/#list). There is clearly huge dissatisfaction with the journal system - criticisms like Stephen's are common - the gatekeepers of publication are often the old guard, and while the vast majority of reviewers and editors force themselves to carefully weigh the value of novel approaches, we are all human, and cannot help our own bias, prefer our own theories, and it is only through accepting one's own mistakes that one can accept new advances in your field. I strongly suggest reading "Mistakes were made (but not by me)" by Carol Tavris.
ReplyDeleteAs for me, I feel the current system is
a) too slow
b) conservative ('cliquey')
c) does not take advantage of the power of the internet (no system to allow innovation, to 'try things out')
d) does not encourage publication of negative results (we learn from our errors, not our successes)
I think papers will increasingly be published online, with peer review open to all and their comments appended for all to see - much like a blog. While it will allow through heaps of rubbish, will also allow the good stuff to be found, as the cream always rises...
I don't want to get involved with the issue of Elsevier's financial model, that is for Elsevier to respond to, not me a mere journal editor. As you know all Eslevier's journals are now published online, but I like your idea of opening up papers for online comment. Maybe this is something which Elsevier should consider for ScienceDirect?
Delete(a) Slow reviews can be a problem. Editors and Reviewers need to be mindful of this. On the other hand, the last review request I had was to agree to complete the review within two weeks (apparently without seeing the manuscript). I don't know the reason for the short timeline, as there was no explanation, but I thought it was too presumptuous and not conducive to a proper review. [N.B. This was not for Minerals Eng., but another well-regarded journal. I did not accept the request.]
Delete(b) Can be conservative/cliquey. Hence choice of reviewers (diversity) is important.
(c) Not sure what you mean here.
(d) Agree.
On point (c) I mean that the internet is making some business models obsolete, think bookshops or travel agents, and only organisations that think about the 'problem they are solving' rather than the product they are selling, will notice competition coming from alternative 'worlds' (Kodak is a good example of such a failure). Here, I am suggesting that our systems of collaboration are evolving (think about how Linux/Ubuntu is created by people collaborating via chat forums). Sure, we can now use email to speed up peer review, and we can access the references faster with SciFinder type services, but we are still limiting review to just a few people when surely for some subjects there are many more people who could made valuable criticisms; forums are now being widely used to review in the arts (photo's, poetry, novels) and the good stuff finds its way to the top - just think of how youtube works. Sure it's mostly junk, but the good stuff gets the exposure via 'likes'. I think this is the way of the future, and if we put 'quality first at all costs' and make no compromise, we may be simply left behind.
DeleteDear Barry
ReplyDeleteAn interesting point of view from Stephen! What he might consider is running his paper past some of his non-competitive colleagues and academics that may be able to give some feedback before it goes to a Journal. A risk one takes with peer review is that it goes to your competitors; one can always request that the editor excludes them.
Another option is to "try out" your concepts at a conference before you submit your publication; the feedback from those presentations is often very valuable in refinig your thinking.
Yes, Stephen has suggested this initial review of the paper in the comments in the posting of 21 March 2011 (peer-review- is it outdated?). I know that papers from a number of high profile research institutes do go through a self-validation process before being submitted to journals. And authors who submit papers to special issues of Minerals Engineering are allowed to test the water first at MEI Conferences.
DeleteAs I have only published in the Minerals Engineering journal, I am not aware of the peer-review procedure that other journals follow, but in the case of my papers, I was required to suggest suitable peer-reviewers to the journal editor.
ReplyDeleteSurely having the author suggest suitable peer-reviewers (as is done by Minerals Engineering) should lead to fair reviews and address Stephen's other main concerns? (papers being sent to competitors and reviewers being incompetent).
Posted on Minerals Engineers Group by Charles Bushell, Mintek, South Africa
Hi Charles. Yes, suggested reviewers can be useful, but I tend to use them with care, as often they are 'friends' of the author. With highly specialised papers I often choose 3 reviewers, one of whom might be a suggested reviewer.
DeleteThere're two issues here: peer review in general, and the specific case of advanced mathematics.
ReplyDeleteThe peer review process is imperfect. You could even say flawed. Some reviewers are knowledgeable and review carefully, others are not or do not. Pressure on researchers to publish more (note "more", not necessarily "better") does not help at all. And authors of pedestrian papers can try their luck.
On the other hand, some reviewers are not open-minded enough. Or think that nothing can be published unless it has been tested on pet mineral system X, and several of the reviewer's own papers have been cited.
Consider the advanced mathematics part. Maybe elliptic integrals have a nifty application in mineral processing. It would be silly to ask an author to explain this starting from an assumed knowledge of high-school mathematics only, when there are plenty of mathematical articles that can be cited. The great difficulties lie in choosing a journal and suggesting reviewers. Does this go into the pure mathematics journal? Surely not, because there is nothing new for them. Into the industry journal? Maybe. But then the mathematics probably have to be shunted off into an appendix, and taken 'on faith' by reviewers. Some 'engineering science' journals exist, that may be appropriate. Anyway, the author does not select reviewers, but only suggests them. There is a risk of sending it to a pure mathematician who finds insufficient novelty in their field of expertise, or of sending it to a practitioner, who distrusts the mathematics they can't understand (at least, until it has been "proven at full scale").
Best for the editor to send to one of each, with instructions:
(1) Dear pure mathematician, please check for mathamatical correctness.
(2) Dear practitioner, assuming the mathematics are correct, is this interesting, useful & a novel application?
Despite these problems, an absence of peer review worse overall than its presence. The flaws can generally be addressed by sending to multiple reviewers and judicious Editorial input.
It is always a possibility to have hostile reviewers for some reasons. It does not mean that the authors of the manuscripts have to give up. Submitting the manuscript to a different journal is an option. It is also the responsibility of the authors to make advanced concepts accessible to less specialized readers.
ReplyDeletePosted by Stephen Gay, Australia, on Minerals Engineers Group:
ReplyDeleteFirstly, I am a touch worried that I am the protagonist in this debate.
I also want to try and distinguish Minerals Engineering from other journals. I had one bad experience with Minerals Engineering only, and I do not think the particular issue is of general interest.
However I want to provide an example of a more significant 'bad example'.
Now I need to say at the outset that some will not agree with my technical position - but I will put that aside for the moment.
I did substantial work in the area of stereological correction (in the context of mineral processing this is the problem of understanding the 3D mineral composition distribution in mineralogical sections).
Now I had done previous work on the binary problem (mineral of interest and gangue only), and also had developed equations to estimate the 3D covariance (the covariance is a measure seldom used by mineralogists).
I came up with a model that linked these two approaches using a multivariate truncated normal distribution (it was truncated because composition must be between 0 and 100%).
The model also included a method for dealing with the 'liberated' particles.
I used a Markov-Chain Monte Carlo (MCMC) approach to get the set of multimineral particles.
I validated the model using a numerical simulation.
So I consider there were lots of claims to originality, and it used advanced methods of geometric probability, multivariate analysis, MCMC and numerical simulation on a problem that is a real mineral processing problem.
I submitted to IJMP, and the paper was rejected.
The reason:
I had used a truncated normal distribution, and in the paper had stated that the truncated normal distribution can be U-shaped as well as bell-shaped.
Any statistician will tell you that a U-shaped truncated normal distribution is perfectly acceptable provided the integral is 1. Yet the reviewer would not accept that the truncated normal distribution could be U-shaped. The reviewer was so dogmatic that they were not even prepared to look at the validation results.
Now this work was later accepted for IMPC - but as far as I am concerned the paper had value to both mineral processing and stereology and should have published in a journal.
I consider I put a huge amount of effort into the ideas and the paper and it was rejected by a reviewer who was unwilling to consider a concept contrary to his 'belief'.
I can cite numerous other examples of where reviewers have been unwilling to accept work contrary to their preconceived beliefs; hence it is my modus operandi that many reviewers are obstructive. I hope that people in this discussion will prove me wrong.
With respect to a list of suitable reviewers, one can act in good faith believing certain individuals will be fair - and yes we learn from our experiences.
I do not know how you define your truncated normal distribution, not reading your paper yet. But by definition, it should be the part within [a,b] in a normal distribution, which, after the normalisation, gives 1 under the curve. We all know a normal distribution is bell-shaped, i.e. it is monotonically decreasing in the left or right side (divided by the peak). With this understanding, I agree with the reviewer, it is not possible to generate a truncated U-shaped distribution from the original normal bell-shaped distribution. Are you talking about an inverse U-shaped distribution?
DeleteI do agree with the comments by others: (1) peer-review is necessary for any serious scientific publication, (2) how to do such review is open for question but the current system is not so bad, (3) theoretical papers are probably more difficult to publish for various reasons (e.g. much less researchers as compared to experimental ones). Remember the famous remark by Albert Einstein: “A theory is something nobody believes, except the person who made it. An experiment is something everyone believes, except the person who made it.” And prepared for difficulties in acceptance/publication of a piece of theoretical work.
The problem with our current research community is that mistakes in experiments are understandable/tolerable, but not many are prepared to accept any mistake, even minor, in theories. Reviewers and editors are those in the best position to change this view by accepting and publishing more theoretical papers. Noting that it is often a theory that can generalise or lift up research efforts in an area or research topic, this would be a good way to stimulate research and avoid repeated research efforts at the same level or just with trivial incremental advance.
I agree with the other respondent: if as a reviewer I read about a U-shaped normal distribution of any kind (including truncated), I would question it.
DeleteIf the response from the author was that this was "well known", I would not be satisfied.
If the author referred me to specific, reputable references that verified (and ideally explained) this to me, I would be satisfied.
I would also be open to compromise if the author was willing to rephrase the text to something like "a U-shaped quasinormal distribution", "a U-shaped pseudonormal distribution", "a U-shaped seminormal distribution", etc., if I could understand the link they were trying to make.
In short: there is no winner in rebutting a "belief" with a "belief" — use logic and cite specific, reputable references.
NOTE: I was not the reviewer in the above case!
—DIV
As we collectively drag mineral processing into this millenium we can hope that there will be a greater proportion of reviewers able to see through mathematical smokescreens and open to sufficiently well presented mathematical content.
ReplyDeleteThe onus is on the author to ensure that all techniques referenced in the paper are clearly explained in principle. The majority of reviewers will grasp the principle of a mathematical technique without knowledge of the details. Knowledge of specialised techniques cannot be assumed if one chooses to publish in a mining industry journal. The alternative is that one has a rather mundane paper for a mathematics journal.
I share Stephen's view point to some degree. I recently submitted a Mathematical paper to MEI for Flotation '11. The paper showed an elegant derivation of the recovery in a flotation cell by the mechanisms of attachment and entrainment using probabilities. The equation corrected a well known equation in flotation modelling that has been in use for around 15 years. It used well know, widely accepted assumptions in flotation. The paper was simply a Mathematical proof of the equation using these assumptions. The reviewers however questioned the lack of experimental data to validate the equation. The beauty of a Mathematical proof is that unlike other scientific theories based on observation, a Mathematical proof requires no experimental data, only the initial assumptions on which it is based. Hopefully my paper will still be accepted.
ReplyDeleteI think you are exaggerating your experience a little here David. I have your paper and the reviews in front of me now. In fact only one of the reviewers questioned the lack of experimental data to validate the equation. The other referee, a member of the Editorial Board, recommended fairly minor revisions. You have been asked to revise the paper and provide rebuttals for any comments that you do not agree with, so all is by no means lost yet! It is my job in the end to balance your comments with those of the referees, who quite often have opposing views.
DeleteI wish only to comment on the place, or otherwise, for (advanced) mathematics in manuscripts, submitted to MEI.
ReplyDeleteFirst, the audience: As I experience MEI, the audience consists of mostly practitioners in Minerals Engineering. That is, engineers of all applicable disciplines, working in the various fields of Minerals Engineering. Mostly, one would expect to find here metallurgical engineers/technologists/technicians.
Second, the expected qualifications of the audience: All engineers must pass a good deal of mathematics at university level - pure mathematics, applied mathematics, and statistics ("for engineers"). Of course, in time, things tend to settle and rust a bit. But, it can be dusted and oiled, as it were, should need be.
Third, appropriate content for this journal and its audience: A tough case of good judgement on the part of the author(s) as much as the editorial staff of MEI and designated reviewers. Applicable novelty is key. Often, engineering novelty involves (advanced) mathematics. We are dealing here with an exact applied science, for most of it. A reviewer worth his/her laptop, would know mathematical smoke and mirrors from true innovation. And, there is nothing wrong with calling in a little help from one's colleagues over there at the maths department.
In conclusion, I do believe MEI has room, and does welcome, appropriate expositions of mathematics, where applicable. Mathematics is the core language of engineering, after all.
JP Barnard, South Africa
Hi
ReplyDeleteFirstly I agree that the comment (truncated normal distribution can be U-shaped) is not immediately obvious.
I also agree that if a reviewer is concerned they should express that viewpoint, and there is onus on me to provide evidence.
If a reviewer was of the firm opinion that the truncated normal distribution is invalid, the author (in this case me) should have been given opportunity to use a more conventional mineralogical distribution (beta distribution). I do not consider rejection of the paper is 'fair'.
Now with respect to the particular example (truncated distribution normal can be U-shaped) I was unable to find a straightforward link (I am also currently not connected to ScienceDirect so a bit limited in accessing papers). I maintain my viewpoint that a statistician would not have considered the comment controversial.
The examples in Wikipedia does not provide sufficient guidance.
Just try: A exp((x-0.5)^2)
in Excel (between 0 and 1).
with A being used to make the integral 1.
This is U-shaped, and satisfies acceptable criteria to be a pdf.
I claim this to be a truncated normal distribution, whereas there could be a counter-claim that even though it has the same form as a truncated normal distribution it is not a truncated normal distribution because it is U-shaped.
I would have been more than happy to call it a truncated Gaussian distribution rather than a truncated normal distribution, but clearly I had not anticipated any controversy in submission.
Further to previous comments:
ReplyDeleteThe error I have made over and over again is not to be aware of when a particular comment I make in a paper could use a phrase that could be 'controversial'.
Therefore the comment by Anomonous 24/5 is totally correct.
Had I used 'pseudo-normal' (or Gaussian as for previous comment) I may have dodged the issue.
But as I said, I had not anticipated controversy.
My point was (returning to the main debate) is that in the peer review process there are many pitfalls, particularly for the inexperienced, and many potentially good papers are rejected.
My personal issue is that I could list numerous 'concepts' I have worked on that are not consistent with established methods. (For example, see how I inadvertently killed off a debate in LinkedIn Geomet. Group by talking about A*b with respect to comminution)
For me personally, I remain very wary of publication, and to some extent this debate has enforced my concerns rather than alleviate them.
It is much easier for me to just develop and apply the models rather than seek 'peer review' approval.
This is a bit of an old thread but I saw this online today it made me think of this.
ReplyDelete"...scientific articles that are less text-heavy and present more equations on each page are not cited by other scientists as often as those that rely more on words to explain their findings"
From "Math is Tough For Scientists Too", on International Science Times, url:
http://www.isciencetimes.com/articles/3296/20120626/math-tough-scientists.htm
I recently received an email from another soccer-parent asking 'was that really you, Stephen?'.
ReplyDeleteWell, yes it was.
But I have to make one more obvious comment... I used the U-shaped truncated normal distribution as an example of how a paper could be easily rejected. But in 29 May I gave a simple explanation of how this could occur (although the phrase 'pseudo-truncated normal distribution' would seem to be the key to getting past some immediately hostile examiners).
Yet not one person responded and said ' Oh I see the distribution can be U-shaped'.
There are a number of plausible explanations for why people are prepared to comment earlier but not on the specific technical issue given later:
1. agree, but couldn't be bothered saying so.
2. aren't actually interested in technical details.
3. still don't understand.
4. don't know how to use Excel for fitting curves.
Now I know there are plenty of people who understand the concept (and remember this is only a small part of the whole paper), but such people generally do not communicate (i.e. category 1).
And this is the ultimate issue, many of the truly competent people are not vocal. Why? Because they are likely to be targeted for criticism.
Hence my earlier comment that journals should be encouraging communication by the truly competent; and this cannot be achieved by allowing their work to be reviewed by those who are 'quick to reject'.