A blog on statistics, methods, and open science. Understanding 20% of statistics will improve 80% of your inferences.

Friday, April 14, 2017

Five reasons blog posts are of higher scientific quality than journal articles



The Dutch toilet cleaner ‘WC-EEND’ (literally: 'Toilet Duck') aired a famous commercial in 1989 that had the slogan ‘We from WC-EEND advise… WC-EEND’. It is now a common saying in The Netherlands whenever someone gives an opinion that is clearly aligned with their self-interest. In this blog, I will examine the hypothesis that blogs are, on average, of higher quality than journal articles. Below, I present 5 arguments in favor of this hypothesis.  [EDIT: I'm an experimental psychologist. Mileage of what you'll read below may vary in other disciplines].

1. Blogs have Open Data, Code, and Materials


When you want to evaluate scientific claims, you need access to the raw data, the code, and the materials. Most journals do not (yet) require authors to make their data publicly available (whenever possible). The worst case example when it comes to data sharing is the American Psychological Association. In the ‘Ethical Principles of Psychologists and Code of Conduct’ of this professional organization that supported torture, point 8.14 says that psychologists only have to share data when asked to by ‘competent professionals’ for the goal to ‘verify claims’, and that these researchers can charge money to compensate any costs that are made when they have to respond to a request for data. Despite empirical proof that most scientists do not share their data when asked, the APA considers this ‘ethical conduct’. It is not. It’s an insult to science. But it’s the standard that many relatively low quality scientific journals, such as the Journal of Experimental Psychology: General, hide behind to practice closed science.

On blogs, the norm is to provide access to the underlying data, code, and materials. For example, here is Hanne Watkins, who uses data she collected to answer some questions about the attitudes of early career researchers and researchers with tenure towards replications. She links to the data and materials, which are all available on the OSF. Most blogs on statistics will link to the underlying code, such as this blog by Will Gervais on whether you should run well-powered studies or many small-powered studies. On average, it seems to me almost all blogs practice open science to a much higher extent than scientific journals.


2. Blogs have Open Peer Review


Scientific journal articles use peer review as quality control. The quality of the peer review process is as high as the quality of the peers that were involved in the review process. The peer review process was as biased as the biases of the peers that were involved in the review process. For most scientific journal articles, I can not see who reviewed a paper, or check the quality, or the presence of bias, because the reviews are not open. Some of the highest quality journals in science, such as PeerJ and Royal Society Open Science, have Open Peer Review, and journals like Frontiers at least specify the names of the reviewers of a publication. Most low quality journals (e.g., Science, Nature) have 100% closed peer review, and we don’t even know the name the handling editor of a publication. It is often impossible to know whether articles were peer reviewed to begin with, and what the quality of the peer review process was.

Some blogs have Open pre-publication Peer Review. If you read the latest DataColada blog post, you can see the two reviews of the post by experts in the field (Tom Stanley and Joe Hilgard) and several other people who shared thoughts before the post went online. On my blog, I sometimes ask people for feedback before I put a blog post online (and these people are thanked in the blog if they provided feedback), but I also have a comment section. This allows people to point out errors and add comments, and you can see how much support or criticism a blog has received. For example, in this blog on why omega squared is a better effect size to use than eta-squared, you can see why Casper Albers disagreed by following a link to a blog post he wrote in response. Overall, the peer review process in blog posts is much more transparent. If you see no comments on a blog post, you have the same information about the quality of the peer review process as you’d have for the average Science article. Sure, you may have subjective priors about the quality of the review process at Science (ranging from ‘you get in if your friend is an editor’ to ‘it’s very rigorous’) but you don’t have any data. But if a blog has comments, at least you can see what peers thought about a blog post, giving you some data, and often very important insights and alternative viewpoints.

3. Blogs have no Eminence Filter


Everyone can say anything they want on a blog, as long as it does not violate laws regarding freedom of speech. It is an egalitarian and democratic medium. This aligns with the norms in science. As Merton (1942) writes: “The acceptance or rejection of claims entering the lists of science is not to depend on the personal or social attributes of their protagonist; his race, nationality, religion, class, and personal qualities are as such irrelevant.” We see even Merton was a child of his times – he of course meant that his *or her* race, etcetera, is irrelevant.

Everyone can write a blog, but not everyone is allowed to publish in a scientific journal. As one example, criticism recently arose about a special section in Perspectives on Psychological Science about ‘eminence’ in which the only contribution from a woman was about gender and eminence. It was then pointed out that this special section only included the perspectives on eminence by old American men, and that there might be an issue with diversity in viewpoints in this outlet.

I was personally not very impressed by the published articles in this special section, probably because the views on how to do science as expressed by this generation of old American men does not align with my views on science. I have nothing against old (or dead) American men in general (Meehl be praised), but I was glad to hear some of the most important voices in my scientific life submitted responses to this special issue. Regrettably, all these responses were rejected. Editors can make those choices, but I am worried about the presence of an Eminence Filter in science, especially one that in this specific case filters out some of the voices that have been most important in shaping me as a scientist. Blogs allows these voices to be heard, which I think is closer to the desired scientific norms discussed by Merton.

4. Blogs have Better Error Correction


In a 2014 article, we published a Table 1 of sample sizes required to design informative studies for different statistical approaches. We stated these are sample sizes per condition, but for 2 columns, these are actually the total sample sizes you need. We corrected this in an erratum. I know this erratum was published, and I would love to link to it, but honest to Meehl, I can not find it. I just spend 15 minutes searching for it in any way I can think of, but there is no link to it on the journal website, and I can’t find it in Google scholar. I don’t see how anyone will become aware of this error when they download our article.

When I make an error in a blog post, I can go in and update it. I am pretty confident that I make approximately as many errors in my published articles as I make in my blog posts, but the latter are much easier to fix, and thus, I would consider my blogs more error-free, and of higher quality. There are some reasons why you can not just update scientific articles (we need a stable scientific record), and there might be arguments for better and more transparent version control of blog posts, but for the consumer, it’s just very convenient that mistakes can easily be fixed in blogs, and that you will always read the best version.

5. Blogs are Open Access (and might be read more).


It’s obvious that blogs are open access. This is a desirable property of high quality science. It makes the content more widely available, and I would not be surprised (but I have no data) that blog posts are *on average* read more than scientific articles because they are more accessible. Getting page views is not, per se, an indication of scientific quality. A video on Pen Pineapple Apple Pen gets close to 8 million views, and we don’t consider that high quality music (I hope). But views are one way to measure how much impact blogs have on what scientists think.

I only have data for page views from my own blog. I’ve made a .csv file with the page views of all my blog posts publicly available (so you can check my claims below about page views of specific blog posts below, cf. point 1 above). There is very little research on the impact of blogs on science. They are not cited a lot (even though you can formally cite them) but they can have clear impact, and it would be interesting to study how big their impact is. I think it would be a fun project to compare the impact of blogs with the impact of scientific articles more formally. Should be a fun thesis project for someone studying scientometrics.

Some blog posts that I wrote get more views than the articles I comment on. One commentary blog post I wrote on a paper which suggested there was ‘A surge of p-values between 0.041 and 0.049 in recent decades’. The paper received 7147 view at the time of writing. My blog post received 11285 views so far. But it is not universally true that my blogs get more pageviews than the articles I comment on. A commentary I wrote on a horribly flawed paper by Gilbert and colleagues in Science, where they misunderstood how confidence intervals work, has only received 12190 hits so far, but the article info of their Science article tells me their article received three times as many views for the abstract, 36334, and also more views for the full text (19124). On the other hand, I do have blog posts that have gotten more views than this specific Science article (e.g., this post on Welch’s t-test which has 38127 hits so far). I guess the main point of these anecdotes is not surprising, but nevertheless worthwhile to point out: Blog are read, sometimes a lot.

Conclusion


I’ve tried to measure blogs and journal articles on some dimensions that, I think, determine their scientific quality. It is my opinion that blogs, on average, score better on some core scientific values, such as open data and code, transparency of the peer review process, egalitarianism, error correction, and open access. It is clear blogs impact the way we think and how science works. For example, Sanjay Srivastava’s pottery barn rule, proposed in a 2012 blog, will be implemented in the journal Royal Society Open Science. This shows blogs can be an important source of scientific communication. If the field agrees with me, we might want to more seriously consider the curation of blogs, to make sure they won’t disappear in the future, and maybe even facilitate assigning DOI’s to blogs, and the citation of blog posts.

Before this turns into a ‘we who write blogs recommend blogs’ post, I want to make clear that there is no intrinsic reason why blogs should have higher scientific quality than journal articles. It’s just that the authors of most blogs I read put some core scientific values into practice to a greater extent than editorial boards at journals. I am not recommending we stop publishing in journals, but I want to challenge the idea that journal publications are the gold standard of scientific output. They fall short on some important dimensions of scientific quality, where they are outperformed by blog posts. Pointing this out might inspire some journals to improve their current standards.

53 comments:

  1. "If the field agrees with me, we might want to more seriously consider the curation of blogs, to make sure they won’t disappear in the future, and maybe even facilitate assigning DOI’s to blogs, and the citation of blog posts."

    Yes!

    Perhaps some sort of collaboration with preprint servers like Psyarxiv could be established. Of course at this point in time you could already just make a pdf-file of your blog and post it on Psyarxiv for instance, but perhaps something more fancy, and easier could be developed.

    I guess/assume Psyarxiv already has some sort of mechanism in place that makes sure the preprints are preserved, and they already make it possible to cite them. The only thing not (yet) in place would be for google scholar to index them so possible citations would actually "count" more concerning things like h-index, etc.

    ReplyDelete
    Replies
    1. "The only thing not (yet) in place would be for google scholar to index them so possible citations would actually "count" more concerning things like h-index, etc."

      Looks like it's happening! This is great news :)

      https://twitter.com/BrianNosek/status/853188665216049152/photo/1

      Delete
    2. Go to figshare. Gives doi and persistency.

      Delete
    3. Crossref (the people behind DOIs for scholarly articles) are working on this. Crossref Event Data is heading into Beta very soon. We're collecting mentions of scholarly articles on social media, blogs etc, via their URLs and DOIs into an open dataset that people can search and build on. More info at https://www.crossref.org/services/event-data and a lot more detail at https://www.eventdata.crossref.org/guide . I'm happy to discuss here or contact jwass@crossref.org

      Delete
  2. Point 4 is interesting -- I thought it was just me, but if you can't find your own correction, then I must wonder if the journals are having some problem server-side. See this meta-analysis, the correction to which, like yours, has gone missing: http://journals.sagepub.com/doi/full/10.1177/0146167213520459

    ReplyDelete
  3. ""If the field agrees with me, we might want to more seriously consider the curation of blogs, to make sure they won’t disappear in the future, and maybe even facilitate assigning DOI’s to blogs, and the citation of blog posts."
    Yes!
    Perhaps some sort of collaboration with preprint servers like Psyarxiv could be established."

    Isn't the winnower (https://thewinnower.com/) for that?

    ReplyDelete
    Replies
    1. I feel like most people don't know about the Winnower, but yes, that's exactly what it allows you to do - getting DOIs for blog posts, which are tracked (at least) by Google Scholar, plus permanent archiving of a snapshot of the post (once you're confident it's truly "done") plus hosted peer review.

      Delete
    2. Hi Richard, thanks - I know about the Winnower (e.g., here's one of my posts on the Winnower https://thewinnower.com/papers/1403-p-curves-are-better-at-effect-size-estimation-than-trim-and-fill-and-michael-jordan-is-better-at-free-throws-than-i-am and there are several others). Yes, it's one way to get blogs a DOI. I just wanted to point out the general point (curating blogs) - The Winnower is indeed one way, but I'm not sure what the best way is, as of yet.

      Delete
  4. I would add that blogs are designed to fit human cognitive capacities. To reach a broad audience, they are written to be broadly comprehensible -- and as it turns out, that's a good thing for science (and philosophy), since it reduces the tendency to hide behind jargon, technical obscurities, and dubious shared subdisciplinary assumptions. The length of a typical substantive blog post (500-1500 words) is also, I think, a good size for human cognition: long enough to have some meat and detail, but short enough that the reader can keep the entire argument in view. These features make blog posts much easier to critique, enabling better evaluation by specialists and non-specialists alike.

    ReplyDelete
  5. Great blog. So much of peer reviews are really "circle of friends" reviews but that's not transparent.

    ReplyDelete
  6. You forgot the sixth argument: blogs are resourceful for the open education. I'd like to manage the "meta-blog" of your's, Andrew Gelman's, Uri Simonsohn's, and etc. This will be the open-accessed textbook of methodology and falsifiable scientific knowledge.

    ReplyDelete
  7. Well... blogs are also laden with useless comments, if not worse.
    Or they are taken hostage by few individuals discussing side issues at length, almost not related with the principal post.
    It take too much time to try to weed out these fields.
    Isn't it the purpose of peer reviewed journals to provide that service?
    The question remaining is : are reviewers doing their job? or do the filter convenient articles from disturbing ones?

    ReplyDelete
    Replies
    1. If a blog is laden with useless comments (as certainly does happen), the reader can easily ignore the trolls and obsessives. And if you're a regular on a blog, you quickly learn who the nutters are.

      In 'it takes too much time,' you seem to have jumped from the comments sections to the blog posts themselves. Peer reviewed journals certainly do not weed out comments - there are none!

      Delete
    2. Is really a good thing, in science, to have the disturbing filtered out in favour of the convenient?

      Delete
    3. Please note that Anonymous comments are not in line with Open Science principle I adhere to - don't expect comments from me when you submit them anonymously.

      Delete
  8. What makes a "blog"? Is it the stylesheet? The interactivity? The social engagement? ..

    Presenting scholarly work in a "blog" is irrelevant IMO. What matters is presenting in a way that's most suitable to the reader and improving the communication. It can be accompanied with interactive components to help the reader understand better. After all, what's the point if one can't communicate/educate or pass on their knowledge to the others?

    I fully support the idea of publishing scholarly work at a "blog", but due to initiatives towards like openness, accessibility, decentralisation, interoperability. Some of which you touch on in your article (above).

    There is an initiative called "Linked Research" https://linkedresearch.org/ ( https://twitter.com/linked_research ) which should interest you. It doesn't mandate precisely how one should publish but whether they can address some challenges like mentioned here http://csarven.ca/linked-research-scholarly-communication#user-stories . Part of that strives for improving human and machine-accessible scholarly knowledge (articles, peer-reviews..).

    As for whether needing DOIs or not, they are a form of centralisation. DOIs do not ensure quality. Anything can be assigned (registered with) a DOI. For DOIs to be useful, they end up being used as HTTP DOIs (via dio.org). That is not better than "http://daniellakens.blogspot.ch/" where this article sits. The difference is that, you in a sense have more "control" over your domain (although in your particular case it is the owner of blogspot.ch) as opposed to the third-party doi.org. At the end of the day, you have more say over what happens under daniellakens.blogspot.ch vs doi.org (in which you practically have zero control). Moreover, to obtain a DOI, one would have to jump through hoops.. instead of simply just publishing work under a webspace that one controls (rents from). Similar arguments hold for ORCID. I've discussed some of this recently in this thread https://groups.google.com/a/force11.org/forum/#!topic/f11discussion/_jE0D4ns_RQ

    ReplyDelete
  9. Daniel, I've put together something in response to a climate conspiracy blog misusing your article. (Feel free to delete this comment as you did my previous, if you want to. I just wanted to let you know about it as a courtesy.)

    http://blog.hotwhopper.com/2017/04/blogs-are-better-delusions-of-grandeur.html

    ReplyDelete
    Replies
    1. Hi Sou, thanks for this blog - I agree with basically everything you say. My blog is indeed not a scientific argument - I have no data - and it is indeed, as you write, meant to make science journals consider improving some core scientific values. Some parts are illustrations of problems (me not being able to find my own correction is just problematic, even when n = 1) but my blog should not be used as an argument by climate skeptics to argue their blog is better than scientific articles. I am not an expert on climate change, but browsing the blog you link to, it did seem to be a good example of the blog posts I read and reference above (note that ALL blog posts I link to in this blog are written by scientists, for scientists).

      P.S. Apologies for deleting your previous comment - I was deleting your double postings, but must have deleted both. If you still have it, feel free to add it again, although I think your blog is probably an excellent reply. I think you see the point of my blog, not as a scientific claim, but as an opinion piece that is aimed at making people think about the way we publish articles, and how we can improve. It's too bad a climate skeptic uses it as an argument, and it's good you try to combat that - but that is not my fight. I'm trying to increase awareness of ways to improve the quality of articles and journals in psychology, but good luck on your end.

      Delete
    2. Thanks, Daniel. I figured you were writing a thought-provoking piece for scientists.

      (Yeah - sorry about the doubling up of my comment, my mistake.)

      Delete
    3. No problem - and to correct my own comment: "it did seem to be a good example" - I actually meant, the climate change blog did NOT seem to be a good example. Many of the blog posts I discuss above could be published in scientific articles (and some end up as publications, both from my own blog, as from other blogs), exactly because they have higher quality standards.

      Delete
  10. "Despite empirical proof that most scientists do not share their data when asked, the APA considers this ‘ethical conduct’. It is not. It’s an insult to science."

    Indeed it means that the paper is not science, but speculation, assertion or rhetoric. Science requires that results can be verified and that research is repeatable. If data, processing code and experimental protocols are not all shared then the results cannot be verified. It is possible that they can be repeated without data, but cost would usually prohibit as in most cases funding would not be forthcoming to repeat research that has already been published, because of a foolish assumption by funders that the original research was almost certainly competently carried out and analysed, so they do not expect new information or a published paper.

    If even data is withheld then it is highly unlikely that the research can be challenged in any way, thus it is not science.

    If I, with a mere B.A. in science, albeit from an excellent university, know this then why do the journal editors not? Why do the researchers, and more importantly their competitors who could gain by shaming them into releasing data, not know this?

    20 years ago I was turned down for a junior job in scientific publishing. I am more confused than ever now as to why, given the incompetence of the majority of people now in senior positions, many of whom must have started their own careers around that time.

    ReplyDelete
  11. Some ideal blog would theoretically be better than a real-live journal. The question is whether a real-live blog is better than a journal.

    I blog and think it has a role, but made the opposite claim as the above post in 2015: No, blog posts cannot replace scientific articles

    "The science the public does not see, neither in the media nor on blogs, is also important for science. We will need a way to disseminate science that also works for the other 99.9% of science. Blogs and "blog review" won't do for this part."

    ReplyDelete
  12. Hi, thanks for the comment. To be clear (and as I say in the conclusion) I am not proposing anything like moving science publications to blogs.

    ReplyDelete
  13. I do not find the majority of these arguments very convincing. That said, I am coming from a different field (medical physics), and I get the impression that these points may have more relevance in fields in which difficulty in replication is more of a problem.

    First I would like to ask: Do these arguments apply to a distinct blogging platform that is used/curated by the scientific community? Or just any blogging platform (blogspot, wordpress, etc.)? If the former, what is the difference from existing platforms like pre-publication arxiv, or ResearchGate? If the latter, there are obvious problems of wide-spreading of popular but inaccurate information, lack of expertise in areas that require it, formation of positive feedback/amplification of misinformation, and overall "muddying of waters". I expect it is the latter, as this article itself is published on blogspot.

    I will address each point separately:

    1) Transparency in data/methods is of course desirable. Again this may be my field, but this is a non-issue. In my experience, all data/methods required for accurate replication or innovation of an experiment is included in the publication. In the few instances that I have requested code/data, it has been shared (admittedly I have not needed to do this often, so maybe I got lucky in those few instances). Looking at the links, evidence provided is only for psychology or medicine, so I believe this problem could exist in these fields and I agree needs to be rectified.

    2) I think most view closed peer review as the lesser of two evils. Yes the issues Daniel brings up definitely exist, but you could apply those same arguments to open peer review. The peer review is still just as biased as the biases of the peers, that does not change if I know who the reviewers are. But now there is an added complication that it is possible for an author to accept illegitimate comments or reject valid comments because they know who they came from. So not only are the reviewers' biases involved, but the author's as well. This is a case where there is no perfect system, so it is a problem of minimizing negatives. Daniel mentions blogs have comments. But arxiv e-mail discussions and ResearchGate comment sections exist, so what would be the difference? Also, side question, by what metrics are Science and Nature "low quality journals"?

    3) This point includes the biggest problem I have with this: "Everyone can say anything they want on a blog". This is a problem, as there is no system in place for fact-checking, and we have all seen the runaway train of misinformation in current media that could occur (scientists are human and are therefore not immune to this). I also disagree with the statement "it is an egalitarian and democratic medium"; Democratic does not mean egalitarian, and the two are often in conflict, especially on open source media such as blogging platforms where popular and easily digestible pieces are amplified in a positive feedback loop (accuracy is secondary), and boring pieces that contain more rigour and thoughtful analysis are suppressed.

    4) Great point. For example, I am always frustrated when reading news and corrections are shoved in an impossible-to-find backwater after the mass majority have already read the main piece and considered it factual (especially if the error correction changes the entire conclusion of the writing!). Any existing corrections should be front and center on any article, which may be easier to do as journals shift to online platforms. However, the ability to update old articles creates new problems with dishonest authors modifying previous research to correspond to new data. Again, there is no realistic system to rigorously moderate this.

    ReplyDelete
    Replies
    1. 5) Again I agree with this, blogs are definitely more accessible, and I do agree that science should be more accessible. However I do see an advantage of having a distinction of easily-accessible "easy reading" format of blogs where I take in the information, but it is taken with a grain of salt, as (like Daniel says) "everyone can say anything they want on a blog". If I am looking for quality assurance, the absence of any system (however flawed) does not increase my expected level of accuracy. Proof of a flawed system does, however, decrease my expected level of accuracy.

      Fundamentally, Daniel is an advocate of "open data and code, transparency of the peer review process, egalitarianism, error correction, and open access." With the exception of the stated misgivings of the second, I (and most) would agree with these. However, I see no reason why free-form blogging should be the solution instead of the existing platforms that are used and curated by the scientific community in a transparent manner, such as ResearchGate (https://www.researchgate.net/about).

      Delete
    2. Hi Cam, as you can see, I'm a psychologist, and if the situation is better in medical physics, good for you! If 1 is a non-issue in your field, even better, but as I explain, it's a big problem in my field. Ad 2 - open peer review is not just knowing *who* reviewed, but having access to the review. Again, arxiv and researchgate (really - researchgate? do people use that?) might have this solved, but they are more similar to blogs with respect to self-publishing than a journal, no? Your major (only?) issue seems to be with 3. If you think journals do a great job gatekeeping, that's great, and I think they often do a pretty good job. But not always, and I favor a review system where there is only an eye for quality (e.g., as Frontiers and PeerJ have) but not for novelty, eminence, etc. That way, you have the quality check, but not the eminence filter.
      See - I think we are pretty close to agreeing here, right?

      Delete
    3. Do you really not have access to the review? What is the point of the review process then? When I submit a paper I receive back an exhaustive list from each (anonymous) reviewer of comments, clarifications, and suggestions that must be addressed. If you do not have access to that then I fail to see the point, and I definitely understand the feeling of frustration with the process.

      And yes, I am definitely an advocate of researchgate. It is presently a small community, but everything starts somewhere. I also believe it is a perfect compromise between our two positions, as there is open discussion about projects in progress and sharing of methods and ideas. I also like following projects that are of interest and get real-time updates on current research done and in-progress (I realize this sounds like a researchgate ad...I should state I receive no compensation for this post :P). But yes, it seems like there is less disagreement than it originally appeared.

      Thanks!
      -Cam

      Delete
  14. Excellent and well thought out article. Thank you. I have been looking for an alternative to journals for decades. There has to be some form of peer review and open review through comments seems to be gaining integrity. The anti-troll community is becoming recognizable. The problem of publishing scientific research lies with accountability; without accountability there is no integrity. Journal editors across the board are not accountable to anyone, let alone being transparent. I very much like your idea of a blog coupled with an open archive. ViXra comes to mind. I am sure there are others. It would be remarkably easy for an accademic institution to give your suggestions a go.

    ReplyDelete
    Replies
    1. One alternative to the monopolistic journals would be grassroots scientific publishing. Single scientists or groups should be able to start a journal relatively easily nowadays. As long as this does not have a reputation and no one sends in manuscripts, these journals can start open review of existing articles and Arxiv manuscripts.

      Could be seen as a mixture between a blog and a journal. The Wordpress blogging software would be a good start to get this working.

      Delete
  15. Hi - I'm writing a thesis (ha, eventually) at Cranfield on epistemology that includes an academic section, and this blog covers some excellent points that have surfaced in bits of academia but don't seem to be resolved. To be a bit rude, it's not so much that blogs are good, but that journal articles are (often) so bad. Published open data (and code), open review, wide review, clear language, etc are all things that 'should' be part of everyday academic discourse but often (usually?) aren't.

    This in turn leads to a community cognitive dissonance that non-academics (such as climage skeptics above) can poke at: that academics claim scientific certainty from expertise and the gold standard of peer review, but then academics (quite rightly) point out how peer review is really not very good. The result seems to be a bolstering of peer review, rather than attacking the problem at source. I had a senior lecturer say, when I gave a list of academic-reported problems with peer review, that I was undermining the whole way that science is done. Which is worrying; we should be looking to improve the quality of scientific output, not protect a tradition.

    In any case the main core problem is that we don't have decent audit, and without audit we can't actually assure quality. Whether blog or journal article, there's really no way to tell if the source data was collected, made up down the pub, generated by code, or borrowed from some other dataset. Until we can fix this, we have a big hole in scientific discourse that is only poorly plugged by replication.

    ReplyDelete
    Replies
    1. Hi Martin, you seem to be hitting the nail on the head! Do send me a copy of your thesis when it's done!

      Delete
  16. Hi, a typo: "7147 view" should be "7147 views". Point 4 in action!

    ReplyDelete
  17. In a reply above you wrote "Hi Cam, as you can see, I'm a psychologist, "
    To my mind, this is the single most import sentence on this web page.
    It sets the context for this entire discussion.

    As a scientific editor (which I sort of was once, though not in a journal), I would insist on having this in the first paragraph, and repeated a few times where appropriate throughout the text. E.g. At the start of the 3rd paragraph, something like "On psychology blogs, ..."

    As you say, a blog can be read by a wider audience. So imagine a non-psychologist scientist reading your blog.
    - You describe how journals publish, archive, and vet. These vary by field. A lot. They even vary a lot between journals in a single field.
    - You describe how blogs publish their data, their overall approach, and how they serve as a basis for discussion. Again, this is all extremely depend on the field. Experimental psychology is undergoing a revolution, where the old approach of using statistics as an excuse for publishing is being challenged by the new-fangled view that statistics are a tool for distinguishing what we know from what we don't. So you have a huge generational gap with the old timers controlling journals and such, and the young crowd airing their views in more modern platforms, such as blogs. That situation is unique to psychology, and has little to do with the intrinsic quality of blogs vs. journal articles. Alas, the blog does not provide this context (BTW: fact-checking myself, I now found it in the sidebar. Maybe I should read those first :-)
    - You mention the APA and JEPG, whose roles are clear to a psychologist. But you do not indicate whether they are a dominant or negligible force in American psychology (or somewhere in between).
    - Nature and Science publish articles in many fields, with criteria that differ strongly between fields so that they achieve the mix of articles they desire. At first, it reads as though your describing their practices in general. But on a second read, I think you're only describing their practices for the field of psychology.

    ReplyDelete
    Replies
    1. Hi Amnon, I've added an edit ([EDIT: I'm an experimental psychologist. Mileage of what you'll read below may vary in other disciplines].) Thanks for suggesting it. This post is quite popular, with over 12.000 hits in 2 days, many from outside my normal readership. Normally, I'd assume most readers know my blog, but this is attracting a lot of new readers, so adding this context indeed seems like a very good idea. Thanks.

      Delete
  18. We're seeing a huge amount of non-traditional scholarly activity (if you'll pardon the dry phrase) happening in blogs. Alternative metrics aka altmetrics (as distinct from altmetric.com) have been taking in to consideration the scholarly activity that happens around traditional publishing for a while. Crossref, the organisation who brought you DOIs for scholarly publications, thought that it would be a good idea to help collect this kind of data as a counterpoint to traditional publishing and citations. We're building Crossref Event Data, which is a free (libre, gratis) service for collecting mentions of articles on blogs and social media, so that it can be used by the community in all kinds of ways. Discoverability, recommendations, and yes, maybe more metrics. How you use it is up to you.

    Your article raises a good point about blogs as primary methods of publishing rather than, for example, as a venue for the discussion of traditionally published articles. Establishing an open 'citation'-graph-like-dataset of blogs is a good first step toward that.

    We're heading into Beta soon, and you can read more about it https://www.crossref.org/services/event-data . The User guide is a work in progress, but might answer any questions you have: https://www.eventdata.crossref.org/guide . You can also contact me at jwass@crossref.org if you have any questions.

    ReplyDelete
    Replies
    1. Hi Joe, I think many journal articles do a good job, but I also think some high quality blogs outperform many lower quality journals - especially on core scientific values as I specify. It would be great if these contributions can get some more credit in the current academic system. They won't replace journal articles, but they can definitely complement it.

      Delete
  19. Someone needs to exhaustively list every argument against this and show how many of them are actually authoritarian. I don't know why so many people miss this...

    ReplyDelete
  20. Very interesting topic!

    Is there scope to separate experimentation from analysis? Currently these are activities siloed together per-study.

    What if (say) a pharmaceutical company could ONLY do research by publically doing a data study, or a series of them, which all release their raw data. Then, separately, analysis is performed on some or all of said data. Could that reduce the insidious practice of only publishing results that agree with what said company wanted to show in the first place?

    ReplyDelete
    Replies
    1. Robert, are you aware of the Registered Reports in psychology?

      Delete
  21. I think that some of this is a little bit overly simplistic.

    The claim that blogs have open data and materials is only true if you get to select which blogs count and which do not. Blogs as a medium include both this web site and any other blog set up by the Church of Scientology, climate change deniers or other groups. The diffenece between genuine scientific blogs and opinion may seem stark, but it may not always be that way.

    I also think that the claim that their is no eminance filter on the Internet is a bit misleading - PageRank and the ability to drive reader traffic to a website determine eminance. Its just a different set of eminance criteria. They are softer in that anything anyone publishes could be read, but the reality is that nobody reads anything that doesn't make it into their newsfeed or the first page of a google search result.

    The error correction point is interesting, but this creates complications for citing blogs as supporting evidence shifts beneath your feet. I think this only works if you also adopt some way of recording version history.

    ReplyDelete
  22. Hi Daniel,
    thank you so much for this article! In German speaking humanities, there are only few scientific online-only journals - peer reviewed or not -, but lots of blogs have emerged, since in 2012 de.hypotheses.org (its French 'mother', hypotheses.org, is a little bit older) was launched as a platform for academic blogs (in humanities and social sciences) in German speaking countries. Some of them provide citation suggestions for their scientific articles (like us), and since https://www.rechercheisidore.fr/ harvests them, handle identifiers are assigned to the blogposts there - and I guess, it's only a question of months until persistens identifiers will be assigned to blog articles directly (we are negotiating with DARIAH-DE). Long-term preservation remains mostly a desideratum, yet, but the German National Library - who decided to assign ISSN to academic blogs in 2013 - does already archive some blogs - although only as complete images, not as single blog posts. Cf., for instance, http://d-nb.info/1052535798

    In short: I agree with you, that the advantages of blogs should inspire some scientific journals to improve their standards, but academic blogs will become an even greater challenge for them in the future.

    ReplyDelete
  23. Hi Daniel,

    This (your) article was published at Principia-Scientific.org. (PSI) and I would be surprised if it creates a single comment; unless it is mine. But I've been wrong before.

    I agree totally with what you wrote. Especially with the critical importance of data. You wrote: "I'm an experimental psychologist." And you focused more on the fact that as a scientist you were a psychologist than the fact that as a scientist you were an experimentalist. I expect that the community of psychologists have theoretical psychologists also.

    The data of your science has to be analyzed statistically because your experiments cannot be rigidly controlled like many of the observations of the physical sciences can. Hence, you tend scholarly discount the significance of your anecdotal experiences. However, the physical scientist, Albert Einstein, is said to have stated: "The only source of knowledge is experience."

    It is unfortunate that you are not really familiar with 'climate'. For I, an experimentalist, consider that none of the parties involved in this well-known controversy are willing to consider 'all' the data which is available to anyone with access to the internet. They would rather argue with words instead of calling attention to several decades of quantitative data (some recorded every minute twenty-four hours a day seven days a week) that would refute many an argument.

    Yes, on your blog no one can prevent you from reporting your data and its analysis and what you conclude from it. And if you want respect you must make a scholarly effort that what you write warrants the respect of the peers you respect.

    Have a good day, Jerry L Krause

    ReplyDelete
    Replies
    1. PSI is a good example for this discussion. These are people who deny the greenhouse effect exists. Consequently they are not able to get their nonsense published in scientific journals and started their own PSI journal/blog.

      In the end the difference between a blog and the scientific literature is one of credibility. The initial credibility from peer review helps fringe researchers by motivating more scientists to invest time in their work.

      If you are not satisfied with the standards of the current psychology journals, it may be an idea to start your own with a group of colleagues. It should be easier to start a scientific journal. Open Access should help reduce the monopoly power of the publishers and I hope the above mentioned idea of grassroots scientific publishing can help in this regard.

      Delete
  24. Hi Victor and Daniel,

    Victor, thank you for your comments about PSI. What you did not say about PSI is it became a site where a broad spectrum of people, scientist or non-scientist can go and find late breaking news about all kinds of science. If you know of a better website (blogsite) that does this, please tell me.

    I consider a fundamental problem of science is specialization so that one group of scientists have little interest in what another group of scientists is learning and not learning. But even a greater problem is that I find (no Daniel, I have not done a statistical study) most current scientists, if they read English, have not even read significant portions of Crew and de Salvio’s translation of Galileo’s classic book or significant portions of Motte’s translation of Newton’s classic book.

    A scientist needs to be curious and I wonder if either of you know what language Crew and de Salvio translated to English. Or, which language Motte translated to English. I consider sometime current ‘scientists’ could profitably ponder is: Why did Galileo ask that his book be published in the common language of the common people of Italy while Newton later published in Latin, the intellectual language at the time both books were published.

    The value of blogs is people with a great variety of experience can meet each other, as we three have, and begin to have a real dialogue instead of the one which Galileo had to invent.

    As an illustration of a summation of what I would like you both to consider, go to: http://principia-scientific.org/ddt-case-study-scientific-fraud/

    Have a good day, Jerry

    ReplyDelete
  25. Pathetic that the author should need to comment on the generic use of "his" pronoun in the language of 1942. I can guarantee that if, by some incredibly unlikely statistical outcome, this brilliant blog post is read by anyone 75 years from now, those readers will note some perceived anachronism that dates the author to some less enlightened period -- as if anyone can avoid adopting and expressing themselves in the common language of their day, with whatever political/social/moral assumptions then shared.

    Criticizing the manners past is gutless. The question we all have to ask every day if what are WE doing NOW that is accepted today that the future will justifiably condemn.

    ReplyDelete
  26. Funny you did not cite the earlier blog post on this by Titus Brown: http://ivory.idyll.org/blog/2017-top-ten-reasons-blog-posts.html that partly overlaps but also has some additional points.

    Jeroen

    ReplyDelete
  27. Its sad that I had to use Google and then PubMed to find the correction to your paper rather than a link on the journal site:

    http://journals.sagepub.com/doi/abs/10.1177/1745691616640268?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub%3dpubmed

    ReplyDelete
  28. I appreciate the effort - but if you hadn't known to look for it, you'd never found it, right?

    ReplyDelete
  29. Yup, its really annoying that these are not included by default in the journal's website.

    ReplyDelete
  30. A very interesting angle. I immediately wonder if some of the up-sides you've listed from blogs might also be down-sides (eg, there is a trade-off involved on many of these points). Blogs are correctable, but are also editable, which could be a disaster for the wider scientific record. I accept that journal articles are editable in theory as well, but in practice, blogs come with an expectation that the content could be edited, journal articles come with the expectation they will remain in the original form. Additionally who's responsibility is it to preserve the archive of a blog? The blog platform? The author? The Wayback Machine? Journal articles are preserved by the publisher.

    Beyond the importance of the wider scientific record, the loss of the eminence filter - while eliminating any kind of institutional bias and democratizing science - also might have negative consequences. If I have to identify the important work outside my field, I can use the eminence filter provided by journal reputations as a proxy. I have a starting point for what is considered the current state of the field. What is my starting point in a world of blogs? Whichever blog has been upvoted the most on Reddit? The first Google search results? What if in the comments, DrBob59 insists this is the best paper on the subject representing the latest in scientific discourse, and ProfJen17 denounces the work and all its supporters as pseudoscientific charlatans? The systemic curation achieved by the journal system may not be democratic, but in a sea of noise it is useful. Perhaps I should only dive into science outside of my field if I am prepared to fully understand it enough to make up my own mind. This is hardly helpful for democratization of science in a world with so many complicated and disparate disciplines...

    ReplyDelete
  31. I don't what your metric for scientific quality is. If it is a single figure of merit metric, you can rank order science blog posts and science journals. I assert that there will be a range, mostly overlapping, range of quality. I opine that the worst blog is worse than the worst journal and that the best journal is better than the best blog, but not by much in either case.

    ReplyDelete