Publication/Paper (NAR 2012)/Reply

• Above: Points and tasks
• Below: Full comments for context

Rebuttal

Reviewer 1

Browsing is bad

First of all the navigation appears to be ill defined. For example browsing for a software tools makes users repeatedly click on tags, that, after several steps and choices may lead to only one or two very generic tools like "CLC Genomic Workbench".

• Tag based navigation is what we have.
• We are working on improving the tags to more accurately reflect the functionality of the tools. Marco and Dan agreed to do this. Could we base "Bioinformatics methods" and "Biological domains" on modified EDAM "Operation" and "Topic" tags? - Andreas

Working on the tags now.--Marcowanger 21:54, 12 September 2011 (PDT)
Andreas, not sure what you mean, may you explain a bit more?, sorry not enough sleep these days.
In the meantime, I try to convert all tags word to start with capital letter.

• We could add other forms of nav. Dan to look how quickly something can be added.

The site is slow

This suboptimal navigation when coupled to very slow load times and makes the aforementioned awkward navigation even more discouraging. There were instances where the page loaded over about 20 seconds! The problem of course is that at the end of suffering through slow load times, often a minute or two, there may be no useful information at all!

• We are sorry that we caused you to be discouraged. Slow server speed was an unfortunately timed technical issue that we have now resolved. Please see the image of average page load times kindly provided by Google webmaster tools. Fixed by Eric. Dan to upload image.

The content is bad

Looking for known and existing tools does not seem to be any better. Even the software tools linked from the front page lead to a overly generic descriptions of the tool with links to the homepage of the tool. Notably some of the information even on these well known tools such as Bowtie or MAQ were occasionally incorrect while being severely lacking

• One of the wonderful things about wikis is that incorrect or insubstantial information, when discovered, can be easily corrected or amended by anyone, with minimal effort. Please provide details of any specific problems, and we will be happy to correct them for you. I am going through pages to update them - Andreas

The same is true on the 'How to' sections, the SNP detection is about two-three paragraphs long and the advice it gives is very basic; seemingly anecdotal.

• I warned you about these sections ;-) Who wants to fix it?- Dan.

working on it (offline) snp is however really not my area of expertise, so any help is appreciated bjoern

Closing comments

Unfortunately I think that even a beginner bioinformaticians would be much better served by reading review papers or asking questions on forums such as Seqanswers Forums than trying to find answers on this site.

• We certainly wouldn't discourage anyone from reading review papers or asking questions on forums such as Seqanswers Forums. SEQwiki was designed to act as a companion database for such resources, providing standardized, queryable data and resources about the tools discussed there.

A resource of this type is a good idea in principle, but it would need to be substantially improved both content wise as well from the point of view of the user experience.

• Thanks for your comments on SEQwiki, which we value and have taken on board. The SEQwiki project is an evolving system. It is one among a few of the first community maintained databases for scientific information. Rather than integrate this effort within Wikipedia, we took the step of closely aligning with a strong community built around an existing forum. We additionally differ from Wikipedia by providing a standard form-based interface to an underlying database of information. Using this model, we have rapidly built a large, detailed database of information about tools for high throughput sequencing. Unlike a static database, user can extend the scope of SEQwiki, adding data on service providers, technologies, formats, and perhaps, in future, performance benchmarks and other cool features.

Reviewer 2

Don't forget the forums!

My main critique of the manuscript is that by focusing mainly on the wiki it doesn't really do justice to the website as a whole. Although Seqanswers has been cited by others this appears to be the first publication about Seqanswers itself. Therefore, the manuscript would benefit from describing both the seqanswers community forum and the Seqanswers wiki. The paper could really use a simple descriptive introductory sentence saying that the resource consists of those two components. This could be done simply by rearranging the last paragraph on page 2.

• Thanks for pointing out this omission. In our enthusiasm for the wiki, we forgot that SEQanswers has not yet been described in the literature, and deserves more attention in the manuscript. We have added a paragraph ... and have rearranged the last paragraph on page 2, as suggested. Eric, please will you write a paragraph on what you feel are the major significant points about SEQanswers? - Dan.

The introduction misses the point

The Introduction would benefit from having less of the generic web 2.0/distributed science benefits (which I assume will be covered by the editorial from Bateman) and more of an emphasis on how the web-based paradigm shift using things like wikis is especially important for the specific area covered by Seqanswers. It would also be nice to see some description of a few examples of how the community has used the forum/wiki. Marco, can you rewrite the introduction along these lines? - Dan.

The discussion misses the point

The discussion of the wiki is a bit too strong on implementation details and weak on content. For example, one useful aspect of Seqanswers wiki pages that is not mentioned is the link back to forum threads. Again, it would benefit from some discussion of an example. I'll have a go at this. - Dan.

Referee: 3

Content is sparse

The tools sections have received the most attention, and are relatively up-to-date. For many tools the only content is a link to the source/home page, and there is in general no edited community comment (on experienced utility).

• Tough to answer.

The potentially useful section, that of workflows and recommendations is extremely thin, and personal, and does not make much reference to the comparisons and workflows that have been published already.

• I guess we can add references at least? - Dan. Yes Björn on it

It would be good to have a keyword-linked melding with the fora etc in seqanswers itself...

• We do have links to the forums (that we forgot to mention in the text), Dan to add this.

The file formats section is brilliant. The extraction and de-complexing of these formats is the bane of many new starts' lives.... however it would be good to have definitions for all of them, and not just the core group's favourites.

• Dan to add plenty more file format descriptions. (We could look at the format section of the EDAM ontology for a curated list of file formats used in biology. - Andreas)

The tags are bad (again)

There are some nice visual search tools, but these rely strongly on the correctness of the tags ascribed to each software page, and do not (to this user) necessarily lead to correct suggestions.

• We have overhauled the tags to check for accuracy... Dan and Marco to do this.

The site is slow (again)

Technically, the server (? perhaps semanticmediawiki itself) is rather slow to respond, annoyingly so when updating, and there are issues with cached pages overwriting the current one.

• This was a server, and not a software issue, which has now been resolved. Please see reply above.

Is the wiki fit for purpose in the long run?

The proposals for the future are interesting. The concept that the wiki could be used for bugreports and support for specified packages is nice, but I would guess that sourceforge/googlecode forums will be more suited to this.

• Perhaps, but having reports for all the tools in one place, with associated discussion is quite nice.

BAMSeek is already using SEQanswers as its main homepage. I still suggest to remove that sentence from the paper since I think the reviewer is right: I (as a developer) would never use a wiki for wish lists or bug reports. I don't think this is SEQwiki's task. --Mmartin 06:51, 13 September 2011 (PDT)

Full comments

On 2 September 2011 20:52,  <nardatabase@gmail.com> wrote:
> 02-Sep-2011
> NAR-02028-DATA-E-2011
> The SEQanswers wiki: A wiki database of tools for high throughput sequencing analysis.
>
> Dear Dr. Bolser
>
> Thank you for giving us the opportunity to consider your manuscript.
>
> The referees have raised substantial criticisms with respect to the database content, the slow response of the server, and the manuscript itself, which are detailed below. We will consider publishing your manuscript only if you can accommodate their suggestions in a revised version or explain satisfactorily why their comments are invalid. The revised version will be subject to another cycle of reviews by the same and/or additional reviewers.
>
> Detailed instructions for submitting your revised manuscript are provided below. Please read these instructions carefully as they differ from the original submission instructions and any error will delay the review of your manuscript. When you submit your revised manuscript, you should provide a concise point-by-point response to the referees' comments. Any text in the manuscript that you change or add in response to referee or Editor comments should be marked in red.
>
> The revised version must be uploaded within 20 days of the date of this letter.
>
> We look forward to receiving your revised manuscript.
>
> Yours sincerely,
>
> Michael Y. Galperin, PhD
> Executive Editor,
> NAR Database Issue
>
>
> **************
> Reviewers' Comments to Author
>
> (Line numbers mentioned in a report may not coincide with the original line numbers.)
>
> Referee: 1
> Comments for the Author
> In their paper titled "The SEQanswers wiki: A wiki database of tools for high throughput sequencing analysis." the authors present a wiki based website that is meant to store  information on a wide variety of bioinformatics tools.
>
> I found the site to be of substantially lower utility than I initially expected. First of all the navigation appears to be ill defined. For example browsing for a software tools makes users repeatedly click on tags, that, after several steps and choices may lead to only one or two very generic tools like "CLC Genomic Workbench". This suboptimal navigation when coupled to very slow load times and makes the aforementioned awkward navigation even more discouraging. There were instances where the page loaded over about 20 seconds! The problem of course is that at the end of suffering through slow load times, often a minute or two, there may be no useful information at all!
>
> Looking for known and existing tools does not seem to be any better. Even the software tools linked from the front page lead to a overly generic descriptions of the tool with links to the homepage of the tool. Notably some of the information even on these well known tools such as Bowtie or MAQ were occasionally incorrect while being severely lacking. The same is true on the 'How to' sections, the SNP detection is about two-three paragraphs long and the advice it gives is very basic; seemingly anecdotal.
>
> Unfortunately I think that even a beginner bioinformaticians would be much better served by reading review papers or asking questions on forums such as Seqanswers Forums  than trying to find answers on this site.
>
> A resource of this type is a good idea in principle, but it would need to be substantially improved both content wise as well from the point of view of the user experience.
>
> Referee: 2
> Comments for the Author
> Seqanswers appears to be a useful resource that has already built a strong user community.  It is clearly suitable for the NAR database issue, and I look forward to watching its further development.  My main critique of the  manuscript is that by focusing mainly on the wiki it doesn't really do justice to the website as a whole.  Although Seqanswers has been cited by others this appears to be the first publication about Seqanswers itself.  Therefore, the manuscript would benefit from describing both the seqanswers community forum and the Seqanswers wiki.  The paper could really use a simple descriptive introductory sentence saying that the resource consists of those two components.  This could be done simply by rearranging the last paragraph on page 2.
>
> The Introduction would benefit from having less of the generic web 2.0/distributed science benefits (which I assume will be covered by the editorial from Bateman) and more of an emphasis on how the web-based paradigm shift using things like wikis is especially important for the specific area covered by Seqanswers.  It would also be nice to see some description of a few examples of how the community has used the forum/wiki.
>
> The discussion of the wiki is a bit too strong on implementation details and weak on content.  For example, one useful aspect of Seqanswers wiki pages that is not mentioned is the link back to forum threads.  Again, it would benefit from some discussion of an example.
>
> Referee: 3
> Comments for the Author
> The seqanswers wiki is an inspired attempt to develop a crowd sourced wiki of information about next generation and other sequencing - mainly on the bioinformatics side but also including a providers hub. It is rather partial, perhaps reflecting its novelty.
>
> The tools sections have received the most attention, and are relatively up-to-date. For many tools the only content is a link to the source/home page, and there is in general no edited community comment (on experienced utility). The potentially useful section, that of workflows and recommendations is extremely thin, and personal, and does not make much reference to the comparisons and workflows that have been published already. It would be good to have a keyword-linked melding with the fora etc in seqanswers itself...
>
> There are some nice visual search tools, but these rely strongly on the correctness of the tags ascribed to each software page, and do not (to this user) necessarily lead to correct suggestions.
>
> The file formats section is brilliant. The extraction and de-complexing of these formats is the bane of many new starts' lives.... however it would be good to have definitions for all of them, and not just the core group's favourites.
>
> Technically, the server (? perhaps semanticmediawiki itself) is rather slow to respond, annoyingly so when updating, and there are issues with cached pages overwriting the current one.
>
> The proposals for the future are interesting. The concept that the wiki could be used for bugreports and support for specified packages is nice, but I would guess that sourceforge/googlecode forums will be more suited to this.
>
> **************