The Development of an On-line Submission and Peer Review System

Transcription

1 The Development of an On-line Submission and Peer Review System Paul Pavey, Steve Probets, David Brailsford. Electronic Publishing Research Group School of Computer Science and IT, The University of Nottingham, Jubilee Campus, Nottingham, England. 1 Introduction Online submission and peer review is emerging as the next step forward for many journal publishers in an ever increasing drive to take advantage of technological improvements in transferring data electronically over the internet. The Electronic Submission and PEer REview (ESPERE) project was initiated in 1996 as an electronic Libraries (elib) initiative of the Higher Education Funding Council for England (HEFCE). Subsequently the project continued as a self-funding group composed of a consortium of learned society and commercial journal publishers intent on utilising the changes in technology to improve the services they provide to their authors as well as cutting their costs and increasing efficiencies. Traditionally the submission and peer review process has been a paper-based system with authors submitting to a journal using the postal service. Referees are then assigned, by a variety of methods, and sent copies of the manuscript to review and report upon. Technological advances have allowed other methods of transferring the information between parties such as facsimile and , but although these methods speed up the process they suffer drawbacks in quality (facsimile) and size/platform limitations ( ). More importantly they do not address how the information can be managed effectively. The initial aim of the project was: to achieve an entry level of article submission by transfer of a file, which should include all figures and tables applicable to the article. Articles should be acceptable from any institution for peer review. It should also be possible to print out the article for offline reading or storage at any stage of the process. 1 However the evolution and increased usage of the Internet has allowed fast and reliable transmission of information anywhere in the world. With an increasing array of scripting languages it is now possible to transmit and also manage information effectively. These developments of the World Wide Web allowed the ESPERE project to consider the Internet both for electronic submission and the collaborative peer review process. Currently Electronic Publishing Solutions, [email protected]

2 2 Evaluation of current working practices and implementation of an experimental system Because online submissions would initially only account for a proportion of all submissions it was essential that the methodology of the electronic system could be incorporated into existing working practices. There would have to be a certain degree of flexibility to account for the different approaches of the parties involved in the process. 2.1 Existing processes Existing processes differ between journals, although all have a common theme, the development of the system would have to consider the implications of the differences. Initial evaluations within the consortium drew the following conclusions: Security was of paramount importance, the submissions had to be securely accessible by the authors, journal administrators and referees. In particular the review of the submission by the referee should be anonymous. Files should be accessible from any Internet connection and available to other specified parties as required The system should be straightforward for authors to use. The publisher should be kept adequately informed at key points of the process. 2.2 Evaluation of authors and preferred formats Any cross-platform system would have to consider the issue of file formats. The authors original file could have been produced from a wide variety of software packages and potentially different platforms. Without a cross-platform format all parties reading the file would require the platform specific version of the software used to produce the file. The consortium had a large proportion of journals from the biomedical community and, a consequence, the submissions commonly included text, graphics and tables. It was necessary to determine how these could be made easily available. Therefore a preliminary questionnaire was sent to a random selection of authors to gain an insight into whether they would be willing to submit electronically and to gauge their familiarity with file formats.

3 Out of 120 authors the following results were compiled 2 : Questionnaire survey of authors contributing to The Royal Society journal Proceedings: Biological Sciences, January 1999 (responses=76) Question % All figures available electronically 94 Familiar with Adobe Acrobat PDF 62 Capable of producing a PDF file 56 Capable of producing a Microsoft Word file and Graphics 88 Capable of producing HTML file 55 Would Submit 36 Quite Likely 46 Not Very Likely 3 A key result was the familiarity with the Adobe Portable Document Format (PDF) 3 ; in addition PDF files are truly cross-platform and utilise compression techniques to reduce the overall size of the create file. The results of the formats evaluation led to the adoption of Adobe Acrobat PDF as the submission format of choice. This was further encouraged by the offer of free Adobe Acrobat software (Acrobat version 3 licences generously donated by Adobe Systems Inc.) to authors willing to submit but unable to create a PDF file. With the full Acrobat software PDF files can be generated from any application that has a `Print' option via the PDFwriter pseudo printer-driver. 2.3 Implementation of an experimental collaborative system During this critical evaluation phase the project utilised a piece of collaborative software called Basic Support for Cooperative Work (BSCW) 4 produced by a European project based at the Federal Institute of Technology (GMD FIT). The software provides shared areas or workspaces where documents can be transferred and manipulated over the Internet. BSCW was developed predominantly as an aid to collaboration within, or between, projects 5. Because of the differing requirements of the submission and peer review process the software was modified to suit this new application. Although these modifications were necessary, the software did provide the key attributes essential for publishers, referees and authors: Files could be uploaded and managed through any web browser from any Internet connection. All submitting/managing parties required a username and password to access the system, so security of files was maintained. Additionally access rights to each workspace could be configured. Referees could easily access relevant files. Journal Publishers received notification of activities within the workspaces.

4 Although the intended user community for which BSCW was designed would certainly have had the time and inclination to learn how to use the system, this would not be true of authors who would be using the software purely as a mechanism to submit a manuscript. It was essential that authors were guided through the process and that it was kept as simple as possible. To accomplish this the entire system, termed esprint, automated many of the BSCW tasks. To submit a paper an author was required to: Fill in an electronic submission request form, the details from this were ed to the journal publisher. This action also created an individual workspace for the author and ed registration instructions. Register with the system by choosing a username and password as requested in the e- mailed instructions. Log on to the BSCW software. Upload a file using the software interface. This completed the required actions by the author. To send subsequent submissions or revisions the process would be repeated, with the exception of registration. In order to meet the requirements of the publishers, the system had to support certain tasks. The journal publisher received a daily event log as an so that any new submissions could be administered. A number of administrative tasks had to be performed upon the submission: The file was copied to a secure workspace that only the publisher could access, this prevented authors from submitting revised versions until requested to do so. The access rights were changed to anonymous - this feature of BSCW allows a file to be viewed, without registering with the system, by following an encrypted universal resource locator (URL). Referees were selected via the journals usual procedures. The encrypted URL was ed to the selected referees. Some of these tasks could have been automated, but were performed manually, this was due to the existing BSCW code-base and interface, which would have made automating the tasks difficult. Upon receipt of an invitation to review, which included the encrypted URL of the file, the referee had to: Copy and paste the URL into a browser (or in some software simply followed the link) to display the file. Complete an online referee reporting form. 2.4 Online submission trial

5 One of the consortium members, The Royal Society, having received an encouraging response to the author evaluation, expressed interest in trying out the system with their Proceedings: Biological Sciences journal. This journal was selected because it s fast turnaround: Proceedings: Biological Sciences specialises in the rapid delivery of the latest research to the scientific community, normally within three months of acceptance. 6 Initially, authors submitting in the traditional manner were asked if they would also be willing to submit electronically, in parallel. In all 23 authors submitted electronically. After gaining initial reactions from authors, referees and the journal administrative staff the option of submitting electronically using the esprint system was adopted by the journal on a more permanent basis. An additional 60 papers were then submitted during this period. 2.5 Summary of important findings The initial trial and subsequent longer term use of the esprint system provided comprehensive feedback from all of the parties concerned. Questionnaires and interviews were used to assess the reactions from authors, referees and journal editorial staff to this initial system. The feedback was overwhelmingly positive but the key issues highlighted were: Authors perceptions The registration process was cumbersome and confusing, the s generated by the BSCW system unclear. Registration itself was a two stage process with the author being given only one opportunity to follow an encrypted registration URL. Even so, only a tiny minority needed assistance with the registration process. The first impressions of the secure workspaces were quite daunting as the BSCW system had a wide range of extraneous icons for operations unrelated to the submission of a paper. This had the compounding effect of delivering an interface to the author that was not particularly intuitive. Having read the supporting help files all authors managed to submit a file to their secure workspace. The concept of a secure personal area for authors to submit to was seen as a positive attribute. Authors could access their submission within the workspace to verify it had been submitted without error. The consensus from the authors was that the process was conceptually correct but was let down by the unfriendly BSCW interface and mechanisms. Referees perceptions All of the referees contacted were willing to review electronically and accept PDF files.

6 The use of a single encrypted URL to view the file was extremely popular as there was no requirement to register with a system and the file could be viewed from any Internet connection. The quality of the PDF files were accepted as considerably better quality than a photocopied manuscript. The referees consensus was that this was an excellent way to review submissions and the vast majority of referees would be willing to review in this manner again. Editorial staff perceptions After training, the administrative staff felt confident in handling the electronic submissions. Although the processes required to manage the files were quite simple, the tasks were laborious and time consuming. The receipt of an authors registration information was not necessarily an indication of an authors immediate intent to submit, the submissions were only reported on a daily basis as a report. The staff were particularly impressed by the review system. The esprint system had proved itself adequate for the electronic submission and peer review process. There were some limitations within the BSCW software but this was to be expected as the software was designed for collaborative cooperation between projects and not for applications where security and integrity of data and participants was paramount. An important issue with the BSCW software was the absence of an accessible underlying database. The only practical way to access information was through the front-end interface, which, due to the complexity of the system and number of icons, was often slow in operation. There was no facility to query the database or to generate usable reports. With no facility to access the database longer term development of the system would be severely limited. 3 Development of a modular, configurable online system The trials of the esprint system had highlighted the enthusiasm of both authors and referees for submitting and reviewing electronically. It was decided within the ESPERE consortium that enough information had been gathered to develop a piece of software for the group specifically for the electronic submission and peer review process. The conclusions drawn from the trials of the esprint submission system formed the basis of the requirements for the new system. 3.1 New system requirements The following were considered to be the chief attributes of the new system:

7 Simple registration to a secure system. File submission to the system should be simple and intuitive, even to a first time user. A status facility for authors to check the progress of their submission. Easy access to files through an encoded URL. Configurable reporting to the editorial office. An administration interface with easy-to-use file management utilities. An easily accessible interface to an underlying database that can be used to generate reports, if required. 3.2 Technical considerations The consortium decided to use Microsoft Windows NT Server as the base platform for the software. It was expected that one or more of the participating publishers would eventually want to host the software, after the trial period. To this end the capital investment and expertise required to manage an NT Server would be considerably less, and a lot more feasible for smaller publishers, than, say, a UNIX based system. Microsoft Access was chosen as the initial database; it would be accessed over the Internet by an Active Server Page (ASP) interface. The use of ASP pages would not preclude a change in the database as they can interface with any modern database supporting the Open Database Connectivity (ODBC) standard. The Microsoft Internet Information Server (IIS) was utilised as server software as this also incorporates a Simple Mail Transfer Protocol (SMTP) mail server for communications from the system. The system can be described from two viewpoints the submitting author and the editorial administrative staff. 3.3 Development of the author interface The over riding consideration in the development of the author interface was that of simplicity. The style of the system directs the author down a chosen option by clearly stipulating the requirement for each action. Registration and security A simple registration procedure was an important requirement for the system. Most password protected systems use a combination of username and password to log on. To keep the system as simple as possible it was decided to use the author s address as a username that would be easily remembered. To register, authors simply enter their address into a registration form; the address is then stored within a passwords table in the database. A password is allocated and ed to the author.

8 When the password is received the author can log on to the system. If authors forget their password they can request a reminder from the system which is ed to them at their registered address. To maintain security when the author logs on to the system, a session cookie (an encrypted code) is set on the author s local machine. For every request to the server this identification is checked in order to authenticate the user at every stage and to prevent unauthorised access to the system. When the author has finished using the system the session cookie is deleted. File submission Once the registration process is complete the author is presented with a list of options (Fig 1). Fig 1 ESPERE System Author Main Menu New Submission presents the author with a Submitting Author Details form requesting author-specific information such as contact information, this is then stored on the database. The author is then presented with a File Upload form where the paper information is requested. After entering the particulars of the paper authors are invited to

9 browse their local file system to select the file they wish to send. Upon receipt of the file by the web server the paper information is stored in the database and an encoded folder is created on the web server into which the submission is uploaded. In addition the author is ed a message confirming that the submission has been successful. The editorial office is also ed with the submitting author data, along with the file data and the encrypted URL of the submitted file. Because the submitting author information is stored within the database for any subsequent submissions a pre-populated Submitting Author Details form is presented to the author so that it can be amended if required. Any change in submitting author details is ed to the editorial office keeping the publisher fully up to date. Viewing submissions To perpetuate the concept of a secure authors area, authors can view the files that they have uploaded. On selecting View Submitted Titles authors are presented with a list of their submissions along with the manuscript number (if assigned) and the submission date. The submission titles are hyperlinked to the location of the file on the server so that following the link opens the file. This provides useful reassurance that the file has been uploaded successfully. Additional author options are available. Authors options It was intended that the system should provide authors with an incentive to submit electronically; to this end a key requirement was to allow authors to check on the status of their submissions. The Authors Options facility allows authors to select one of their submissions and perform a number of tasks on that submission. The status of the submission can be displayed, but at present this is limited to manuscript number assigned, referees assigned and decision sent (these criteria could be expanded to provide more detailed feedback information). Additional files or revisions linked to the main submission can be uploaded. A full submission history can be viewed detailing the main submission and any subsidiary files or revisions, all with hyperlinks to the file itself. Other utilities To complete the development of the author interface a number of utilities were developed to allow the user to change password, change address, display the editorial office contact information and, to maintain security, a log off option for ending the session. The peer review process is closely linked to the status of the paper and incorporates locking, to prevent submission of revised files while the original submission is with referees. 3.4 Development of the administration interface The administration interface was developed in the same style as the author interface. The same security procedures were applied and although the administrative staff had no

10 requirement to register with the system they were assigned user names and passwords. The administrative tasks were grouped into three areas, assignation of the manuscript number, management of the submitted files and management of the registered users, enabling editorial staff to keep up to date as new submissions are made (Fig. 2). Fig 2 ESPERE System Administration Main Menu Assign manuscript number The initially submitted file has no manuscript number assigned; this information is generated by the publishers manuscript tracking database and, until this is assigned, the

11 submission is classed as pending. The Assign Manuscript Number utility lists any submissions that have not had a manuscript number assigned to them. As manuscript numbers are assigned to the submissions they are removed from the pending list. It is a simple task to check the list and make sure all manuscripts have been assigned a number. Submission administration This provides the administrator with a search facility for finding a submission (or group of submissions) using a number of search criteria. When the specific submission is located the administrator is presented with a list of options. The first option allows the administrator to amend the status of the paper (the status incorporates locking to prevent the author sending a revised paper while the original submission is with referees). It is these amendments that enable the author to check the progress of the submission. The other options allow amendment of the title, a listing of the full submission history, showing all files submitted, and file management. If an author mistakenly sends an incorrect submission or additional file the administrator can delete where appropriate. User administration This allows the administrator to select an author and amend the corresponding author contact information. It also allows the administrator to the author from the system. Electronic Review Referees are selected via the usual procedure for each journal, usually either by the editorial office or through an editorial board. The referees are ed the encrypted URL of the file (or files) and can access the file by following the link within the or by copying and pasting the URL into a web browser. Depending on the policy of the journal, referees may report electronically (see section 4). 4 Areas of possible future development There are a number of areas of possible future development of the system. At present most publishers have manuscript tracking databases to coordinate the flow of submissions from point of submission through review and on to print. The information the publisher receives from the system is in format. In the short term this information could be imported directly into a quarantine area of the publishers manuscript tracking database. In the longer term the system could be developed to entirely supercede existing tracking databases. The concept of having a Web resource of secure files, accessed through a secure system or by encrypted URL, is useful in many other applications where files need to be transferred over any distance. It is particularly applicable to the transfer of files from the

12 author during initial submission, at the point of acceptance of the article and during the subsequent copy-editing process. Online referee reporting could be further developed so that the author can receive some of the (screened) review comments within the status facility. This would continue to improve the service to authors. 5 Conclusions and latest developments The evaluation of authors and current working practices coupled with the trials of the esprint system have shown the willingness of authors and referees to adopt electronic submission and peer review. The enthusiasm of authors was such that they were prepared to persevere with a groupware system not specifically designed for the online submission process. An important step in the development of an electronic system has been the implementation of a completely new user interface, based on experience from the BSCW groupware prototype, coupled with the adoption of Adobe Acrobat PDF files as a submission format. PDF has allowed authors to incorporate all of their files into a single, cross-platform compatible, compressed format. The lessons learnt from the esprint trials have allowed the project consortium to move forward and develop it s own software. The key requirements of the publisher, the authors and the referees have been implemented in the new system in a simple intuitive manner. The system provides an incentive for authors to submit as they can view their submissions and check the progress of the submission throughout the entire process. The modularity of the system will allow additional processes to be incorporated as a publisher adopts electronic workflow. Perhaps more importantly the technical considerations in the system implementation will give publishers the option of hosting the software themselves and integrating it with their existing manuscript tracking systems. At the time of writing two publishers have adopted the new system, The Royal Society with it s Proceedings: Biological Sciences journal and The Company of Biologists with their Journal of Cell Science. A further three members of the consortium will be adopting the system in the near future. 6 Acknowledgements Many thanks to Dee Wood, ESPERE Project Manager, for her invaluable help and support and to the consortium partners for their patience and advice during the development of the new system. 7 References 1 Wood DJ 1997 ESPERE Annual Report

13 2 Wood DJ & Hurst P 2000 Online peer review: perceptions in the biological sciences Learned Publishing Adobe Systems Incorporated Portable Document Format Reference Manual V1.3 4 GMD FIT About BSCW 5 Bentley R, Appelt W, Busbach E, Hinrichs E, Kerr D, Sikkel K, Trevor J & Woetzel G 1997 Basic Support for Cooperative Work on the World Wide Web. International Journal of Human Computer Studies 46 no The Royal Society Proceedings: Biological Sciences