TRANSFERRING PROJECT DOCUMENTS AND ASSOCIATED METADATA BETWEEN COMPANY DOCUMENT MANAGEMENT SYSTEMS AND PROJECT EXTRANETS

Transcription

1 TRANSFERRING PROJECT DOCUMENTS AND ASSOCIATED METADATA BETWEEN COMPANY DOCUMENT MANAGEMENT Alastair Watson CAE group, Civil Engineering, Leeds University, LS2 9JT Mehdi Davoodi CAE group, Civil Engineering, Leeds University, LS2 9JT Alastair Watson, Mehdi Davoodi ABSTRACT: This paper reports on work to enable the automatic transfer of project documents and the associated metadata between company-level document management systems and project-level project extranets. The industrial need for such a capability is described, the functional needs defined and the subsequent realisation as XML transactions is explained. The resulting DocLink specifications are introduced and their practical application is illustrated. 1. INTRODUCTION Construction projects generate a considerable volume of documents that are transferred between the members of various project teams prior to, during and after construction. Today the primary form of most of these documents is digital, while developments in networking and Internet technologies have provided the means both to manage digital documents and to them between the members of a virtual project team. These advances in technology have improved productivity within project teams, and are now beginning to have wider impact on the way that construction projects are executed. Possibly the most significant indicator of wider changes is the growing use of project extranets (or project websites ). At its simplest, a project extranet is a secure website that provides a project-level document management capability enabling the participants in a virtual project team to remotely publish and access project documents. Typically control is provided over which participants can download which documents, and support is included for related functions such as notifications, handling revisions and maintaining audit trails [AJ+, 2002]. The logistical advantage of being able to provide all members of a construction project with easy access to the current versions of all relevant documents, irrespective of their geographical location, is substantial. Such a platform also has considerable potential to help improve the collaborative culture across a virtual project team [Smith, 2002]. The ability to graft other added-value services onto a project extranet, together with the dot.com bubble, fuelled growth in both the numbers of project extranet service providers and the breadth of the additional (potentially revenue generating) services. Consolidation has since taken place, but a substantial number of competing project extranet service providers still exist. While the basic projectlevel document management services tend to be functionally similar, and will probably tend to converge, there is still considerable diversity in the extent and the function of any added-value esm@rt and CISEMIC

2 Alastair Watson, Mehdi Davoodi services. At the upper end of the spectrum the more complex sites, which tend to be described as project portals, provide comprehensive additional services. For example, applications software (the provision of fully managed software) and information services (the marketing of information resources and product data), both of which are seen as significant generators of future revenue streams. Various charging models are currently employed by project extranet service providers ranging from low-cost with adverts services through fixed charge per project to a charge per project participant. Typically the charges are per month with a surcharge for large project storage requirements. Additionally, some project extranets focus more strongly on particular stages of the total project life-cycle (for example, by emphasizing the post-construction advantages in supporting facilities management). The current diversity of project extranets and the relative immaturity of the market is part of the reason for the work described in this paper. In this context it is only the basic (but universal) document management capabilities of project extranets that are of concern. This functionality is implemented using standard website technology in which a soft web interface is generated dynamically from an underlying database. The database holds the location of each project document that has been uploaded to the website together with a significant amount of metadata. The metadata includes information about each participant in the project, and the organization they work for, plus information about each document (including who is to be granted access). Typically a transaction log is also maintained within the database. The software employed normally allows multiple projects to be hosted on a single physical web server, and is usually designed to be highly configurable to the individual needs of each project. Significantly, it is still the norm for the project extranet software to be custom written by (or for) the particular project extranet service provider. A major factor in the growing use of project extranets has been their web-based interface. This means that to be able to access the project extranet, a project participant only needs an internet connection and a PC with a standard web browser installed. The implicit need for specialist application software to be installed to view and print particular document file formats is minimised by the provision of downloadable plug-in viewers and, in some cases, by the automatic generation of (say) PDF versions of uploaded document files. Thus any member of a virtual project team is able make use of a project extranet with a minimum of local IT infrastructure (see Fig. 1.). The participants in a project can be offered tangible improvements in operational efficiency, based on the controlled transfer of digital documents, without the need to make major changes in industrial practice or face unfeasibly high entry costs. Project Extranet Documents and metadata Web Browser Member of virtual project team Fig. 1. Using a Project Extranet esm@rt and CISEMIC

3 TRANSFERRING PROJECT DOCUMENTS AND ASSOCIATED METADATA BETWEEN COMPANY DOCUMENT MANAGEMENT Project extranet usage in the UK is estimated to exceed 3,500 projects [AJ+, 2002]. The restraints that have been acknowledged include the evident need for the leading members of the project team to take the risk of introducing a project extranet, legal and contractual issues, plus more practical aspects such as the implicit shift in the burden of document printing costs. 2. THE INDUSTRIAL NEED The trend towards using project extranets has been encouraged by some of the more knowledgeable repeat-customers of the construction industry. In the absence of dominant players within the supply chain, organic growth is also taking place (particularly on larger projects) driven by the practical advantages. Some of the major players in the industry have developed in-house project-extranet solutions, but concerns from other project participants that an independent third-party should hold the project documents have favoured independent project extranet service providers. While the use of project extranets via a web-based interface has many practical advantages, it also has a number of significant limitations: The project extranet software employed by each service provider has a different user interface. Even when using the same project extranet software, this may well be configured differently for each project. Uploading a document to a project extranet is a manual operation (during which relevant metadata must be input by hand). Downloading a document from a project extranet is also a manual operation. Any company-level document management system is unable to interact directly with project-level project extranets. The project-centric nature of the construction industry means that most companies are concurrently involved in many projects. Potentially each company may need to interact with many different project extranets, with the consequence that an individual may need to frequently switch between several different systems. These limitations are thrown into sharper focus when a company also makes significant internal use of a document management system. Each document must then be manually double handled: check-out and upload or download and check-in (see Fig. 2.). When transferring documents in either direction the relevant metadata must be re-entered manually, a labour intensive and error prone task. More critically, where internal workflow or quality related procedures have been based on the company-level system these procedures are likely to be invalidated. Project Extranet Documents and Metadata Company Fire Wall Document Management System Fig. 2. Manual Double Handling 3

4 Alastair Watson, Mehdi Davoodi Some project extranets have the option of using an alternative user interface that is not web-based. Typically this requires the installation of Windows Explorer like client software on the user s PC. The task of uploading document files to (and downloading files from) the project extranet is made a more transparent, extranet files appear to be an extension to the local Windows filestore and can be dragged and dropped. This is an effective strategy that also allows some metadata to be acquired automatically from Windows, or implicitly from the location of the document within a project file structure. While this can make a project extranet more user-friendly, the more strategic objectives of a corporate-level document management system are still likely to be violated. The approach may also require a particular user to install multiple, and potentially incompatible, project extranet clients on his PC. This paper addresses the growing industrial need to complement the current human centric interfaces of project extranets by a standardised interface for the direct transfer of documents and metadata to and from company-level document management systems. It is anticipated that this need is replicated in other project-centric industrial sectors beyond mainstream construction. 3. GENERIC METADATA MODEL An initial list of what metadata might need to be transferred between company and project level systems was assembled by studying sources relating to document metadata [ISO, 1993 and 2000]. Informed by assessments of a number of project extranets, this initial list was refined, reviewed and structured into a model. This model has since matured into a generic model of metadata, the metadata entities and their relationships are illustrated in Fig. 3. Group User Organisation Project Set Structure Document Revision Fig. 3. : DocLink Generic Metadata Model File Metadata needs to be transferred about users and organisations, about projects and about documents. These documents may each have one or more revisions, and each revision may be held as one or more files. Metadata about collections can also be transferred: groups of users and either sets or structures of documents. A group or a set is a simple collection of users or documents. In contrast a structure is a collection in which a specified document forms the root of the structure, and the other documents each have a specified filestore pathname relative to the root document. esm@rt and CISEMIC

5 TRANSFERRING PROJECT DOCUMENTS AND ASSOCIATED METADATA BETWEEN COMPANY DOCUMENT MANAGEMENT Each of the attributes of these metadata entities are defined as being mandatory or optional. The power and flexibility of the generic metadata model is substantially increased by the inclusion of optional classification and (project extranet specific) additional parameter attributes for the document, revision and file entities. At each of these three levels, classifications can be specified by a sequence of classification nodes. Each classification corresponds to a valid path through any predefined (explicit or implicit) hierarchical classification structure that the target project extranet may employ for documents, revisions or files. The additional parameters comprise label and value pairs, thus providing a complementary ability to capture additional discrete metadata attributes that are not included explicitly within the generic model. 4. FUNCTIONAL REQUIREMENTS An assessment of the required interface between company-level and project-level systems (now known as the DocLink specifications) identified a number of significant functional requirements: Must work with a wide range of different project extranets, implying differing philosophies and thus significant differences in their metadata. This implies the requirement that DocLink employs a robust generic model of the metadata, and suggests the use of a suite of relatively simple modular interactions. It also implies that in practice, the mapping of the metadata between a particular company-level and a particular project-level system is unlikely to be complete. Replicating the current functional role of a project extranet, the interface should be implemented as a client-server architecture with the project extranet acting as the server. This leads to the key requirement that the DocLink interface must pass metadata values that are aligned with the project extranet (rather than being aligned with the document management system). Server-side implementation and validation would need to take place first, thus establishing a mapping between the database of the project extranet software and the DocLink metadata model. Hence the requirement that the specifications (and any testing regime) should focus on defining what services a particular project extranet could support, what services have been implemented and what services have been implemented correctly. It is envisaged that most client-side implementation will be in two-stages, with a generic adaptor providing the basic mapping between that database of the document management system and the DocLink metadata model. A plug-in adaptor for each target project would complement the generic adaptor. Based on common code for each target project extranet software, each plug-in will need to be highly configurable (via lookup tables for example) to define the capability to correctly convert metadata values to (and from) the requirements of a particular project. Hence the practical requirement that server-side implementations be able to assist in the necessary initial configuration of a client-side implementation (by the provision of easily understandable error messages and warning messages). The last point is a consequence of the decision that the metadata values carried by the interactions need to be aligned with the project extranet, and the related fact that this will employ an underlying database. All references to a particular user, organization, project, document, revision or file will thus be by means of their corresponding IDs within the database of the project extranet. For a document management system to interact successfully with a particular project extranet, it must have some ability (implemented within the plug-in adaptor) to track these entity IDs. Other database IDs are likely to be encountered where the necessary ability of a project administrator to configure the project extranet has been implemented using look-up tables. For example, where document classification is against a set of enumeration values that are project-configurable. 5

6 Alastair Watson, Mehdi Davoodi 5. THE DOCLINK SPECIFICATIONS 5.1 Indicative Scope An extranet for more than one project may be hosted on a physical website. Users can be collected into groups. A users is assumed to belong to an organisation. Documents can be collected into sets. Documents can be collected into structures. Documents can be published as one or more revisions. Revisions can be published as one or more files. Files can be flagged as being simple or complex (but no explicit support for complex files is assumed of the project extranet). Access control can be down to file level. Flexible capability for conveying document, revision or file level classifications. Flexible capability for conveying additional project extranet specific document, revision or file level parameters. Table 1. The DocLink Transactions # Transaction Use [1] Upload document Upload the existence of a new document [2] Upload revision Upload the file(s) for a new document Upload a revision to an existing document [3] List access rights List current visibility of specified file [4] Set access rights Overwrite current visibility of specified files [5] Notify users (of documents, revisions, files) Request that a specified message be sent to specified users [6] Download document information Download information about a specified document [7] Download revision information Download information about a specified document revision [8] Download file Download a specified file [9] Download user information Download information about a specified user [10] Download group information Download information about a specified group [11] Download organisation information Download information about a specified organisation [12] Update document information Overwrite current information about a specified document [13] Update revision information Overwrite current information about a specified document revision [14] Create set of documents Form specified documents into a set [15] Download set information Download information about a specified set [16] Create structure of documents Form specified documents into specified structure [17] Download structure information Download information about a specified structure [18] Create new user Create a new user [19] Update user information Overwrite current information about a specified user [20] List all IDs Download a structured list of the current entity IDs [21] List publishers documents Download a structured list of the documents, revisions, files that a specified user uploaded [22] List visible documents Download a structured list of the documents, revisions, files visible to a specified user [23] List who accessed file Download a structured list of the accesses to a specified file [24] List possible classifications Download all possible document, revision and file classifications [25] List possible parameters Download the labels of any additional document, revision and file parameters esm@rt and CISEMIC

7 TRANSFERRING PROJECT DOCUMENTS AND ASSOCIATED METADATA BETWEEN COMPANY DOCUMENT MANAGEMENT 5.2 XML Transactions The DocLink specifications [Leeds University, 2002] are based on XML, the extensible Markup Language which is an open web standard published by the World Wide Web Consortium [W3C, 2000]. XML is a simplified descendent of SGML and has similarities to HTML. While HTML provides predefined markup tags to control the presentation of information within a web browser, XML allows the definition of new markup tags that give structure to data. XML is well suited to conveying information like the DocLink metadata and has the major advantage that XML technology is easy to use, inexpensive and readily available. The flexibility and utility of XML is such that it is currently finding widespread application, for example: The reformulation of HTML by [W3C, 2001] to create XHTML, effectively making XML the parent web document markup standard. IfcXML a light weight means of accessing product model data [IAI, 2001]. As the basis of new electronic business initiatives such as ebxml [ebxml, 2000] and, within the construction industry, aecxml [IAI, 2000] and bcxml [econstruct, 2001]. A means for reworking existing EDI standards to improve take-up in sectors such as construction [CITE, 2001]. While it is still too early to judge the long-term significance of XML, it is already a very influential and widespread standard for data transfer and data access (and has other application areas). The DocLink specifications define twenty five modular transactions as listed in Table 1. A given project extranet may only implement a subset of these transactions, but each must be implemented in full. A DocLink transaction comprises a request message to the project extranet and a corresponding response message from the project extranet as illustrated in Fig. 4. For each message there is a governing XML schema [W3C, 2001] that was derived from the generic metadata model. The DocLink messages all have a common header section, which is normally followed by a payload section. The header of a request message must contain the project extranet ID of the project and the ID of the user (i.e. of the company-level system) plus the password of that user. No meaningful response is returned unless these attribute values are correct. The header of a request message must also include a unique transaction ID that is assigned by the company-level system. Assuming the header is valid, the project extranet attempts to act on the payload and returns a response message. The header of the response message contains the same transaction ID, allowing the response to be matched with the request. If a fatal error occurs the response message will not include the expected payload, but will provide full details of why the error has occurred (e.g. schema violation, transaction sequence, missing value, invalid value, or illegal operation). If the request message includes parameters that are not required by that particular project extranet, the transaction will be acted upon but the response message will include explicit extra value warning messages. Should a project extranet receive a request message for a transaction that it does not support a special error response message is returned. A DocLink message takes the form of an XML document that is transported as an attachment. A project extranet with a DocLink implementation may receive concurrent streams of request messages from different company-level systems. Most of the DocLink transactions are discrete and can thus be issued in any sequence. 5.3 Implementation Aspects Within the context of the Procure project, the concept of the DocLink transactions was initially trialled by Leeds University using a simulation of a project extranet [Underwood, 2002]. Subsequently a server-side implementation against the beta version of the DocLink specifications was implemented by Sarcophagus on their project extranet the-project [Sarcophagus, 2002]. This implementation was extensively tested by Leeds University. A prototype client-side implementation by Taylor Woodrow for Athena, their corporate document management system, followed. The industrial objective was to 7

8 Alastair Watson, Mehdi Davoodi provide an information acquisition capability to automatically replicate within the file structure and metadata of Athena all the (currently visible) project documents that are held on the project extranet [Stevens, 2002]. Server-side Request Message Payload Response Message Error Warnings Payload Client-side Fig. 4. A DocLink Transaction The formal DocLink specifications [Leeds University, 2002] includes a twenty page overview that addresses the implementation aspects in detail, provides an outline specification of each transaction and a formal definition of each metadata attribute. A specification for each of the individual transactions is linked to the overview, providing a diagram and listing of the XLM Schema plus an example for each message. An ASCII file of each schema is also provided. Implementation has proven to be relatively straightforward using tools such as the Microsoft XML parser [Microsoft, 2001] which can be downloaded at no cost and which supports the Document Object Model (DOM). Any server-side project extranet DocLink implementation must be completely independent of which company-level systems interact with it. The implementation must support and correctly populate all the mandatory parameters, in both the request and the response messages, of the transactions it supports. It must also correctly return the required error and warning messages. An initial concern was the security of the project extranet but adequate safeguards have been incorporated into the specifications. 5.4 Illustration of Use The following example illustrates how a company-level document management system, with a DocLink implementation, might periodically check that it currently holds the latest revision of the project documents as published on the project extranet. Firstly a transaction [22] List Visible Documents request message is sent with UserID set to that of the company-level system. The response message returns a nested list containing the DocumentIDs, RevisionIDs, and FileIDs for the visible documents: <ListVisibleDocuments_Response> <TransactionID>122</TransactionID> <Payload> <DocumentID>1487<RevisionID>2318</RevisionID> <FileID>4531<FileID/> </DocumentID> <DocumentID>1647<RevisionID>2654</RevisionID> <FileID>5534<FileID/> Response not shown for other documents esm@rt and CISEMIC

9 TRANSFERRING PROJECT DOCUMENTS AND ASSOCIATED METADATA BETWEEN COMPANY DOCUMENT MANAGEMENT <RevisionID>2586</RevisionID> <FileID>5734<FileID/> <RevisionID>2867</RevisionID> <FileID>5788<FileID/> </DocumentID> </Payload> </ListVisibleDocuments_Response> Each of these documents is then considered: If the DocumentID has been encountered previously, further action is only required if that document has been revised since the last periodic check. Thus, if the last RevisionID for that document has not been encountered previously, a transaction [7] Download Revision Information request message is sent with that RevisionID. The response message returns the metadata for that revision plus the FileID and medatata for the files(s) that hold that revision. <DownloadRevisionInformation_Response> <TransactionID>123</TransactionID> Response with <Payload> <UploadDateAndTime> RevisionID=2867 <Date> </Date> <Time>16:35</Time> </UploadDateAndTime> <RevisionDate> </RevisionDate> <RevisionCode>3</RevisionCode> <RevisionPublisher>Steven Howarth</RevisionPublisher> <RevisionClassification> <Node>Stage</Node> <Node>3</Node> </RevisionClassification> <RevisionParameters> <Label>pagesize</Label> <Value>A4</Value> </RevisionParameters> <File> <FileID>5788</FileID> <FileSize>107008</FileSize> <FileComplex>0</FileComplex> <FileFormat>doc</FileFormat> <FileName>Extranet Protocols.doc</FileName> </File> </Payload> </DownloadRevisionInformation_Response> After extracting the revision metadata required by the company system, a transaction [8] Download File request message is sent for each FileID. Each response message returns the metadata for a file and the file itself, the required metadata and the file is extracted to the company system. If the DocumentID has not been encountered previously, that document must have been uploaded since the last periodic check. A transaction [6] Download File Information request message is sent with that DocumentID. The response message returns the metadata for that document and the RevisionID for its revision(s). After extracting the required document metadata, transactions [7] and [8] are used as above to acquire the latest revision of that document. This example illustrates several points about DocLink: The generic metadata model views the initial version of a document as being a revision. Company-level implementations need to track DocumentIDs, RevisionIDs and FileIDs. The same data occurs in the responses to a number of transactions. This is to maximise the utility of each transaction, but also has the effect of simplifying company-level implementations. The example can readily be extended to ensure that any changes made on the project extranet to the metadata relating to an existing document, revision or file are also replicated within the company-level system. 9

10 Alastair Watson, Mehdi Davoodi 6. CONCLUSIONS Various approaches to the growing industrial need for more effective sharing of project documents have been proposed (e.g. Rezgui, 1998), currently the project extranet is the most accessible of these approaches. The DocLink specifications add value to this approach by offering the project-centric industries the possibility of seamless interworking between company-level and project-level document management systems. DocLink has been developed, implemented and tested in the context of an industrially lead European research and development project, and the specifications are freely available for implementation and deployment. Based on XML technology, modular transactions and a generic metadata model, the specifications can be implemented relatively easily on any project extranet. By overcoming inherent problems in the use of project extranets by any company that has a local document management system, DocLink should give further impetus to the adoption of project extranets as a more effective means of supporting virtual project teams. DocLink currently uses as the means of transporting the XML documents (messages) between the company and project level systems. This is appropriate since support is easily implemented and encryption could readily be introduced if required. As the volume of transactions increases, it is anticipated that the DocLink transport mechanism will be switched from to SOAP-based direct messaging. In the shorter-term, increased industrial usage of technologies such as project websites and DocLink will improve the efficiency of working within virtual project teams. In the longer term the industrial adoption of such technologies will encourage new ways of working and thus precipitate structural change. 7. ACKNOWLEDGMENT The work described in this paper was conducted in the context of the Procure project (Esprit 29948). The authors acknowledge the financial support from the European Commission and the technical input from the industrial partners. The DocLink specifications can be accessed in full on from where details of the wider Procure project and the collaborators can also be accessed. REFERENCES AJ+, 2002, Project Extranets, CITE, 2001, Emerging Technologies XML, econstruct, 2001, Final edition of the bcxml Specifications, public/bcxml_cd/publicdeliverables/d103_v2.pdf edxml, 2000, Enabling a global electronic market, IAI, 2000, aecxml Mission Statement, IAI, 2001, IfcXML proposed standard, ISO, 1993, Records Management, ISO 15489, ISO, 1993, Technical Product Documentation: Handling of Computer-based Technical Information, ISO 11441, esm@rt and CISEMIC

11 TRANSFERRING PROJECT DOCUMENTS AND ASSOCIATED METADATA BETWEEN COMPANY DOCUMENT MANAGEMENT ISO, 2000, Management Data (Metadata) for Technical Documents, IEC/ISO 82045, ISO, 2000, Technical Product Documentation Metadata for Construction Documentation, PD ISO 19033, Leeds University, 2002, The DocLink Specifications, Microsoft, 2001, Microsoft XML Parser 4.0 July 2001 Technology Preview, Rezgui, Y., Cooper, G., 1998, A proposed open infrastructure for construction project document sharing, ITcon, Vol.3 (1998), p11-24, Sarcophagus, 2002, Sarcophagus Ltd. application service providers, Smith, S., 2002, Update on AEC Project Collaboration Solutions, Stevens, 2002, The TW DocLink Implementation, Meeting Slides JS.pdf Underwood, J., Watson, A., 2002, An XML metadata approach to seamless document exchange, Submitted to Engineering Construction and Architectural Management. W3C, 2000, Extensible Markup Language (XML) 1.0 (Second Edition), W3C, 2001, XHTM 1.0: The Extensible HyperText Markup Language (Second Edition), W3C, 2001, XML Schema, 11