E-ARK Progress Update March 2015

Transcription

1 E-ARK Prgress Update March Prepared by: Kuldar Aas (Natinal Archives f Estnia), Andrew Wilsn (University f Prtsmuth) Cntent: This reprt summarises E-ARK actins between December 2014 and March 2015, and prvides a brief verview f wrk planned fr the fllwing few mnths. 1. Intrductin 2. Overview f actins frm December 2014 t March Deliverables prduced in the perid 4. Next steps (March June 2015) 5. Next meetings Intrductin The primary scpe f E-ARK is t prvide standardisatin in fur key areas: delivering electrnic data t lng-term preservatin (i.e. preparatin and transfer f data t archives), pen frmats t supprt submissin, preservatin and access t archived data (i.e. what kind f metadata wuld be needed t stre next t the actual data, and hw the data and metadata shuld be frmatted), access t data stred in lng-term repsitries (i.e. which methds and tls culd be used t advance the level f access which currently is rather lacking fr brndigital recrds), using data mining and big data technlgies t facilitate large scale research n archived data. The Prject started n 1 February 2014 and has at the end f January 2015 finished its first year f activity. Overview f actins frm December 2014 t March 2015 The main effrt f E-ARK in mnths has been cntinuatin f the develpment f E- ARK standards used in data preparatin, transfer, preservatin and access. A majr utcme was the frwarding f three deliverables t the Eurpean Cmmissin in January Tw f these deliverables describe the principles f E-ARK Submissin and Archival Infrmatin Packages (accrdingly deliverables D3.2 and D4.2). The third presents a first draft f the E-ARK Maturity Mdel fr digital preservatin systems. The prject cntinues with refining the details f the infrmatin packages and thereafter shifts fcus twards the develpment f the E-ARK tlset. One imprtant result in

2 preparatin fr the develpment phase has been the delivery f a Requirements Specificatin template. The template allws all wrk packages t specify tl and infrmatin package requirements in a cmmn way thrughut the whle E-ARK prject. The bjectives f the template are tw-fld. Firstly, it is cnsidered imprtant t streamline the way archivists cmmunicate with develpers; develpers wuld be illserved with different methdlgies and frmats fr requirements. Secndly, acknwledging that the infrmatin gathering in the first tasks represented a bttm-up apprach t identifying requirements, the template establishes a tp-dwn apprach. A prject all staff meeting, anticipated in the previus newsletter was held at the University f Prtsmuth n February. A separate reprt n the meeting is at the end f the wrk package reprt. As fr the ther actins in the Wrk Packages the main effrt has been as fllws: WP1 (Prject Crdinatin) has carried ut its usual range f prject management activities; WP2 (Use Cases & Pilts) has cncentrated n supprting WP3 WP5 in the develpment f Infrmatin Package and tl requirements by prviding input abut the exact needs f E-ARK pilt sites; WP3 (Transfer f Recrds t Archives) has finalised and delivered t the Eurpean Cmmissin the draft SIP specificatin, deliverable D3.2. This SIP specificatin is in cnfrmance with the cmmn specificatin fr E-ARK IPs. We have als aligned the specificatin with ther knwn SIP frmats which are knwn t the prject frm the best practice reprt (D3.1 Reprt n Available Best Practices ). Despite that we wuld like t encurage readers t inspect deliverable D3.2 clsely and share their pinin n it. We wuld especially like t ask Advisry Bard members t give feedback t the prject n tw questins: Is the E-ARK SIP Draft Specificatin useful and why (r why nt)? Wuld I use it in my slutin(s) and why (r why nt)? In additin WP3 cntinued wrking n pre-ingest and ingest wrkflws and has updated the draft diagrams first created in D2.1. WP4 (Archival Recrds Preservatin) has mainly been wrking n deliverable D4.2 which was successfully published and frwarded t the Eurpean Cmmissin in January The Wrk Package has als cntinued with tl develpment and has added supprt fr relatinal database archiving (SIARD frmat) int their SIP2AIP tl. In cllabratin with WP6, wrk has als started n gathering further requirements fr updating the SIP2AIP tl t a mre mature state and integrating with ther E-ARK tls; 2

3 WP5 (Archival Recrds Access Services) has primarily been wrking n the requirements specificatin template mentined abve. The template has als been put int use by creating high-level illustratins and descriptins f the entire access prcess t ensure that prject participants have a cmmn understanding f the archival access prcess. Based n the high-level descriptins, wrk has started n analysing metadata requirements and discussing a prpsed structure fr the E-ARK DIP; WP6 (Archival Strage, Services and Integratin) has cntinued wrking n the E- ARK reference implementatin. Three main areas f fcus have been: evaluatin f design issues fr implementing and integrating utcmes f WP4 with the integrated prttype develped in WP6 and prttyping f an initial apprach fr integrating EPP, SIP-t-AIP cnversin, and a scalable search facility; cntinued cntributins t the technical evaluatin f the WP4 SIP2AIP cnverter with respect t its integratin and deplyment n the AIT infrastructure; develpment f an initial versin f a custmizable graphical search interface fr demnstratin and prttype develpment purpses. WP7 (Evaluatin & Assessment) has fcused n develping the deliverable D7.1, the first versin f the infrmatin gvernance maturity mdel (see belw). Other activities included a draft design fr a knwledge centre infrmatin system t supprt the harmnisatin f best practices, standards, and ther apprpriate references relating t infrmatin gvernance; WP8 (Disseminatin & Explitatin) has cntinuusly updated the prject website as well as maintained ur newsfeed. WP8 was als the main rganiser f the February prject meetings and the database preservatin wrkshp described belw. Deliverables prduced in the perid There were three deliverables prduced fr the Eurpean Cmmissin in the December 2014 t March 2015 perid, and we wuld very much appreciate it if yu culd take the time t read these clsely and send us feedback by the end f April The dcuments are available n ur public web site ( D3.2 E-ARK SIP Draft Specificatin: This dcument describes a draft SIP specificatin fr the E-ARK prject. It prvides an verview f the SIP structure and sets ut the main metadata elements fr E-ARK SIP. This dcument will prvide the initial input fr the technical implementatins f E-ARK ingest tls. D4.2 E-ARK AIP draft specificatin: This deliverable describes a blueprint fr the structure f an E-ARK AIP. Since an AIP has a ptentially unlimited life span, the dcument als includes a discussin f hw such an AIP may keep its identity 3

4 unchanged, while its physical representatin may change ver time. Finally, the dcument cntains a chapter n the way in which the AIP frmat is t be embedded int the technical wrkflw within which an AIP exists. D7.1 A Maturity Mdel fr Infrmatin Gvernance An Initial Versin: This deliverable sets ut a draft maturity mdel fr infrmatin gvernance which will be used t assess the E-ARK Prject use cases. The mdel has an intentinal fcus n the prcesses being harmnized in the prject, i.e. ingest, archival preservatin, and disseminatin (access). The mdel dimensins are based n SEI CMMI. The maturity levels are based n ISO9001. The definitin f the maturity mdel is based n a develpment methd f J. Becker (fr mre infrmatin please cnsult the deliverable). The deliverable describes the utputs f the first fur phases f the methd, i.e. it (1) defines the prblem, (2) reviews existing maturity mdels, (3) defines the strategy used fr the develpment, and (4) develps the mdel. The remaining steps f the methd, which include applicatin, validatin and review f the maturity mdel, are due t be cmpleted by the end f June Next steps (March June 2015) During the spring perid in 2015, wrk packages 3 6 will cntinue t develp the draft E- ARK standards fr data preparatin, transfer, preservatin and access. Mst crucially the wrk n IP specificatins will cntinue and braden. Tw additinal subgrups have been established t wrk mre explicitly n: E-ARK requirements fr exprting recrds and metadata frm electrnic recrds management systems (ERMS): this subgrup will develp the principles and frmats fr exprting archival recrds frm business systems that manage dcuments and recrds, as a pre-ingest prcess. It will als fcus n develping metadata requirements based n the MReq standard. The primary gals fr early 2015 are: the definitin f recrds system specific exprt requirements t be implemented by a recrds system that exprts t an archive; the specificatin f a reference data mdel and metadata elements. These are t be used fr the definitin f the cre E-ARK SIP prfile fr ERMS; E-ARK SIARD-E frmat: this subgrup will develp a specificatin f an E-ARK archival relatinal database frmat (wrking name SIARD-E), based n the best practices frm the existing SIARD, SIARDK and DBML frmats. The subgrup will als gather additinal requirements fr updating apprpriate tls with supprt fr the validatin, quality and reuse f archived databases. In mre detail the wrk in individual Wrk Packages will be as fllws: WP1 (Prject Crdinatin) will cntinue the management f the prject. WP2 (Use Cases & Pilts) will cntinue supprting WP3 WP5 in the current requirements gathering phase frm the pint f view f the E-ARK pilts. It will als start gathering infrmatin arund legal issues in E-ARK pilt implementatins. The aim is t prduce the apprpriate deliverable D2.2 by mnth 17 (June 2015). 4

5 WP3 (Transfer f Recrds t Archives) will expand and cntinue t develp the SIP draft specificatin t include supprt fr MReq 2010 based metadata and the prpsed SIARD-E frmat. By May 2015, the next versin f the SIP specificatins as well as tl requirements will be available and large-scale develpment f specific E-ARK tls (recrds management exprt mdule fr Alfresc, database preservatin tlkit and SIP creatin tls) will start. WP4 (Archival Recrds Preservatin) will cntinue wrking n the SIP2AIP tl and the AIP specificatin, particularly fcussing n the needs and pssibilities fr enriching and de-nrmalising structured data during ingest. A cnceptual slutin fr string parallel representatins f structured data in an AIP (riginal database structure and the de-nrmalised presentatin) will be available by summer Wrk will als cntinue arund the integratin f the SIP2AIP tl with the EPP digital preservatin platfrm. WP5 (Archival Recrds Access Services) will finalise the access use cases and the functinal requirements fr the E-ARK access scenaris. A draft DIP frmat is scheduled fr release at the end f April and we will be seeking Advisry Bard cmments at that time. In additin the use cases and requirements will be used t start the develpment f the E-ARK access tlkit. WP6 (Archival Strage, Services and Integratin) will fcus n the upcming deliverable D6.1 Faceted Query Interface and API, which is due t be delivered at the end f April. Technical wrk will include develping supprt fr full text indexing f ingested ffice dcuments. In this cntext, it is planned t deply a publicly accessible demnstratr which allws ther wrk packages t evaluate the WP6 search service API. The WP6 search interfaces will be evaluated with respect t supprting the DIP creatin tl tgether with WP5. WP7 (Evaluatin & Assessment) cntinues t wrk n a first versin f the E-ARK knwledge base. The knwledge base will cmprise: A Requirements Surces Manager t manage the standards and reference dcuments that cntain Infrmatin Gvernance requirements. The cmpnent will allw semi-autmatic extractin f requirements frm different surces; A Vcabulary Manager t manage the terms and definitins in the existing surces; A Requirement Manager that will manage requirements frm existing surces; Assessment Services that will prvide a set f services that will allw verificatin f cmpliance with existing requirements. An On-Line Self- Assessment tl will prvide an nline assessment based n the Infrmatin Gvernance Maturity Mdel while the Schemas Validatr will allw the verificatin f XML Schemas against defined XSD r requirements (e.g. validatin f XML Schemas based n the SIP, AIP and DIP schemas); 5

6 The Data Stre will manage and stre all the infrmatin frm the previus cmpnents. The View Manager will be respnsible fr cnslidating, persnalising and displaying the infrmatin stred in the Data Stre. WP7 is als currently gathering feedback n the draft E-ARK maturity mdel which will be used t measure the success f the pilts in year three f the prject. WP8 (Disseminatin & Explitatin) will cntinue the update f cmmunicatin channels and carry ut disseminatin. E-ARK Prject All Staff Meetings, February 2015 Frm February, E-ARK Prject all staff meetings, as well as a meeting f the Prject Bard and the Executive Steering Cmmittee, were held at the University f Prtsmuth. The prgram started with an all staff meeting n the afternn f 16 February which discussed a number f issues, including: Cmmn Specificatin Requirements Specificatin template Specificatin f tls d be develped in E-ARK Use f MReq metadata in archiving Extending SIARD The all staff meeting cntinued n the mrning f 17 February with an intrductin by the Prject Crdinatr, Janet Delve, fllwed by a discussin f a number f administrative issues, including end f year reprting. The next sessin f the meeting cnsisted f Advisry Bard and Wrk Package updates frm the respective WP leads. On Tuesday afternn a develpers meeting was held in parallel with sequential sessins n editing the E-ARK website using the JOOMLA sftware, and using the EC NEF system. These sessins were fllwed by a prject bard meeting in the late afternn. Wednesday mrning saw a series f Wrk Package meetings including a WP2 & 7 jint meeting and a parallel jint meeting f WPS 4 & 6. In the afternn a WP5 meeting was held, with an Executive Steering Cmmittee meeting taking place in parallel. We wuld like t highlight that these meetings helped essentially t visualise the cnnectins between individual wrk packages and grups. In practical terms the meetings helped especially t transfer knwledge between develpment, research and archival partners in preparatin f the large-scale develpment phase (t be started in late spring 2015). As the utcme we were ensured that we can be cnfident in ur radmap twards a set f interperable E-ARK specificatins and tls. 6

7 Database/Data Mining Wrkshp: Is a Data Warehuse a Data Archive and Why Des It Matter? February 2015 One f the items in the cre f the E-ARK wrk prgramme is the archiving and reuse f structured data derived frm relatinal database management systems. These systems are ne f the essential building blcks f infrmatin technlgy. Ubiquitus but ften bscured behind layers f scripting, prcessing, frms and queries, they are arguably the mst imprtant inventin f the Twentieth Century. There s n questin that databases present a cmplex challenge t preservatin. They can be difficult t dcument and difficult t understand even when they are dcumented. The cmplex interdependencies f data, query and scripting make migratin prblematic and highly specialised. Relatinal databases and t sme extent data warehusing appraches, which favur structure and hmgeneity, are smetimes cntrasted with big data appraches that tend t favur hetergeneity and de-nrmalisatin. It culd be suggested that a cncern with relatinal databases is utmded and that the preservatin cmmunity culd simply adpt big data appraches. But the cntrast can be verstated, especially when preservatin issues are discussed. In practice big data tls seem t ffer imprved wrkflws that cmplement rather than replace existing data warehuse tls. And even if big data tls are the slutin fr access they still need t integrate with fundamental preservatin prcesses and standards. Better technical guidance and rganisatinal knw-hw are needed if the digital preservatin cmmunity is t ffer cnfident and cnsistent slutins t lng-term access fr relatinal databases. This wrkshp, held ver ne and a half days at the University f Prtsmuth, was jintly spnsred by the E-ARK Prject and the Digital Preservatin Calitin (DPC). The wrkshp was split ver tw days, with day ne aiming t: Clarify where databases, data warehuses and big data cmplement / verlap each ther Review the state f the art in the preservatin f databases Present case studies f current tls and practices arund the preservatin f relatinal databases Intrduce cmmercial appraches t data warehusing and explre the relatinship with preservatin Intrduce big data appraches fr database preservatin Day tw s aims were t: Review the state f the art in the use f big data and its implicatins fr preservatin Examine and debate the use cases fr archived databases / big data Identify recmmendatins fr further research and guidance in the preservatin f big data 7

8 Cpies f the presentatins are available at: Vides f the event will be available sn, als frm the E-ARK website. 8