OPEN CALL FOR TENDERS No MARE/2014/14. Appendix 2. EUMOFA IT architecture and design

Transcription

1 EUROPEAN COMMISSION DIRECTORATE-GENERAL FOR MARITIME AFFAIRS AND FISHERIES INTERNATIONAL AFFAIRS AND MARKETS TRADE AND MARKETS OPEN CALL FOR TENDERS No MARE/2014/14 Appendix 2 EUMOFA IT architecture and design

2 EUMOFA EUMOFA ARCHITECTURE page 2 of 12 Contents 0 Purpose of EUMOFA portal System Architecture Schema of the system Organisation of the data center Expertise required to currently run EUMOFA portal Content management system Access to the EUMOFA Security

3 EUMOFA EUMOFA ARCHITECTURE page 3 of 12 0 Purpose of EUMOFA portal The EUMOFA project encompasses the implementation of a Web Portal, featuring an interactive webinterface for consultation and on-line queries, and a tool for data extraction. The Portal is available in four languages: English, French, Spanish and German. The EUMOFA Portal covers three distinct time scopes, featuring the following data tools: Weekly prices Predefined queries Ad-hoc queries Monthly trends (monthly/quarterly) Monthly highlights Predefined queries Ad-hoc queries Yearly data The EU fish market Predefined queries Ad-hoc queries The publication of case studies concerning price structure as for identified products in specific countries is also envisaged. 1 System Architecture 1.1 Schema of the system The software used for developing all applications includes: Talend Open Studio (ETL Tool) is a powerful and versatile open source solution for data integration. It generates Java code that accesses the database through DBC drivers. Talend is a valid alternative to more expensive commercial tools to meet the needs of integration, harmonisation and data processing; Liferay (Web Application Data Collection / Web Portal) is a Java-based portal development tool, thus no problems are expected when running in the DG DIGIT environment. Jaspersoft (Web Application Business Intelligence Tool) is one of the open source Business Intelligence leaders. The Enterprise edition adopted includes tools for producing reports, analysis, ad hoc querying and dashboard building. The commercial licence, while not including the costs of software since it is free of charge, provides support certification and complementary tools useful for meeting the EUMOFA s requirements. 3

4 EUMOFA EUMOFA ARCHITECTURE page 4 of 12 The figure below outlines the logical IT architecture of the EUMOFA, showing in each box on the left column which technology has been adopted: Figure 1 Logical architecture of the EUMOFA: three-tier Enterprise Application 4

5 EUMOFA EUMOFA ARCHITECTURE page 5 of 12 Figure 2 Figure presents the current deployment: Figure 2 EUMOFA current deployment The EUMOFA IT system can be briefly outlined as follows: Table 1 Schema of the IT system designed for the EUMOFA Component Features Technology/product Data Collection Application Business Intelligence Tool Web Portal Processing Engine Collects data from National and other Data Sources and International Statistics bodies, through a web interface Tool for data queries and analysis (Structured and Unstructured) Web site that works as a point of access to the EUMOFA information and data dissemination Performs Data Quality checks, Integration, Harmonisation, Historicisation and Loading of data Java portlets Java/JasperReports Server on Tomcat Java/Liferay on Tomcat Java/Talend Data Warehouse Stores Data for the Business Intelligence tool Oracle 5

6 EUMOFA EUMOFA ARCHITECTURE page 6 of 12 Figure 3 - Schema of the IT system designed for the EUMOFA Objects available in each tool: Talend The system consists of 2 ETL macro processes: Elaboration, Check and Upload in a stage area of Data Collection Application (DCA); Transformation and upload of data in the DWH for the creations of reports and statistics. These systems are composed by about 150 objects. Jaspersoft on reporting Jaspersoft reporting system is composed by the following components: WSO2: It is the Daemon intercepting the DCA files uploaded by users and activating the Q_consumer for managing the uploaded info; Oracle PL-SQL objects Q_Consumer: Java/PL-Sql procedure is used, activated by the WSO2, checking i) the DCA files in queue, ii) their content and iii) the correctness of the structure envisaged for each field. It imports these checked data in an unstructured file, input for the talend processes for the predisposition of the stage area. JasperReport Server:Two types of reports are available: The predefined reports enable the users to perform multi- and mono-dimensional queries by using a set of dimensions and receiving predefined reports as a result of the query. They are divided into 3 typologies: Weekly: 3 macro categories containing 8 predefined reports; Monthly: 3 macro categories containing 16 predefined reports and 60 drill down; Yearly: 6 macro categories containing 11 predefined reports and 21 drill down. For each typology, there are 4 universes (domain), 1 for each language (EN-FR-DE- ES), which can be consulted through ad-hoc queries. The ad-hoc queries are built by the users on the basis of defined parameters. For the consultation and creation of reports generated through ad-hoc queries, 12 Topics have been defined, 1 for each universe, for managing the info selected by 6

7 EUMOFA EUMOFA ARCHITECTURE page 7 of 12 the user which can be used for building a customized report. The queries built by the user can be saved for further consultations. Liferay portlets Accounts management: it allows the creation and management of users accounts; Annual and monthly publications: it allows the management of documents; Jaspersoft reports (weekly monthly - yearly), with the possibility to make queries either predefined or built by users (ad-hoc): o Predefined queries enable the users to perform multi- and mono-dimensional queries by using a set of dimensions and receiving predefined reports as a result of the query. o The ad-hoc query system relies on special features provided by the Open Source BI (Business Intelligence) tool Jaspersoft, customized with the aim at integrating it with other features of the portal. Oracle schemas Six schemas are envisaged within Oracle: free repository; jasper; talend; data collection; datamart DWH; MIS2000, encompassing the storage of FIDES data up to 2nd quarter of

8 EUMOFA EUMOFA ARCHITECTURE page 8 of Organisation of the data center Figure 4 Organisation of the Data Center The implemented system is composed by a virtual machine, housed by the Directorate-General for Informatics - European Commission (DIGIT), running Open Source Software for the BI, ETL and CMS tool, in connection with an Oracle RDBMS managed by DG DIGIT in a hosted server. DIGIT virtual machine Allows EUMOFA to benefit of open source software; DG DIGIT deals only with the server power on and off; Does not involve maintenance costs for DG DIGIT. THE HOSTED ORACLE SERVICE RDBMS is managed by DG DIGIT, the critical data are stored in DWH environment; Enables to benefit from high quality service for DWH storage. 1.3 Expertise required to currently run EUMOFA portal Oracle DBA (part time) Senior java developer (part time) Portlets Java junior (full time): administration, liferay, Jasper report good knowledge and experience with Liferay, Jasper and Talend products for the Java developers (Senior and Junior) 8

9 EUMOFA EUMOFA ARCHITECTURE page 9 of Content management system The system consists of about 40 pages. The following table summarizes the size of the database: Table 2 Size of the EUMOFA database Tablespace IDX Gb DATA Gb Totale Gb Free Mb Used Mb % Free % Used TS_EUMOFA_DCA 18, TS_EUMOFA_DCA_IDX 9,53 27, TS_EUMOFA_DWH TS_EUMOFA_DWH_IDX 8,8 26, ,33 36,1 54,43 Recent increase: about 40% per year Expected increase: about 40% per year Number of users per day: accounts created in the period between the EUMOFA launch of 24 th April accounts created since the beginning of the Test period. 98 test users registered before the EUMOFA launch. On average, the users daily accessing the website are 50 per day. The monthly increase expected is by 10%. 9

10 EUMOFA EUMOFA ARCHITECTURE page 10 of 12 2 EUMOFA Environments 2.1 Application Servers As mentioned above, IT items are located in DIGIT where three instances of the whole system has been installed: ENVIRONMENT DETAILS Test It is the development environment Acceptance Environment devoted to final approval of new developed items before moving to production Production Operational environment, devoted to be open to the web Physical detail of the server: Memory: kb Linux version: Red Hat Enterprise Linux Server release 5.9 (Tikanga) CPU: Amount of disks: Used: Available: Back-up policy: Every Day, one week cycle before erase Physical detail of the server: Memory: kb Linux version: Red Hat Enterprise Linux Server release 5.8 (Tikanga) CPU: Amount of disks: Used: Available: Back-up policy: Every Day, one week cycle before erase Physical detail of the server: Memory: kb Linux version: Red Hat Enterprise Linux Server release 5.8 (Tikanga) CPU: Amount of disks: Used: Available: Back-up policy: Every Day, one week cycle before erase Access to these three environments is granted through a VPN connection via token, achievable following the signature of a Security Convention between the external IT contractor and the European Commission. 10

11 EUMOFA EUMOFA ARCHITECTURE page 11 of Oracle Database Servers MACHINE ENVIRONMENT DETAILS Physical detail of the server: 64 cpus, SPARC64-VII+ at 2660MHz Physical detail of the virtual machine: 64 cpus, 192GB memory Acceptance (db EX1MARB) Linux version: Solaris 10 CPU: 64 cpus (uncapped zone) Database server Production (db EX1MARA) Amount of disks:n/a (shared by several databases) Back-up policy: backup policy for databases is 1 full per week, 1 archived logs backup per day. Physical detail of the server: 64 cpus, SPARC64-VII+ at 2660MHz Physical detail of the virtual machine: cpus, 56GB memory Linux version: Solaris 10 CPU:18.67 (zone) Amount of disks: N/A (shared by several databases) Back-up policy: backup policy for databases is 1 full per week, 1 archived logs backup per day. MACHINE ENVIRONMENT DETAILS Database server Test (db EX1UMART) Physical detail of the server: 64 cpus, SPARC64-VII+ at 2660MHz Physical detail of the virtual machine: 64 cpus, 192GB memory Linux version: Solaris 10 CPU: 64 cpus (uncapped zone) Amount of disks: N/A (shared by several databases) Back-up policy: backup policy for databases is 1 full per week, 1 archived logs backup per day. * These specifications are in regards with other Oracle databases included in the same zone at Data Center of DIGIT. Exclusively for the EUMOFA Oracle Database server specifications can be lower but an Oracle High Availability solution must be provided for the production environment. 11

12 EUMOFA EUMOFA ARCHITECTURE page 12 of 12 3 Security The management of the EUMOFA portal s accesses is monitored through the Single Sign On system CAS, a YASIG s project, enabling to integrate in a single access point the authentication to both Liferay and Jaspersoft. Three user types are envisaged: Administrator: Allowed to define and profile users from the access to all available functions; Users: they browse the portal for consulting available data and publications. The registration is free; Publishers: they disseminate data which are elaborate by the system for creating the Talend repository The registration is monitored by the Administrator. 12