Data Services as a Service for the G-Cloud

Transcription

1 Service Definition The name of the Service is: Data Services as a Service An overview of Data Services as a Service The Data Services as a Service is developed by Cloud Intelligence Services Ltd and delivered as a managed service by UK services partner Amtex Solutions Ltd. The service enables data stored in legacy/operational systems, in the cloud and on-premise, to be exposed as a web service and publish via the Cloud as an API. It is the basis for enablement of e-government allowing collaboration and re-use of the existing systems/processes and the orchestration of new services/business processes. APIs are exposed in a controlled and secure manner. Data access is provided without ever having to open up databases to networks or the cloud. One of the key initiatives that can be supported is open data - making public sector data accessible on the internet through a secure and manageable process. Data Services support a number of business use cases including: Open Data Enterprise Application Integration Master Data Management Operational Intelligence Content Management and Electronic Data Management Mobile and Web Applications Data Services is architected for the cloud including following capabilities: Innovative NoSQL data stores Associative, Document and Graph Secure and scalable data store at IL2 and IL3 levels Full multi tenancy allows each department/agency to deploy services/apis meanwhile maintaining separate records for citizens, employees, properties etc Elasticity and load balancing allows computing resources to be dynamically allocated as needed Identity management based on roles and policies to ensure only authorized users have access to the services, API and data sets Data services permit new cloud applications and APIs to pull data from disparate systems in order to deliver the necessary information or complete a transaction API management allows extension of government data/information and services to other government agencies or authorized 3 rd parties (e.g private sector, voluntary sector) who can then deliver own applications or services on the cloud platform Monitoring and billing allows government agencies to track and bill citizens or 3 rd parties including the monetization of APIs and common services

2 Governance ensures data services, new processes and APIs comply with government laws, policies and procedures Data services enable queries, composite queries, nested queries and full data sets to be published using open APIs. Update transactions are also supported. Data sources supported include relational databases (eg Oracle, DB2), web services, comma separated files, Excel files, Google spreadsheets, LDAP, ODS, RDF, Hadoop, Cassandra and MongoDB. Supported secure API interfaces are WS-* style web services (SOAP) or an API (REST). Supported data formats are XML and JSON. Drivers are supported for multiple languages/environments including Java, Javascript,.NET, Perl, PHP, Ruby and Scala. APIs supported for SOAP/XML and REST/JSON. Many standards are supported including WSDL, MTOM, XOP, SOAP with Attachments, WS-Trust, WSSecureConnection, WSSecurityPolicy, WS-ReliableMessaging, WS-Policy, WS-BPEL and WS-HT. Data transformations are supported using XSLT and XQuery. An ETL process is supported to load data from external data sources into the embedded MongoDB NoSQL database. This stores data in a native JSON format and enables rapid integration and aggregation of the data. The embedded MongoDB 2.6 database supports a NoSQL model to deliver highly flexible, high performance and scalable data management platform in support of agile service delivery, mobile application delivery and web application delivery. Key capabilities include: Flexible schema a document oriented model makes it easy to store new content without altering the schema of the database Data as documents the document model can store rich pre-aggregated cubes and joined data (from multiple data sources) supporting high performance and fast response times Scale-out storage and processing architecture the database scales out for query/reads and updates/writes processing; and provides 24x7 uptime leveraging cloud platform Memory mapped storage engine keeps active data sets in memory leaving rest of the data on cloud storage until it is accessed; delivers fast response times for dashboards and drill-downs In-database aggregation use of aggregation framework and map-reduce enables dynamic rollups and aggregations of the data sets on demand Full-Text Search - integrated full-text search will allow users to navigate content with Googlelike syntax Atomic updates new content can be added to the database in real-time without requiring schema changes Large object and rich media storage large objects can be stored such as audio, video, images and large documents in the same database including workflow processing for content management processes The Data Services solution can accomplish a fold performance improvement and a 90% reduction in disk storage requirement in MySQL and MongoDB databases. This is accomplished using the Tokutek

3 Fractal Tree indexing technology. TokuMX is an open source performance engine for MongoDB that make MongoDB more performant in large application with demanding requirements. In addition to replacing B-tree indexing with more modern technology, TokuMX adds transaction support, documentlevel locking for concurrent writes, and replication. TokuMX vastly improves the efficiency of the storage engine within MongoDB by replacing traditional B- tree indexing with patented Fractal Tree indexing. B-tree indexing, which was invented in 1972, has been used for databases and file systems for decades but was not designed to meet the Big Data needs of the 21 st century. Key benefits of TokuMX include: MongoDB Performance With significant improvement in insertions and indexing, TokuMX delivers faster, more complex ad hoc queries in live production systems without rewriting or tuning applications even when tables are too large for memory. MongoDB Compression: By leveraging write-optimized compression, TokuMX achieves up to a 90% reduction in HDD and flash storage requirements without impacting performance, and reduces the number servers needed to host a MongoDB database. MongoDB Transactions: TokuMX adds transactions with MVCC and ACID reliability to any MongoDB database making it possible to take advantage of the leading NoSQL database in a wider range of applications The benefits for MySQL databases are: MySQL Performance: with a 10x or more improvement in insertions and indexing, TokuDB delivers faster, more complex ad hoc queries in live production systems without rewriting or tuning applications. Offering high MySQL performance even when tables are too large for memory, MySQL Compression, MariaDB Compression: By leveraging write-optimized compression, TokuDB achieves up to a 90% reduction in HDD and flash storage requirements, without impacting performance. Agility: Hot Indexing enables ad hoc queries to run fast with realtime, optimized index support. Hot Column Addition/Deletion/Rename gives database administrators the freedom to rapidly define and add new fields. Additional Benefits: Providing immunity to database aging by eliminating the need to rebuild indexes, TokuDB ensures no more query slowdowns, no more dump/reload, and no more need for dedicated maintenance windows. In addition, by utilizing larger, less frequent I/O, TokuDB reduces wear for flash. Finally, with high insertion rates, TokuDB addresses the common and persistent problem of slave lag in which a replication server is unable to keep up with the query load borne by the master server. Advanced Tools: Tools such as Hot Backup allow a backup to be completed while database is running

4 Content Management Application Framework Data Services as a Service includes a leading content management and workflow system that is based on open source technologies including Apache, PHP and Django. This has quickly become one of the leading CMS in the fashion industry. We have now adapted the solution for the G-Cloud and focus to bring solutions to the UK Public Sector. The CMS solution is based on open source software and has following out of box capabilities to: Add cases Edit cases Delete cases Associate documents and images under case Archive Preview Package/Approve Calendar API The application is easy to use following a web design approach similar to Facebook, Tumblr etc. This encourages low cost adoption by end-users since there is no need for extensive traning. The technology foundation is Python (Django), JQuery, CSS, MySQL/MongoDB and Ubuntu Linux.

5 Open Data Portal based on CKAS and OrientDB multi-model database. OrientDB is an Open Source NoSQL DBMS with the features of both Document and Graph DBMSs. Written in Java, it is incredibly fast: it can store up to 150,000 records per second on common hardware. Even for a Document based database, the relationships are managed as in Graph Databases with direct connections among records. You can traverse parts of or entire trees and graphs of records in a few milliseconds. Supports schema-less, schema-full and schema-mixed modes. Has a strong security profiling system based on user and roles and supports SQL amongst the query languages. Thanks to the SQL layer, it s straightforward to use for those skilled in the relational database world OrientDB has been designed to be very fast. It inherits the best features and concepts from Document, Graph and Object Databases. Furthermore it uses the own MVRB-Tree algorithm as a mix of Red-Black Tree and B+Tree. MVRB-Tree consumes about half memory of the Red-Black Tree implementation maintaining the original speed while it balances the tree on insertion/update. Furthermore the MVRB- Tree allows fast retrieving and storing of nodes in persistent way. The most common reason applications scale up badly is, very often, the database. The database is the bottleneck of most applications. OrientDB scales up very well on a single machine. A single OrientDB server does the work of about 125 servers running MySQL or Oracle. The transactional engine can run in distributed systems supporting up to 302,231,454,903,657 billion (2^78) records for the maximum capacity of 19,807,040,628,566,084 Terabytes of data distributed on multiple disks in multiple nodes. OrientDB can embed documents like any other Document Database, but can also connect documents like a Relational Database. The main difference is that OrientDB doesn t use the costly JOIN, but rather uses direct, super-fast links taken from the Graph Database world. Supports the TinkerPop Graph Database interface and RDF using SPARQL query language and OpenRDF Sesame for Linked Datasets CKAN is a powerful data management system that makes data accessible by providing tools to streamline publishing, sharing, finding and using data. CKAN is aimed at data publishers (national and regional governments, companies and organizations) wanting to make their data open and available. - CKAN allows non technical uses to publish and find datasets through a web interface. The raw data and metadata is stored under the portal framework and can be visualised with interactive tables, graphs and maps. CKAN provides a rich search experience which allows for quick Google-style keyword search as well as faceting by tags and browsing between related datasets. Users can quickly see what datasets are available, in which formats and with which licence, straight from the search results. All dataset fields are searchable. CKAN can be customised and extended using APIs and extensions. Data can be imported through web interface wizards, JSON APIs and through spreadsheet uploads.

6 Features of CKAN: Complete catalog system with easy to use web interface and a powerful API Strong integration with third-party CMS s like Drupal and WordPress Data visualization and analytics Workflow support lets departments or groups manage their own data publishing Fine-grained access control Integrated data storage and full data API Federated structure: easily set up new instances with common search Associative Database Option AtomicDB AtomicDB delivers an order of magnitude improvement in the integration of applications and data. Allowing all databases, applications and external information to be rapidly integrated and presented through a centralized dashboard and through open APIs. The power behind the AtomicDB patented core technology comes from a long term project with the US Defense Department (DOD). Data is stored in a structure-less paradigm. Data items are stored much like grains of sand on a beach independent of how the items are inter-related or associated. The result is an agile platform for information management and information integration. As an example of the power of the associative data model, the AtomicDB technology supported a major Genomics Research project which combined 4 public genome databases with 2 proprietary databases into a single data warehouse using associative database technology. With AtomicDB there are no more limitations from the relational two dimension table model or many derived OLAP and SQL data management technologies. Integration that took months or years can be done with AtomicDB in days or weeks. And the agile approach makes change management and future expansion an immediate supported task. The underlying AtomicDB storage engine has patent pending technology for compression, encryption and obfuscation. The technology enables very high security and massive scalability to Exabyte levels. This is possible with commodity hardware and cloud platforms not requiring expensive proprietary appliances. Parallel processing and in-memory associative processing enables complex queries over large secure data sets up to two orders or magnitudes faster than alternatives. The AtomicDB associative database technology is based upon associations in an atomic environment, mimicking the brain in storing and retrieving information. Instead of storing data in cells, It uses information atoms within their full context, which know where and how they fit in the system. This allows for very flexible and dynamic information systems, where data can easily be added to the system and represented in a new way without disrupting the existing structure of the information. The system is powerful enough to map the human genome. AtomicDB is highly secure. Developed for the US Department of Defense (DOD) the core technology in our solution has some unique patented security capabilities. The added value security is exposed in three ways the storage of items of data on the disk, the obfuscation of data and the security features of the API for delivering secure data to applications and external services. This is of particular concern in

7 industries like healthcare, financial services and government where there is a need to protect patient, citizen and customer data. Data is unbundled, compressed and obfuscated at the storage layer and only accessible with HASP keys and four 128 bit tokens. With the AtomicDB agile associative model it is possible to build and deploy a working data warehouse in less than a week! It is possible to reduce project costs from millions of pounds to only a few hundred thousand. It will reduce and mitigate business risks by delivering on-time and under budget. There are no expensive proprietary hardware appliances topurchase, install, configure and maintain. The solution is provisioned for you in a secure and highly efficient federation of on-premise data sources with availability to cloud storage if applicable or necessary. There is no need to hire expensive consultants or teams of staff to manage, as it is delivered as a managed service. The platform follows agile software development and agile data warehouse techniques for rapid and incremental delivery of business functionality. This is possible because AtomicDB is driven in a better way that stores and retrieves information like the human brain. The brain stores each atom of information in a network of associated concepts. When you recall something you know, you do it by association not by writing a SQL or MDX query. The associative data warehouse can reduce or eliminate many of the steps connected with building a traditional data warehouse or business intelligence (BI) project. The data model is greatly simplified because it is based on a logical business information model through concept maps. Thus, not requiring expensive resources to build relational entity-relationships and dimensional models. ETL processes are greatly simplified. Importantly, the AtomicDB solution can easily accommodate future changes without having to rework the data models. The solution is open to any BI tool or application development domain. Open data can be published using the embedded API Manager. A centralized dashboard is built so that it allows the user to view all data in real-time. In summary, the AtomicDB agile associative approach means that development cycles and costs can be cut by up to 80%. Costs can be published in advance, allowing projects to keep within budget and delivered ahead of schedule. The benefits include: 1) Lower Cost of Implementation 2) Lower Cost of Maintenance/Support 3) Agile Information Model 4) Open API 5) Rich Dashboards and Applications 6) Integrates with Existing and Future Applications 7) Highly Secure

8 Deployment Models How can the service be deployed? By partnering with Memset Ltd we offer the Data Services service on a secure private cloud IL2 accredited for G-Cloud (IL3 optional). On the secure private cloud deployment each client (organisation) deployment will have a dedicated VM (container) and a dedicated database. A dedicated secure private container with multi-tenant service can be provided for a single organization. Networks to which the service is connected (directly)? The Data Services service is accessible from a browser with an internet SSL connection. Other networks (PSN, N3) can be enabled by working (for optional cost) with our partner Memset Ltd. API access available documented and supported? SOAP and REST APIs are available. Open standards documented and supported? Data interchange is supported through XML and JSON formats. Data Services can be published as REST and SOAP services supporting a number of WS-* standards. Open source used and documented? Function/Purpose Operating System Application Server NoSQL Database ESB Java Libraries Data Services Open Data Portal Content Management Business Process Open Source Components Ubuntu Linux WS02 Application Server (Apache) MongoDB, MySQL, TokuDB, OrientDB WSO2 Enterprise Service Bus ExtJS, JQuery, D3.JS WSO2 Data Services Server CKAN Django CMS, Apache, PHP Activiti BPM (optional) Service Management Technical boundaries/interfaces of the service documented? The Data Services service interfaces are fully documented for both the role of administrators and endusers.

9 Services available to other suppliers so they can use them to provide services to government? The Data Services service is available to any other suppliers who wish to leverage the G-Cloud framework. On-boarding process e.g moving to the service? An administrator account is created for the organization. The administrator can then create end user accounts. Additionally the on-boarding is provided as a managed service. The SLA for on-boarding is 24 hours. Off-boarding process e.g moving off the service? The administrator can remove the end user accounts and the data. Additionally the off-boarding is provided as a managed service. The SLA for off-boarding is 72 hours. Data extraction/removal criteria met? The criteria are met in accordance with the G-Cloud ITT glossary definition. Data processing and storage locations defined? For the G-Cloud the IaaS option available is Memset (located in Guilford, UK). For an IL2 compliant and UK sovereign G-Cloud solution the preferred approach is to use Memset IaaS. IL3 is available as an option. Data location option can be defined by user? This is not possible. The organization purchasing through G-Cloud would only have the option of the Memset cloud which is managed in a UK sovereign Tier-3 data centre. Data centre tier? Tier-3 UK data centre through our partner Memset Ltd. Support boundaries/interfaces documented? This is defined in the support guide. Service roadmap provided? The next release of the Data Services service will include: CESG IL3 accreditation

10 PSN connectivity Advanced visualizations for end-users based on the D3.JS data driven documents. Additional roadmap from OrientDB, Memset, Tokutek, WSO2 and MongoDB open source projects AtomicDB 3.2 Performance attributes defined and documented? Performance uptime and response time are documented in the support guide. Database and VM servers are constantly monitored. Backup & Disaster Recovery? The instance is backed up with nightly snapshots. Databases are constantly monitored. Is a support service provided and documented? The support service is documented in the support guide. Real-Time management information available? Management information is provided in a monthly report. Real-time monitoring of the data services server and the NoSQL database is provided through API and web pages. Self service provisioning/de-provisioning? This is partially supported. The service itself is provisioned by vendor. But the end users can be added and removed by the client s administrator. Time for provisioning/de-provisioning documented? 24 hours for on-boarding and 72 hours for off-boarding. Administrators can add end-users in less than a minute. Service desk can be used by 3 rd party suppliers? The service can be used by any 3 rd party suppliers in the G-Cloud framework. Commercial Unit based pricing model? Detailed in the pricing description. Aggregated billing? Billing options can be provided for organizations with multiple departments.

11 Minimum Contract/Billing Period? 12 months. Free Option? A free option is not standard offering. Trial Option? A trial option can be provided on public cloud (shared container, multi-tenant). Termination Costs? If terminated within the 12 month term then the remainder of the annual fees will be billed. If terminated at the end of the term then there is no cost. Trial subscriptions can t be terminated. Supplier contract terms/jurisdiction? Cloud Intelligence Services Ltd and Amtex Solutions Ltd are located in the UK under jurisdiction of English law. The IaaS provider is also located in England. Payment Options? Purchase Order and BACS are supported Data Visualization Authorized users may store datasets? This is core functionality of Data Services. End users can upload their datasets to the cloud service into a embedded NoSQL database. Or alternatively the access is provided to databases such as MySQL, Oracle, SQL Server, Spreadsheets, Files etc. Authorised users may make datasets public? This functionality can be enabled using API Manager publishing data as an API (SOAP/XML or REST/JSON) for the query and export of data sets Part of an integrated collaborative tool set?

12 The service is not part of a collaboration tool although API can be integrated into other collaborative tools or workflow processes. Additional support for WS-BPEL and WS-HT can be provided to support collaborative workflow processes. Publish datasets as visualizations? This is core functionality End users may search for datasets? Search is native to MongoDB and OrientDB. Types of display? Some advanced visualization documents include box plots, bubble charts, bullet charts, calendar view, cartogram, chord diagram, dendrogram, force directed graph, circle packing, population pyramid, stacked bars, steamgraph, sunburst, node-link tree, treemap, voronoi diagram and tag clouds. Maximum dataset size? The standard embedded data storage for the service is 10TB although federated access is available to any data set. Specialist Cloud Services - Generic Do you provide vendor specific services? Services are provided for the implementation of the Data service and connectivity. If the vendor(s) have accreditation are you accredited? UK G-Cloud IaaS partner provider Memset Ltd has a number of accreditations and certifications e.g ISO 9001, ISO 14001, ISO 27001, PGA IL2 and PGA IL3. Amtex Systems has ISO 9001 certificate for IT enabled services, business process outsourcing, software testing, data integration and reporting. Other certificates include PCI-DSS and CMMI Level 3. Amtex is presently undertaking global certification for ISO Vendor accreditation? We will submit the Data Services Software as a Service for PGA IL3 accreditation in 2014.

13 Service Levels Platform IaaS provider partner Memset Ltd has a 99.95% uptime guarantee for networks, servers and storage. 24x7 support is provided for the IaaS and this support is wrapped into the Data Services support levels. For the Data Services and API Management service production support is provided 24x7x365. Incidents are submitted via and/or telephone. The target response and resolution times for incidents are: Severity Level Response Time Resolution Time 1 1 hour 24 hours 2 4 hours 48 hours 3 8 hours 72 hours 4 24 hours None The severity levels are defined based on impact on a production usage of the Data Services service in the table below: Production is defined as one by end users where a failure of the Data Services service in production will have immediate economic impact on the client s business operations Severity Level Definition 1 Client s business operations have been disrupted either through a loss of data or a loss of service. 2 Client s business operations can continue in a restricted fashion. Client s business operations are at risk. 3 Some aspects of the service are impaired but client can continue to use the service. Client s business operations are at minimal risk. 4 General usage questions, requests for new features or documentation issues.

14 Termination Options Clients may terminate the production Data Services service with 30 days notice for any reason, subject to full payment of remaining SaaS fees for the 12 month subscription period. Trial service cannot be terminated. Client will be invoiced the full amount for the 3 month trial in advance. Vendor can terminate the production Data Services service at the end of the initial 12 month subscription period with 90 days notice for any reason. Government Open Standards, Information Principles for UK Public Sector and Government ICT Strategy The Service is conforming to the published strategy documents and policies. Data Restoration/Service Migration Data restoration is provided by vendor for recovering from backups. Daily snapshots are taken with options to restore from the automated backups. Service migration can be provided as a service through an export of the database environment or a migration of the complete VM onto another cloud instance. Customer Responsibilities The client responsibilities are set out in the support guide. More Information More information on the service can be found at the cloudstore or please contact: Mr. David Watson Managing Director dave.watson@bigdatagov.org Cloud Intelligence Services Ltd, St John Street, London EC1V 4PY Telephone