SUBMISSION CASE STUDY ROUND 3 CHALLENGE 1 DIGITAL VIDEO & IMAGE LIBRARY PUBLIC IMAGE & VIDEO LIBRARY Public - Private Partners
EXECUTIVE SUMMARY Challenge one of round three in the Digital Canberra Challenge was a perfect fit for Link Digital and our extended team of ACT Government collaborators. Facilitated by the egov Cluster, NICTA, the collective team has created a unique media management and sharing platform based on the world s best open data and public cloud technology. The Public Video and Image Library that has been established allows Canberra to bring together the video and image assets of Government, local organisations and the general public in a visual and machine readable platform, providing multiple integration points for further innovation. By selecting open source and community supported tools we ve ensured the platform will remain sustainable and relevant over the long term. We ve even written the product into our business plan and started working on creating an international market for its use. The ACT Government can now assess our Proof of Concept as evidence of a viable production system. Whether selected as winners of round three or not Link Digital is now committed to commercialising the platform via an international open source distribution model that includes options for paid support and managed hosting. The following case study can be read as a journal entry as we take stock on what has already been accomplished. The team is already prepared for the next chapter and is looking forward to a great journey ahead. Hoots! Steven De Costa
SECTIONS What have we built? Take a look! Why is this awesome? Are you with us? 01 02 11 19
WHAT HAVE WE BUILT? Via the Digital Canberra Challenge process administered by the egov cluster, NICTA, Link Digital has established a whole of government image and video library which can be used not only by ACT Government employees but the private sector and the public. The image & video library ensures consistency across the service & reduces costs for a number of items including data storage, ability to browse and share, licensing and talent arrangements, copy right and acknowledgment, archiving and staff resources. Our open data platform is based on CKAN, which is used by Governments worldwide for their open data platforms and has an active community of developers and users. Images and videos available within the platform are made available as public goods by default, but can be restricted via licence conditions or authority to access as required. The platform provides intrinsic benefits via its open access to the community. The API additionally allows for innovative commercial and public goods to be developed that will benefit multiple sectors within the ACT economy. By having core functionality centralised around low cost media storage, attribution, governance and access services, the ACT Government investment is minimised. The platform is feature rich in the areas of media storage, management and access. We have also demonstrated integration points with social feeds taken from Flickr and Twitter. Working against ACT Government requirements the platform provides demonstrated workflows for media asset collection and management that can be made available for all of the ACT directorates. Via our links within the CKAN Association and international open data platform owners we are establishing a cohort of users and contributors for the project. Ongoing development of the software and architecture will be run as an open source project, aligned with CKAN itself. Technically, we ve built our platform on an enterprise hosting architecture which provides compliance against standard Government requirements surrounding backup, disaster recovery, security, high availability and performance under high load. All of this has been provisioned on low cost public cloud services hosted within Australia. 1
TAKE A LOOK! Digital First Media Platform (DFMP) The DFMP is powered by CKAN, an open source data platform that is capable of handling all types of data. CKAN is the platform that also powers the open data intiatives for the Australian Federal, NSW, South Australia and Queensland governments. DATA FIRST MEDIA PLATFORM 2
Data organisation It provides tools and frameworks for organising data and datasets in groups and organisations, and accessing via the web interface or through dynamically-generated machine-readable API points. There are also open source extensions for visualising data and mapping geospatial information. DATA FIRST MEDIA PLATFORM 3
The DFMP will host mainly two types of data: photos and videos. Photos and videos can be brought into the DFMP through several ways: Bulk import of files directly into the cloud Uploading of files by registered DFMP users Uploading of files through the general public portal on the CBR website Automatic harvesting of images and videos attached to tweets and from Flickr DATA FIRST MEDIA PLATFORM 4
Users can experience the DFMP and the PIVL Whilst users can access the CKAN component of the DFMP we have provided a user friendly interface built in Drupal for the majority of people who are mainly interested in browsing and using photos and videos without adopting a data-first view. We call this user friendly interface the Public Image and Video Library (PIVL). DATA FIRST MEDIA PLATFORM 5
The Public Image and Video Library (PIVL) The home page for the PIVL allows users to search and filter the collection of photos and videos or browse by organisation, group or dataset. PUBLIC IMAGE & VIDEO LIBRARY 6
Uploading images and videos to PIVL Members of the public can register as users and upload and maintain their own collections or datasets of images and videos. They can choose to share those assets under a Creative Commons license. Organisations can also upload and maintain their own collections of images and videos with various roles available for member users to create, edit or remove resources. Licensing can be set at the dataset level and inherited by all resources under it for consistency and ease of usage rights control. Assets can also be restricted so they are not available to the public. PUBLIC IMAGE & VIDEO LIBRARY 7
Displaying the datasets Administrators can view and approve or deny requests from users to be granted access to restricted repositories of images and videos although any user with a valid government email address will automatically have access to their government s private resources for use in marketing collateral, websites and campaigns. Site curators can change the display of groups, organisations or datasets featured on the home page in response to current events, trending locally-relevant tags, newly-imported bulk collections of assets and government agencies who have recently come on-board onto the DFMP. Geospatial data visualisation capabilities in CKAN allow people to view maps of harvested tweets (such as those tagged with #CBR) and geotagged photos on a map of Canberra. PUBLIC IMAGE & VIDEO LIBRARY 8
Searching through PIVL Images and videos can be searched for by title and description, tags and date created. With the inclusion of historical imagery of Canberra in the DFMP we will roll out functionality for people to browse and search photos and videos from specific significant events, festivals, and from eras in the city s history. Groups allow people to curate resources from different datasets and organisations into topics that may include Art & Culture or Live Music, Tourism, Entertainment, Northside and Southside, Environment. Perhaps even collections of images across time that show the transformation of places like Civic and now Braddon, or the National Arboretum. PUBLIC IMAGE & VIDEO LIBRARY 9
Exploring between DFMP and PIVL We provide links between the CKAN component of the platform and the Drupal user-friendly interface so that users can move between the two components of the site, perhaps to access metadata and machine-readable data behind datasets and resources or to see a nice image gallery view for datasets located through CKAN. PUBLIC IMAGE & VIDEO LIBRARY 10
WHY IS THIS AWESOME? THE VALUE CREATED FOR ACT GOVERNMENT Previous situation: Image and video assets were held on various attached and unattached hard drives, unorganised. Requests for media arrive in various ways and were fulfilled in an ad hoc manner, often involving physical distribution via USB hard drives and sticks to various locations. Tracking and cataloguing of media assets is fragmented, typically done within various agency or programme silos within ACT Government. With our solution: We have developed a whole of government image and video library which can be used not only by ACT Government employees but the private sector and the public. The image & video library ensures consistency across the service & reduces costs for a number of items including data storage, ability to browse and share, licensing and talent arrangements, copy right and acknowledgment, archiving and staff resources. Our open data platform creates an economic good with the dual benefit of unlimited multi-tenant users and n+1 reuse of media items that might otherwise be created for a single primary use. Providing universal access to this media content has created the opportunity for private-public goods to be created within secondary markets to grow the economic value of the platform. That results in higher value public goods, but it also provides the platform for innovative commercial goods that can benefit multiple sectors within the ACT economy. By having core functionality centralised around low cost media storage, attribution, governance and access services, an ACT Government investment in the platform will target only the essential parts of service delivery. An open data approach provides for a core platform around which innovative people and organisations can attach additional services to meet specific needs over the long term. Via our links within the CKAN Association the ongoing development of the software and architecture will be run as an open source project, aligned with CKAN itself. The association will run an expression of interest globally to attract further development funding from the CKAN installation base. It is worth noting how large this install base now is, and how rapidly it is growing. A short list of CKAN sites from around the world can be found at http://ckan.org/instances/ We have worked with ACT Government to understand their stated requirements and have considered this within the design of our proof of concept platform. A comprehensive listing of enhancements required to make the platform production ready is known and ready for implementation. 11
UNIQUENESS OF VISION The use cases of an open data platform for Government service delivery are being developed worldwide. Our approach is unique in that it has been developed from the ground up as a data first media platform that will service a number of broad use cases related to public image and video collections. The requirement of digital asset management within Government is universal. Our approach has aligned this with the universal drivers for open Government, open source technology and open access. We have designed a platform that can be used by communities freely to establish their own public image and video collections. The technology choice ensures wide adoption within technical communities as well. This will facilitate longevity and enhancement of the project via its links into both the CKAN project and the Drupal project. The cloud platform chosen has allowed for seamless integration between the application layer and the hosting infrastructure to provide a range of additional benefits. Two key benefits are the ability to transcode video on demand and the ability to securely share restricted media assets. By reinventing the underlying platform of what is essentially a digital asset management system we have unlocked the potential for a vast array of opportunities to enhance the collection, storage and management of the ACT Government s existing Video and Image collections. By designing a platform that is open by default we have removed artificial barriers which other media management systems have with regard to asset collection and access. The full spectrum of Canberra s visual identity can now be captured and shared worldwide. 12
FUNCTIONALITIES / FEATURES The following system components were integrated as the Public Image and Video Library. 1. Media asset cataloguing 1. CKAN hosted on AWS 2. Themed to represent CBR branding with the working title of DFMP 3. Updated to respect minimal required metadata 4. Update to preview data resources of the type image and video 2. Media asset storage 1. Enterprise AWS architecture for public cloud platform 2. Assets stored in standard S3 bucket 3. A class of assets stored in Glacier, lower cost archival storage 4. Assets transcoded to allow degraded preview in catalogue 5. Assets transcoded to suite reuse for digital 6. Assets available for request at highest quality - print and broadcast quality is considered maximum quality level 7. Security model for storage and distribution (logging and audit controls) 8. Bulk ingest of Centenary images 3. Media asset access 1. Drupal site hosted on AWS 2. Public access for web based upload of files into public datasets 3. Authenticated access for web based upload of files into restricted upload datasets 4. Automated ingest of assets from public feeds into specified datasets (ie, Flickr and Twitter) 5. Automated tagging and classification of media, where possible 6. Assisted tagging and classification of media assets via web form 7. Faceted search for public media assets 8. Global search for media assets 9. Serendipitous discovery of media assets 10. Brand Canberra theme for web interfaces Use cases demonstrated include: 1. Those who tweet via hashtag #CBR 2. Those who post to a Flickr Group 3. Those in ACT Tourism who wish to store large video assets 4. Those in Brand Canberra who wish to release an image collection from the Centenary year 5. A local photographer who wishes to make images available 6. A local non-government organisation who wish to store image assets 7. A Community collaboration project that wishes to source images from the general public 13
OUR APPLICATION OF TECHNOLOGY The core technical components of our platform are: 1. CKAN the world s leading open data catalogue software used by data.gov.au, data.vic.gov.au, data.sa.gov.au, data.nsw.gov.au (all clients of Link Digital) and many others around Australia and the world. 2. Amazon Web Services (AWS) specifically for S3 storage of media assets, but also for the use of Glacier, which is a very low cost solution for archived media. We also use AWS for the web application hosting.. We have also leveraged video transcoding and playback technology. 3. Drupal this was used to provide user registration and form submission with a user experience suitable for the end users. It should be noted that API access via CKAN means that anyone can create a new approach, but we demonstrate this as part of the initial build via Drupal. Drupal is also used to provide a visual search and Apache Solr based faceted search capability for the content that is stored within the CKAN repository. All software used is open source. The only direct cost of the platform relates to AWS hosting fees. We expect that an ACT Government deployment of the platform would require some management service to be offered to users within Government, which can be arranged via a reasonably priced managed service agreement with Link Digital. As the repository will be 100% available via API for both read and write access, the use cases supported by Drupal can be improved upon or recreated via other approaches led by community efforts. Integration with digital asset management systems and other third party platforms can be achieved with minimal effort. The platform has been loaded with over 100,000 assets for our proof of concept and can be ready to launch publicly within a month of contracts being established. 14
SOME AMAZING THINGS ARE NOW POSSIBLE Link has investigated integration with machine learning systems to help increase the amount of metadata that can be captured without human input. Via NICTA we gained access to research being undertaken by Stephen Gould on scene understanding. Through his research and existing work, where pixels labelled with a scene s semantics and geometry can allow computers describe what they see, we are looking at integrating an algorithm that will detect Honda CBR motorcycles attached to tweets with the hashtag #CBR. Such tweets will be tagged and not displayed in the dataset being created in real time from the #CBR twitter hashtag. Our platform already extracts exif data from images and stores these in the media file datasets. This is done in a flexible manner, as used by Google Analytics, which retains data integrity of the dataset while allowing for a variety of field label and value pairs. Existing exif data can be used in many ways. Some of the possibilities include: 1. Creating image timelines base on the actual date the photo was taken rather than the date it was entered into the platform 2. Suggesting collections of images based on author/creator independent of the dataset they were entered into 3. Providing photography students with a rich trove of information about the camera type and settings from which they can understand how images were originally captured 4. Removing the need for human curation of images commissioned by ACT Government by automatically identifying commissioned photography 5. Using a myriad of data visualisation techniques to aid image discovery for end users including geospatial mapping With the open platform architecture, possibilities are now only bound by our imagination. 15
OUR ABILITY TO EXECUTE ON THE PROJECT S VISION Link Digital is an established digital agency, located in Canberra, Australia. With over 14 years in the industry it has built a reputation within the Government and commercial sectors as a high quality supplier of online design, development, hosting and consulting services. Over the last three years, Link s investments within the technology segments of public cloud and open data services has seen it take on the primary open data portals for the Federal, NSW, Victorian and South Australian Governments. By building it s competency in cloud and open data it has built on an existing knowledge base that is underpinned by a dedication and drive to bring leading edge digital services to end-users. Link Digital s Executive Director, Steven De Costa is an established open data and open knowledge community contributor. In addition to his role at Link he is on the Board of Open Knowledge Australia, member of the Steering Group and Community Team Lead for the CKAN Association, an organizer of GovHack.org and founder of DataShades.com. As an Advanced Consulting Partner of Amazon Web Services (AWS) Link Digital is one of just three founding partners of the AWS worldwide open data program. The ACT Government has established clear expectations for a production platform and developed a set of documented internal requirements. While our submission is the result of a proof of concept developed for the Digital Canberra Challenge the entire team has worked collaboratively with the goal of a production platform being commissioned. The synergy evident via the public-private partnership process facilitated by the egov Cluster has established mutual understanding and trust within the team and all parties have demonstrated their commitment and willingness to complete the project. While commercial and service level agreements are yet to be negotiated, all parties are working without reservation to bring the project forward beyond the Digital Canberra Challenge and go live. 16
WE ARE TAKING THIS ON THE ROAD ALREADY After presenting on the economics of open data at the Kiev International Open Data Conference recently, Steven De Costa was asked if he had any examples of private-public partnerships involving open data. The response provided is available on YouTube here: https://m.youtube.com/watch?feature=youtube_gdata_player&v=q0mkbis90la TRANSCRIPT: The Australian Capital Territory Government, where Canberra is, they ve rebranded. They re calling themselves CBR, which is the name of the airport. What they are trying to do is attract all of the sentiment of what the city is into a single platform. So they launched a challenge, like a hackathon, but it is a specific challenge. Who could build them a public video and image library? So we put forward against that challenge our submission to build a Data First Media Platform (DFMP). All that is, is it s an instance of CKAN that has been modified slightly so that it can correctly catalogue image repositories. We then built a couple of innovative integrations with that. That s going to be adopted by whole of ACT Government as a whole platform for all government agencies; for tourism, for international relations, for local events. Because we have a lot of events in Canberra and photographers will take pictures of them. But we ll also take photography images and library footage from community organisations. INTERNATIONAL OPEN DATA CONFERENCE UKRAINE That thing, the DFMP, is going to be turned into a product. We re going to take their money as seed capital to build an international product for the DFMP. So I m actually here as part of a roadmap for launching that. I m meeting with some people in London the day after tomorrow to get more funding for it. And that for me is a great example, because it s very personal, of creating a new product based on a privatepublic partnership. 17
TAKING A CLOSER LOOK To take a look at what a general user can experience of the Public Video and Image library, go here: To take look at the CKAN platform which drives everything (working title is the Data First Media Platform, or DFMP) go here: http://web.actgdfmp.links.com.au/ http://web.actgdfmp.links.com.au/data/ Want to see the gallery of #CBR tweeted images? Go here: http://web.actgdfmp.links.com.au/gallery/cbr Want to see a gallery of #WeAreCKAN tweeted images? Go here: http://web.actgdfmp.links.com.au/gallery/weareckan Want to see a gallery of images sourced via Flickr? Go here: http://web.actgdfmp.links.com.au/gallery/pool_abccanberra 18
ARE YOU WITH US? Link Digital has taken the opportunity to work with the ACT Government via an innovative challenge process and invested heavily in the creation a world s first open data, image and video management system. The product is ready for worldwide open source distribution with commercialisation opportunities available via a managed platform as a service model under our international brand Data Shades http://www.datashades.com. We have identified additional funding options and aim to undertake a worldwide launch of the platform on the 27th of May 2015 in Ottawa, Canada. This will be on the eve of the 3rd International Open Data Conference. We would like the ACT Government to be our first customer as we bring our platform to market. Moving forward with a production release will increase the value of the Government s existing image and video assets while also reducing it s existing management and distribution costs via increased workflow and publishing efficiencies. The open data architecture will create further value via over time, courtesy of innovation from within the existing open source CKAN community. We hope the ACT Government will continue this journey with us beyond the Digital Canberra Challenge. 19
PUBLIC IMAGE & VIDEO LIBRARY