SC News Issue 08. Contents. What's New? 29 September What's New? Research highlights Did you know Partner news Our people News & Events

Transcription

1 SC News Issue September 2015 Contents What's New? Research highlights Did you know Partner news Our people News & Events What's New? New Visual Analytics Capability in Scientific Computing The Scientific Computing Visualisation Service offers a range of advanced capabilities including: Adding new or improved visualisation algorithms to existing applications Improving visualisation performance through the use of dedicated hardware Integrating visualisation into existing Scientific Computing workflows Customised dataset visualisation Applying hardware and software tools to allow true stereoscopic visualisation Facilitating remote or distributed visualisation Designing 3D models and animation content for improved outreach Accessing external visualisation facilities Now a new Visual Analytics capability has been added to this list as well. Visual analytics enables researchers to gain greater understanding and insight into complex scientific datasets by: Developing interactive analytical capabilities and tools based on various techniques including data transformations and representations

2 Data mining Statistical analysis Pattern recognition Like most Scientific Computing Services, Visualisation capabilities are generally accessed by submitting an eresearch Collaboration Project proposal. While the call for new projects submissions is currently conducted twice a year - June and December - submissions are welcome at any time. If you would like more information about Visual Analytics, Visualisation in general or any other Scientific Computing service, please contact schelp@csiro.au Galaxy Call for Early Adopters Galaxy is a web-based open source workflow tool that allows users to run the latest bioinformatics algorithms and create workflows without the need to be familiar with the command line. The Galaxy tools have been installed onto new system infrastructure and are currently being tested. There will be an early adopter period to allow users to test the new system before it becomes operational. Note that reference data will be available on the new system but working data from the existing service will not be available. If you are interested in trying out this new service and offering feedback, please schelp@csiro.au and let us know which Galaxy Tools and data are required by your workflow. Containerising and sharing applications via Docker Operating system-level virtualisation, commonly referred to as container-based virtualisation or containerisation, enables a virtualisation layer that runs as an application within an operating system (OS). In this approach, the kernel of an operating system runs multiple isolated guest virtual instances. Such guest instances are called containers. Container-based virtualisation allows users to run far more virtual instances (i.e. containers) than whole-os virtual machines on a host because with containerisation, there is only one operating system (and one kernel) shared by all containers, whereas traditional virtual machines are self-contained systems with their own operating system, virtualised hardware and allocated resources. In comparison, containers consist of only the resources (e.g. software stack) required to run the applications they are hosting, thus allowing much more efficient use of the underlying system resources. Different from traditional virtualisation, OS-level virtualisation usually imposes little to no overhead. This is because containerisation does not require any hypervisor or any intermediate whole-os virtual machine, and no virtualised hardware is required. Applications are deployed in their own containers within a single operating system, which offers close to bare metal performance. In addition, applications deployed via containers can start up very quickly on-demand, while traditional virtual machines require time to start the operating system before any applications can be run. Docker is an open source project that enables the deployment of applications via software containers. It provides a platform for users to package, run and manage an application and all its dependencies (such as codes, system tools and system libraries) in a software container that can run on any Linux server in a flexible and efficient manner. Applications distributed as Docker images incorporate all the required dependencies and configuration, and thus end-users do not need to be concerned with installing extra software packages and troubleshooting dependencies. In addition, Dockerised applications can be run on any Docker-enabled environment, including Windows, Mac and Linux, as well as many public cloud service providers. Docker is particularly well suited to scientific applications. Many scientists are publishing their application code and data but it is often difficult for others to run the code and reproduce the results. Docker enables scientists to create an exact compute environment for running applications because they can publish reproducible computational applications, usable by any researchers and on any Docker-supported infrastructure to produce the same output.

3 Some CSIRO researchers are packaging their applications in Docker containers so that their partners and collaborators can reproduce the research outputs in a reliable and transparent manner. There is also a growing demand from researchers who want to share their packaged containers with other researchers within CSIRO. The public Docker Hub repository allows users to publish and share their Docker images. However, in some cases, it is not appropriate for some of CSIRO s Docker images (for example, work-inprogress software prototypes that are not yet ready to be published and work that involves licensing). IM&T Scientific Computing has recently announced the availability of the CSIRO Docker Registry, which is available to all CSIRO staff and affiliates. This registry allows users to store and share Docker images, and is built on the Bowen Research Cloud infrastructure and supports CSIRO credentials based authentication and uses secure network protocol for all data transmission. More general information about the Docker Registry can be found at In addition, we are also working on a Docker-based setup for Jenkins, which is an extensible continuous integration server. Our current setup is based on the official Jenkins Docker image, with additional customisations, including support for Bowen cloud storage, secure network protocol and CSIRO credentials authentication. This Docker-based approach allows us to rapidly provision new Jenkins CI servers in the Bowen cloud. If you want to try out Jenkins, please schelp@csiro.au. For more information about Bowen resources, please visit Research highlights Spark Spark is a framework for the end-to-end processing, simulation and analysis of bushfires, providing a more accurate view of fire behaviour, informing decisions that could minimise property damage and save lives. The framework uses geospatial data, such as vegetation types, topography and un-burnable elements such as roads and bodies of water, as well as weather data such as air temperature, humidity, wind speed and wind direction. Spark then uses this data to model the spread of a fire based on its specific location and current environmental conditions. Users can design custom fire propagation models by building on Spark s computational fire propagation solver and incorporating various input, processing and visualisation components, each tailored for wildfire modelling. The framework can periodically suspend the simulation to allow analysis, visualisation or output of the simulation results. There are a number of built-in modules for data analysis ranging from inspection of point values, data plotting and statistical calculations over the domain to image analysis using OpenCV. This analysis can be performed on individual simulations or over ensemble sets of simulations. Both the propagation and the resulting analysis can be interactively visualised in a graphical user interface (GUI) using OpenLayers or a range of two and three dimensional display types. The framework is based on the CSIRO Workspace Platform. Workspace is a cross-platform application development framework, built on the QT toolkit that allows individual operations to be connected and managed in a workflow environment. Each modular component within Spark is implemented as a Workspace operation, allowing solver workflows to be rapidly constructed in a GUI environment. A range of open-source modules are included in Workspace, such as OpenCV, NetCDF, GDAL and Python which are fully compatible with the Spark framework. Complex user interfaces can also easily be built in Qt designer and used with Workspace workflows, such as the spark-gui demo application. Spark has been developed as an open framework for fire prediction and analysis, making it simple for fire agencies to incorporate the software into their existing systems and also more easily allow the fire science community to collaborate and keep adding new fire behaviour knowledge. A demonstration version of Spark is available to download via the CSIRO website.

4 Did you know R Shiny Server Pilot Interactive web presentation of data is increasingly common, but getting started and choosing and deploying software can be daunting. R Shiny and Shiny server provide a particularly easy framework to get started with web applications, allowing you to concentrate on scripting in R, where the web presentation and interaction is already done for you. A Shiny app provides a set of widgets to interact with the app, and display elements, for text and graphics that are generated in response to the interaction. The display elements are typically dynamically generated static graphs that can be from any of the diverse and powerful R extension packages. Optimized versions of R are available on SC clusters and Bowen cloud hosts, including the Shiny R package, enabling you to develop and debug Shiny applications independently. For example, you can connect to one of the pearcey interactive nodes (with a vnc session) create a new directory, copy ui.r and server.r from one of the shiny examples or the Shiny tutorial and run it interactively with R (a firefox browser will be launched to connect with your service so you can test the web interaction). When you are ready to deploy the shiny application for others to use from their web browsers, we have a pilot Shiny server instance for evaluation. This instance is running on SC Bowen cloud infrastructure with both externally accessible and internal only services. Interested users can see the examples that are distributed with shiny and there are more examples in the Shiny gallery and User Showcase. The Shiny server host can mount data on Bowen Cloud storage resources so you can bring CSIRO data to the server without having to make copies. To use shiny.csiro.au please contact schelp@csiro.au You will need to provide your R code (and potentially data) for use to deploy on the pilot service. In the future we hope to have a more streamlined approach for using Shiny server as an option for deploying web apps that have code and data published in the CSIRO Data Access Portal. We also intend to provide support for other web application frameworks (for python in particular). Feedback and/or requests to serve Shiny apps are welcome. Partner news CSIRO IM&T Scientific Computing - access to partner and national facilities As part of its portfolio of resource offerings, CSIRO IM&T provides access to various partner facilities, to meet specific needs. See our wiki for more informaon. Here is an update on some of the partner facilities and services. Pawsey Supercomputing Centre at CSIRO Kensington WA The Cray XC40 system magnus is in full production, and is supported by the SGI mass storage system with a cluster and largememory UV2000. As well, the Pawsey Centre hosts the galaxy Cray system for the radio-astronomy community, an Research Data Service node and a NeCTAR cloud. National Computational Infrastructure - NCI NCI provides services through its National Facility at ANU, and through specialist facilities. The National Facility provides HPC systems, Data Storage systems and Cloud Computing, being a host for RDS and NeCTAR. NCI also has a Specialised Facility in Bioinformatics, located at the University of Queensland. This facility (barrine) was established to provide specialised resources for bioinformatics and computational biology. See

5 The running of barrine has now passed to the University of Queensland Research Computing Centre. Barrine is due to be decommissioned around the end of 2015, but replacement systems are coming into operation. These include QRIScloud and FlashLite, a very large memory cluster. FlashLite includes three main innovations: high speed flash memory (instead of spinning disk), large amounts of high-speed main memory and software shared memory. FlashLite specifications are available here. Note that FlashLite is open to a wider range of applications than the bioinformatics focus of barrine. Users with "big data" needs in any discipline should consider applying for resources. IM&T Scientific Computing staff will contact CSIRO users on barrine and offer advice on the transitioning of user workloads and data from barrine to other systems. If you have any concerns or questions, please contact the SC team by ing to schelp@csiro.au. Our people Ric Pascual Can you tell us about your role? I am an application support specialist with the Science Applications team. I have been with this team for about 3 months. A more descriptive title might be Scientific Software Engineer. It is a superposition of a scientist and a software engineer. I help people in CSIRO who need a software engineer s assistance to do things like design an application, use our High Performance Computing facilities or find a better way to use scientific oriented apps like Matlab. Similarly I assist those who need to solve scientific/numeric problems like solving partial differential equations or modelling a part of an ecosystem using information related technologies. As a rule most projects will need a combination of both, the proportion of each varying from project to project. I am currently working on two projects with CSIRO Energy. I am working with staff on the Optimising Central Receiver project to adopt collaboration tools (Confluence, JIRA, Stash) and other software development best practices such as refactoring, code review, and continuous testing (Jenkins). These tools enable the team to work effectively across three states, ensuring changes in requirements or even developers are sustainable. What did you do before this? I have a PhD in Physics (Flinders University) and PhD in minerals process engineering (University of Queensland). Some things I did before CSIRO were: doing research in electron-atom-photon scattering, lecturing honours level quantum mechanics, doing research in mineral processing, being a part of the team that created an application for simulating and optimising industrial flotation plants (JKSimFloat). I have been with CSIRO for about eight years before joining IM&T. In those years I have done physics-based numeric modelling like the adsorption of gold onto activated carbon and ecology-based matrix population models of coral reefs. I have also done application development ranging from Alcoa sponsored Gibbsite Precipitator Simulator to the Queensland Healthy Waterways sponsored Water Management Strategy Evaluation C# application. What appeals to you most about your work? Scientific Computing! The most pressing knowledge problems that we have currently whether in biology or solar energy or food production or the environment or climate change or physics I believe can only be solved by using the tried and true scientific methods in conjunction with the computer/information technologies. These problems are so big that I think human being s innate abilities alone are not enough to solve them. It is exciting to be part of the team that tackles these kinds of problems. What do you like to do outside of work? There are many things. To mention a few, I enjoy listening to good music from rock to classical. I like eating with family and good

6 friends, good wine with good food, which makes for a good time indeed! I also enjoy cooking, otherwise eating good food frequently in restaurants can be an expensive activity. News & Events National and International Conferences These conferences may be of interest to Scientific Computing users CSIRO CSS and eresearch Annual Conference, Melbourne, 1-4 March 2016 National Conferences eresearch Australasia 2015, Brisbane, October 2015 CSIRO will have a large presence at this event including multiple conference presentations, an exhibition booth, posters, and numerous demonstratons International Conferences The 8th ACM SIGGRAPH Asia Conference, Kyoto, 2-5 November 2015 SC 15, Austin, USA, Nov 15-20, 2015 Computational Merit Applications National Computational Merit Allocation Scheme (NCMAS) - applications close on 16 October. Pawsey Energy and Resources Merit Allocation Scheme - applications close on 23 October. Pawsey Partner Merit Allocation Scheme - applications close on 23 October. Applicants seeking assistance are encouraged to contact schelp@csiro.au. National/International News In July 2015, President Obama announced the National Strategic Computing Initiative to maximise the benefits of high-performance computing (HPC) research, development and deployment. The Initiative clearly acknowledges the vital role that HPC has already contributed to economic prosperity and scientific discovery. The NSCI's four key principles are: Deploying and applying new HPC technologies for economic competitiveness and scientific discovery. Fostering collaboration across government, industry and academia to maximise the benefits of HPC. Adopting a whole of government approach to HPC. Developing a comprehensive approach to HPC research that translates into new developments and operations. To meet these principles, a series of objectives are outlined including accelerating the delivery of an exascale computing system, as well as a path forward for future HPC systems in the post-moore s Law era. The announcement has elicited a favourable response from the research community and computing vendors. SC Circular is now externally accessible

7 View archive MyCSIRO Intranet