The evolution of DAVE

Transcription

1 The evolution of DAVE The evolution of a large IDL application Data Analysis and Visualization Environment (DAVE) project. Richard Tumanjong Azuah Larry Kneller Yiming Qui John Copley Robert M Dimeo Craig Brown Philip Tregenna-Piggott David Fanning Craig B. Markwardt 1

2 NIST Center for Neutron Research What is the NCNR? A neutron scattering facility Used by roughly 2000 scientists a year 20 neutron scattering instruments available For materials research eg atomic structure, dynamics and magnetism. How it works. Users apply for beam time Proposals are peer review Limited beam time (a few days) awarded to best ones User brings own sample and works with a staff scientists Data is generated counts as a function of angle, time, energy, temperature, magnetic field, etc Raw data is instrument dependent instruments are uniquely optimized to cover different regimes or science areas. 2

3 Data Treatment Process The 3 main wishes Data Reduction convert raw data to a universal scattering function, S(Q,E). Q atomic structure E atomic motion Data reduction is instrument dependent Visualization of raw or reduced data can provide useful information Analyse reduced data extract parameters that determine physical quantities atomic structure, magnetic moments, vibrational motion, etc. Analysis involves a mix of basic data operations, data modelling, visualization, etc. 3

4 Origins of DAVE Project Motivation Seven years ago at the NCNR: there were few or non-existent software tools for basic data reduction and analysis. traditional approach of user-written software was time-consuming. complexity steadily increasing with improvements in spectrometer designs. Software development process more error prone. 4

5 Origins of DAVE Project A pragmatic solution DAVE project created in 2001 to address shortcomings. Goals: User-friendly tool which can accommodate the software requirements for a range of instruments at the NCNR. Full availability at no cost to users. Instrument scientist should play key role in development, particularly data reduction aspects. IDL chosen as development platform Full support on several OS platforms Comprehensive support for scientific programming and optimized for large array operations. Built-in widget toolkit for providing GUIs Some IDL programming expertise License agreement with RSI Inc allows free and unlimited distribution of DAVE executable with an embedded run-time license. 5

6 The DAVE suite Main Application Defines and maintains application wide user preferences (eg data directories, useful symbols, etc) Define an internal data structure for storing experimental data Basic infrastructure to launch modules and pass data and to display PDF based help files. Main Parts Data Reduction Data Visualization Data Analysis least squares fitting Planning and other Miscellaneous Tools 6

7 The DAVE suite Data Reduction 10 supported instruments Performing data reduction with the FANS Data Reduction module. 7

8 Data Visualization Line, contour, image and surface plots The DAVE suite The Data Browser interface showing data and graphic containers. Also shown is an image view of a 2D dataset as well as a line plot of multiple 1D datasets. 8

9 The DAVE suite Data Analysis Dataset manipulations (merging, scaling, combining, etc) Fit data to model functions using least-squares fit Single crystal analysis (projections, cuts, etc) The PAN module in action. Fitting a multi-function model to HFBS data. 9

10 The DAVE suite Miscellaneous Tools More than a dozen tools Eg Neutron cross-section tables and Calculations for compounds 10

11 DAVE Limitations Limitations of DAVE Had evolved organically (little common API, too much duplication of effort) Was too modular with little communication between components Was becoming harder to maintain and even more difficult to extend Therefore, time to consolidate 11

12 The arrival of itools Experimented with itools (iplot, isurface, etc) in 2004 Contained many powerful features Undo/Redo capability, Messaging system for sending notification messages between components, highly customizable visualization graphics especially annotations, saving application state, and many more Was very impressed. Wanted similar functionality in DAVE! 12

13 What are itools Has two parts: 1 The itool Framework a set of classes with well defined API. It is a comprehensive component library consisting of dozens of classes. 2 The Intelligent Tools or itools are a set of 7 flagship applications built on the framework. They are end-user applications (iplot, iimage, icontour, isurface, ivolume, ivector, imap). For maximum flexibility, the framework must be used to develop own applications if they deviate too much from the example itools. iplot in action (from IDL demo) 13

14 Inadequate Documentation Inadequate Documentation Many undocumented classes. Lack of complete tutorials even in itool Developer's Guide book. Took me 6 months to reach sufficient level of understanding. Some improvements since the early days for example the addition of the object model inheritance diagram. 14

15 Data-centric Interface Data-centric UI Example itools present image processing interface dominated by graphics window. DAVE primarily manipulates experimental data Therefore it needs a data processing interface. Experimental dataset must be presented and maintained within a single entity. These datasets have many components besides the plottable data. Graphics and tree widgets are only viable UI elements for user interaction with data. Can't use graphics because dataset is more than just the plottable data. Adopt the itool Visualization Editor widget interface. But add a custom DAVE Data Manager for storing experimental datasets. Design application Tool Class (DAVETool which subclasses from IDLitTool) that Maintains a separate data manager folder from the default system data manager to prevent automatic modification of experimental data. Activate or Deactivate menus based on selected data types from the Data Manager Design appropriate data hierarchies for all supported file formats. Make use of IDLitData, IDLitDataContainer and IDLitParameterSet classes. 15

16 Data-centric Interface The main interface. Consists of: 1 Data Browser shows contents of the Data Manager 2 A property sheet for for viewing details of selected data items 3 Menus 4 A Property sheet for selecting visualizations and associating data with the visualization Parameters. Note: Note that some menus are deactivated because those operations cannot be applied to the selected data item, S(Q,w) 16

17 Using built-in Visualization classes Extend Visualization Classes IDLitVis* classes eg IDLitVisPlot Accommodate metadata eg axes titles and units (OnDataChangeUpdate) Handle histogram or point mode data the mode is also specified as metadata. Automatically remove supporting data created during a visualization when no longer required eg color palettes. Set IDLitData's auto_delete property for such data. Also extend the visualization tool class so that it is dependent on the DAVE application tool (DAVETool) class. 17

18 Simplify visualization creation Automate Visualization Creation Automatically associate experimental data with visualization parameters for all supported visualizations. Extended IDLitData class with an axistype property. It specifies how each data element is to be used in a visualization. AxisType property Meaning in a visualization 0 First indep. axis data (x-axis) 1 Second indep. axis data (y-axis) 2 Dependent axis data (z-axis) 3 Error in dependent axis data 18

19 Replacing a Default Component ASCII Text Reader component Cannot successfully unregister/register the same component name using different class names without first deleting the IDL settings file (itools_prefs.sav). Default ASCII text Reader Modified ASCII text Reader 19

20 What has been accomplished with itools Project management users can now save and resume data + visualizations + application state in a DAVE project file. Data Manager. Easily handle neutron scattering datasets. Visualizations. Adapt built-in IDLitVis* visualizations to work well with DAVE datasets. Support for various data file formats (file readers) DAVE and ASCII (SPE, column, group) Support for various data file formats (file writers) DAVE and ASCII (SPE, column, group) Basic analysis functionality data operations. 20

21 A resumed DAVE session 21

22 An operation component Operations retrieve data from the Data Manager User customizes operations through UI Service components For most Operations, the Property Sheet widget can be used to provide the UI Service dialog no need to waste time designing custom dialogs! With careful thought, it is possible to check user input for consistency. E.g. The Nos of datasets to sum field can not be more than the nos of datasets in the Data Manager. 22

23 What is Good What is good about itool Framework Really like the Operations component API especially the undo/redo buffer - excellent for a data-processing application Use of Property Sheet widget and other built-in dialogs give consistent look and feel to a large application. The data classes are comprehensive and can be used to create sophisticated data hierarchies. 23

24 Suggestions for Improvement Widget toolkit is dated needs to be replaced! New UI Elements such as worksheets are needed. Rich text support will be wonderful. Cut and paste should work on all platforms. Font support needs significant enhancement. ħ and Å symbols are a must for scientific applications. Better control over which components are registered by default (by the system object). Without having to examine IDLitSystem class code to determine names of components to unregister. Update the itool Developer's Guide with more meaningful tutorials. Also complete the class reference documentation. For example the excellent metadata interface in IDLitData is still undocumented. Bug? Saved preferences file (itools_prefs.sav) may be preventing component re-registration. 24

25 Conclusion The itool Framework architecture has introduced several new concepts that will result in a fresh approach to application development in IDL. However, there are key problem areas such as documentation, a dated widget toolkit and inadequate font support. 25