April 30,

Size: px

Start display at page:

Download "April 30, 2014. https://sites.google.com/site/openmodeldata"

Horatio Webster
8 years ago
Views:

1 April 30,

2 Purpose Statement Create a simple standard matrix format for storing and transferring matrix data that can be easily adopted For Who That Unlike Anyone managing transportation data with a matrix structure Wants a portable and open format for storing and exchanging that data Is easy to understand, can be used in custom programs/scripts, and supports commercial products Existing solutions that are either proprietary, cumbersome, slow, and/or too large

3 Who We Are A diverse group of agency, consultant, and research modelers Formed a working group to define and implement a the file format for use in modeling Similar to the HTML working group for example Formed in early 2013, shortly after TRB Periodically meet to update the standard/project

4 We Believe An open matrix file will: make model development and application easier across models and software packages help spur industry research and innovation by having a shared data structure (think shapefile) enable researchers, students, agencies, and practitioners to collaborate in ways that are currently difficult, cumbersome and errorprone

5 What is an Open Matrix? Open matrix is a file format (i.e. skims.omx) Based on open-source HDF-5 file storage technology Binary data storage with compression Can store multiple matrices in one file Can include multiple indexes/lookups Can contain additional attributes for both matrices and indexes

6 Availability Specification available on our website: Interfaces in Python, R, Java, and now C# Supported by PECAS INRO and PTV have committed to developing interfaces as well Hoping to get broad industry acceptance for it

7 OMX Review of Existing Solutions OMX Specification Jumping into OMX Using OMX for your project or model What s Next

8 Review of Existing Solutions (a.k.a. why create another data format?)

9 Text Files Pros: Simple Universally readable Cons: Enormous Unwieldy for large datasets Difficult to view by column Hard to include multiple tables

10 SQLite Files Pros: Widely understood/readable Supports multiple tables Cons: Not well-suited to matrix data Slow, on some implementations

11 BSON ( Binary JSON ) Pros: Lightweight Traversable Cons: Designed as a wire protocol Primarily used for tabular data Viewing is difficult

12 Zip Matrix Pros: Just a zipped folder of binaries Cons: Viewing is difficult Slow for random access to data

13 Vendor Formats Pros: Commercial support Familiar, fast, feature-rich Cons: Licenses isolate users Public can t have access Long-term archival problems Zero collaboration potential

14 HDF5 Hierarchical Data Format Pros: Well-established format in use since 1987 (NCSA, physics, etc) Compact & Fast Any platform, any language Flexible Cons: Unfamiliar to travel modelers Extremely flexible; too flexible!

15 More about HDF5 - Open source, permissive license - Every HDF5 file is a hierarchical container (like a zipfile) - The root node can have child nodes & subfolders - Every node can have key/value attributes attached - Data can be any shape, size, or structure

16 The OMX Specification (or, how to rein in the complexity of HDF5)

17 OMX Specification An OMX matrix file is a structured collection of two-dimensional array objects and associated metadata.

19 ROOT node contains: SHAPE (rows x cols) OMX_VERSION (0.2)

20 DATA folder contains all your matrices

21 MATRICES Are named Are all the same size Are all two dimensions

22 MATRICES Can have unlimited key/value attributes: TIMEPER: AM MODE: transit VOT: 11.69

23 LOOKUP folder contains any mappings of row numbers e.g., zone-to-district

25 Jumping into OMX: Python example

26 Using OMX for your project or model

28 OMX API is already available in: Python - Java - C# - R OMX API is coming soon in: Fortran - C++ OMX native support coming soon in: Emme - PTV - DTAlite/Nexta - PECAS

29 Next Steps

30 Next Steps: Converters for existing matrices We will soon have converter programs for existing Cube and Emme matrices (running converter requires valid license)

31 Next Steps: More and better installers and docs The website is feature complete but we could use some volunteers to help with testing and documentation. Platform-specific installers would also help get new users up to speed quickly.

32 Conclusions OMX is a simple standard matrix format for storing and transferring matrix data APIs are available for the language you re using GUI Viewer is available for Win, Mac, Linux Join us, let s modernize the industry together! Publish and share your existing trip tables in OMX right now. Check out our website and list for more information

33 Thanks! Ben Stabler Billy Charlton Brian Gregor Chris Frazier Elizabeth Sall Jeremy Raw John Abraham Suzanne Childress

Databases in Organizations

Databases in Organizations The following is an excerpt from a draft chapter of a new enterprise architecture text book that is currently under development entitled Enterprise Architecture: Principles and Practice by Brian Cameron