Introduction to Cube Cloud (CC) Introduction. Senior Transportation Engineer

Transcription

1 Introduction to Cube Cloud (CC) Presented By Ahmed Mohideen Citilabs Inc. October 15 th, 2014 Introduction Ahmed Mohideen Senior Transportation Engineer Rob Schiffer Director of Professional Services 2 1

2 Agenda Background Model and contract administration Preparing your model for CUBE Cloud Uploading your model to CUBE Cloud (CC Exporter) Initial Questions Your model on CUBE Cloud 3 Managing Scenarios Running your Model Model Administration Model Outputs and Analysis Agenda (cont d) Contract Administration Cube Cloud Store Final Questions 4 2

3 BACKGROUND 5 Background What is Cube Cloud? Why Cloud? Speed Scalability Sharing Access Management Benefits Work Flow Sign up 6 3

4 What is Cube Cloud? A cloud based environment for travel demand modeling in CUBE 7 Speed Scalability Sharing Ease of access Easy to manage Why Cloud? 8 4

5 Speed Santa Clara Valley, California Availability of large number of computing cores Reduce run times 50% to 80% for full model runs 85% to 95% for highway assignment only runs Scalability Scalable resources Processor cores Memory/disk Network bandwidth Independent model runs Resources scalable across simultaneous runs and within single run (speed) 5

6 Sharing Sharing of Computing resources Model Data Analysis Permission levels Administrator Regular user Access Access Anywhere Anytime Any number/type of users Browser based All device types PC, Tablet, Smart phone 6

7 Management Central repository for model/data Version control/less fragmentation Better collaboration Benefits Internet: movement from a desktop-bound, locked environment to an internet-based, open, sharable, work from anywhere/anytime environment Community Resource: model application and planning analysis done by non-experts using common web-browsers moving models to an active role in collaborative transportation planning Cloud-Computing: placement of the models, data and software in a cloud-computing environment lowering hardware costs locally while providing unlimited high-spec resources Lower costs for Users: movement from locally licensed desktops to software as a service model. Monthly subscription business model allows many to use the model at low, or even, no cost Lessens IT complexity: much of the IT burden of modeling is shifted from the user to the vendor Data and Software Integration: easier to integrate with external systems for development reviews, regional air quality analysis, pavement maintenance systems, traffic and transit ITS systems and to receive and use data from data probes, detectors and static data sources 14 7

8 Work Flow Publish the Model from Cube to the Cube Cloud Create, run and analyze scenarios from anywhere 15 Develop the Model with Cube in the Desktop Environment Sign-up Create a user account on CUBE Cloud services Registration code CUBE2012 Use a valid web address Free to register 16 8

9 CONTRACTS 17 Contracts Agreement between user and Cloud service provider (i.e., Citilabs) Contract defines number of hours, core, users, amount of storage, etc. Same user can have multiple contracts Required to run models Not all users need to sign contracts Contract hours can be shared among users 18 9

10 PREPARING YOUR MODEL FOR CUBE CLOUD Preparing Your Model for Cube Cloud Understanding CC directory structure Input file references Output file references User programs CC cluster setup Best practices 20 10

11 Understanding CC Directory Structure Minimal/fixed directory structure 21 Understanding CC Directory Structure Model This has a unique folder name for each model Working directory/scenario directory Apps Model script file and catalog information (.xml) Inputs All inputs (scenario specific, fixed inputs) Outputs Empty and used to store fixed outputs when model is run 22 11

12 Understanding CC Directory Structure (Cont d) CC Export Compile model scripts and model catalog Create one model script and a.xml file in web\apps Copy input files to web\inputs Compress web folder for upload CC Model Run Each scenario run gets its own copy of the model folder All outputs are saved in the Outputs folder or the Model folder (scenario/working directory) 23 Input File References Scenario specific inputs (e.g., zone data, highway network, etc.) FILEI input file statements with {SCENARIO_DIR} in path name e.g., FILEI ZONEI[1]= {SCENARIO_DIR}\Zones\ZoneData.DBF e.g., READ FILE ="{SCENARIO_DIR}\INPUT\ACCESS.PRN 24 12

13 Input File References (Cont d) Fixed model inputs (e.g., friction factors, mode choice factors, etc.) FILEI statements with fixed file path e.g., FILEI RECI = C:\Model\Parameters\MC_PAR.DBF 25 Input File References (Cont d) Use file keys for scenario specific input files e.g., FILEI ZONEI[1]= {ZoneData} e.g., READ FILE = "{Accessibility} Fixed Input Files Use file keys if the input should be changed by user later e.g., FILEI RECI = {ModeChoicePar} 26 13

14 Output File References FILEO, PRINT FILE= and COPY ENDCOPY statements e.g., FILEO MATO[1] = "{SCENARIO_DIR}\OUTPUT\MATRIX\OUTPUT.MAT CC model run will output these files to the outputs directory unless the file is output directly to the scenario directory Output all scenario specific files directly to {SCENARIO_DIR} e.g., FILEO MATO[1] = "{SCENARIO_DIR}\OUTPUT.MAT 27 User Programs Programs built by model developer external to Cube Activity based models Some post processors Select trip generation and mode choice models Programs with special set-ups 28 14

15 User Programs (Cont d) User supplied user programs e.g., any.exe files to perform special functions All input files and output files to/from the user program should be referenced to the {SCENARIO_DIR} Update any control files/input files with file paths Execute the user programs from {SCENARIO_DIR} 29 User Programs (Cont d) User programs requiring special set-up Certain user programs (e.g., Java Parallel Processing Framework) might require additional folder set-up and special configuration for running Additional set-up might include background processes to support the model run Citilabs will create a new CC image to handle these special cases (e.g., CT-RAMP) according to each client s needs 30 15

16 CC Cluster Setup CC provides scalability of available cluster nodes for each model run ProcessNum/ProcessList for DistributeMultistep and DistributeIntrastep statements should be calculated within the script based on the number of cores available for that run The number of cores available for the model run is input using a scenario key. No need for cluster commands to start and close cluster nodes. e.g., *Cluster Test 1-10 start exit 31 CC Cluster Setup (Cont d) {CORES}=### PILOT: If ({CORES}>1) ClusterNodes={CORES}-1 Distribute Multistep=T Intrastep=T Else ClusterNodes=1 Distribute Multistep=F Intrastep=F EndIf Intranodes = INT((ClusterNodes-2)/3) Start1=3 End1= Start1 + Intranodes Start2= End1 + 1 End2= Start2 + Intranodes Start3= End2 + 1 End3= Start3 + Intranodes DistributeMultistep ProcessId= Training ProcessNum=1 RUN PGM=MATRIX DistributeIntrastep ProcessId= Training ProcessList=@Start1@-@End1@ ENDRUN DistributeMultistep ProcessId= Training ProcessNum=2 RUN PGM=HIGHWAY DistributeIntrastep ProcessId= Training ProcessList=@Start2@-@End2@ ENDRUN RUN PGM=MATRIX DistributeIntrastep ProcessId= Training ProcessList=@Start3@-@End3@ ENDRUN 32 16

17 Best Practices Use file keys for input files Use unique file names for scenario specific input files Reference all model run outputs to the {SCENARIO_DIR} Directly link output files in application manager instead of reading the file internally Always use double quotes for file paths and names Execute user programs from scenario dir 33 CC EXPORTER UPLOADING YOUR MODEL TO CUBE CLOUD 17

18 CC Exporter What is CC Exporter? Register\Login Switching users Active\Pending catalogs Exporting Handling errors and warnings Uploading Pending 35 What is CC Exporter? Built-in CUBE 6.0 tool Prepares your model for CC Uploads the model to CC Lists your existing CC models (pending/active) Located at File>Export>Upload a Model 36 18

19 Register/Log-in Requires a CC account to access the functions of exporter Same user name/password as 37 Exporter Export/Upload Models Errors, Warnings & Logs Status 38 19

20 Switching Users Allows to switch CC user accounts inside exporter File>Switch User 39 Active/Pending Catalogs Active Catalog CC model which is approved and is ready to be run Pending Catalog CC model which has been uploaded by the user and is waiting for Citilabs to approve the model (as described below) Approval process- Check to make sure the model is uploaded by a trusted source Assign the right contract for the model Provide model admin permissions 40 20

21 Exporting Scenario Manager and Application Manager should be open Check the catalog name Export Compiles model scripts Build CC script Build catalog information (.xml) Copy inputs Compress upload folder Creates web and web.zip file Lists errors and warnings from the export process 41 Handling Errors and Warnings Errors Critical issues which have to be addressed before uploading the model Warnings Possible issues to be reviewed by the user Details Error or Warning Description Number of instances Location of each instance 42 21

22 Uploading Title Name of the model in CC Description brief description of the model This description will be listed in Check upload log to confirm model upload 43 What s Next? Model will be listed as Pending on Review and Approval by Citilabs Assign admin privileges to user Invite user to edit/run model 44 22

23 Initial Questions 45 YOUR MODEL ON CC 23

24 Activate Approved Model User invite sent from Citilabs User to click on login link Login with the username associated with address Multiple user names with same is allowed 47 Models Page Lists all your current models and applications purchased from the store Select parties responsible for your models 48 24

25 MANAGING SCENARIOS Items Add new scenarios Upload/edit scenario inputs Delete scenarios 50 25

26 Add Scenarios All new scenarios are located under Base scenario Create Child Scenario creates a new scenario under the currently selected scenario Name Description Share type Input files and key values are copied from parent Collapse and expand scenario families 51 Upload/Edit Scenario Inputs Edit Scenario activates edit controls for the currently selected scenario Input files (file keys) All input files can be updated by uploading a new file Check boxes All other key types are listed as text edit boxes 52 26

27 Delete Scenario Delete Scenario deletes the currently selected scenario Requires user confirmation 53 RUNNING YOUR MODEL 27

28 Items Model/Cluster set-up Selecting Contract (i.e., responsible party) Scheduling runs Checking run status 55 Model/Cluster set-up CUBE/CC model run environment e.g., CUBE 5.1.4, CTRAMP 5.1.4, etc Cluster process name should match with name used in your model Number of cores to be used for model run 56 28

29 Select Contract Contract to be assessed/billed for current model run Lists available contracts under your administration 57 Schedule Model Runs Select scenario Schedule Scenario starts model run Requires user confirmation 58 29

30 Status Check Model Run Status Lists currently running scenarios and completed scenario runs Status Pending the model run is pending to be started Running the model is currently running Completed the model run was completed 59 Run Status (Cont d) 60 30

31 MODEL ADMINISTRATION Model Administration Model set-up Sharing your model Managing model users Tracking model run activity 62 31

32 Model set-up Model logo appears on model page Model run environment CUBE version/cc image Cluster process name same one used in your model scripts Number of cores Number of process cores for model run In multiples of 8 (8 cores on each CC machine) 63 Invite users User Sharing your model Users can have multiple accounts under same user Model User vs. Model Administrator Model user will not have access to model admin page Model user cannot invite/manage other users 64 32

33 Managing Model Access Disable/Enable admin permissions Set-up model run payment options Delete user from model access 65 Deleting a Model Only the user who uploaded the model can delete it The delete button is available in Admin>Model Setup Other users who share the model will not have access to the model once you delete it 66 33

34 MODEL OUTPUTS Model Outputs Data page Working with output files Managing data folders 68 34

35 Data Page Provides access to all your scenario run outputs Organized by models and scenarios Follows MS-Windows like folder structure Tools to organize your output data 69 Working With Files & Folders Copy files between folders Download files Upload files from your PC Delete files Add/Download/Delete folders 70 35

36 ANALYZING YOUR MODEL OUTPUTS Analyzing Model Outputs Networks/Maps Database/Charts Tables 72 36

37 Networks/Maps CC has map tools to plot network data The loaded network has to be in a geodatabase (.mdb) in order to use mapping capabilities The geo-database should have the correct projection defined Copy mdb to {SCENARIO_DIR} and write loaded networks to mdb Use GeometrySource keyword in the NETWORK program to assign shape information to loaded network 73 Database\Charts CC provides 4 chart types to plot data The database should be available in a geodatabase (.mdb) to produce charts The data should be summarized Chart types supported: Pie Stacked Bar Line Scatter 74 37

38 Tables Tables are used for publishing smaller datasets such as summary tables. Bigger data sets can be accessed and downloaded from the Data page. This page lists two types of files.dbf.table 75 Tables (Cont d) DBF tables are regular.dbf format files. However, in order to be listed as a table in this page, these files need to have table_ as a prefix to their name..table files are any output text file named with extension.table..table should be pipe ( ) delimited. Below is a sample format Title="Table Name" Column1 Name Column2 Name Column3 Name

39 CONTRACT ADMINISTRATION Contract Administration Contract details Contract Activity Invite Users Manage Users 78 39

40 Contract Details/Activity Contract Details Contract hours Contract Activity Lists all model runs under the contract activity 79 Invite Users Contract User vs Contract Administrator Contract user cannot invite other users to the contract Contract user cannot change user privileges or remove users from the contract 80 40

41 Manage Users Change user privileges Remove user from contract 81 CUBE CLOUD STORE 41

42 Cube Cloud Store Cube Cloud store provides access to additional model applications (such as pre/post processing scripts) It also serves as a repository of model data (such as Census data) and other modeling resources (e.g., research reports) Users can buy model applications and add them to their user account Model data can be downloaded by user 83 Cube Cloud Store (Cont d) Free apps provide access to commonly used analytic tools. Comes standard with Cube Cloud! Paid apps allow users to sell their models on the Cloud 84 42

43 Questions 85 43