9 ITS Standards Specification Catalog and Testing Framework

Transcription

1 New Yrk State ITS Standards Specificatin Develpment Guide 9 ITS Standards Specificatin Catalg and Testing Framewrk This chapter cvers cncepts related t develpment f an ITS Standards Specificatin Catalg and Testing Framewrk, and includes the fllwing tpics: Standardizing Language and Cntent f Prcurement Specificatins (the previus chapter begins this prcess). Interchangeability f Devices and Certificatin Testing and Prcedures Develpment Test Tls One such testing framewrk is dcumented in the NTCIP 9012 Testing and Cnfrmity Assessment Users Guide. Thugh, the 9012 dcument cvers nly the tpic f center-t-field cmmunicatins, many f the cncepts translate t ther areas, such as center-t-center and WAVE/DSRC. It is wrth nting that the OmniAir Cnsrtium has initiated a certificatin testing prgram fr WAVE/DSRC prducts. 9.1 ITS Standards Specificatin Catalg An ITS Standards Specificatins Catalg wuld define the fllwing (as a minimum): - Sample cncepts f peratin and functinal requirements fr field device cmmunicatins, center-t-center traffic, traveler, and incident infrmatin sharing, and WAVE/DSRC ITS applicatins. This will define the infrmatin needed t supprt develpment f the infrmatin level specificatins. - Sample detailed specificatins included filled in PICS (prtcl implementatin cnfrmance specificatin) and MIBs (management infrmatin base) fr field device cmmunicatins (infrmatin level specificatins). - Infrmatin level standards fr cre dialg and message specificatins fr center-tcenter cmmunicatins, including WSDL (Web Services Descriptin Language) and XML Schemas. And cre dialg and MIB definitins fr center-t-field cmmunicatins. - Sample specificatins t supprt specific ITS Standards Cmmunicatins Packages ( a cmplete cmmunicatins prtcl stack ) fr applicatin, transprt, subnetwrk, and plant level standards). Given these pre-defined specificatins, ITS engineers culd quickly pull tgether draft ITS specificatins, draft feasibility studies fr alternatives fr deplyment f ITS cmmunicatins, and Systems Engineering Analysis reprts. 9-1

2 New Yrk State ITS Standards Specificatin Develpment Guide The cncept f using an ITS Standards Catalg is shwn belw. Figure 9-1. ITS Standards Specificatin Develpment and Testing Framewrk Natinal ITS Standards New Yrk State Requirements S tatew ide ITS S tandards Specificatins Catalg Statewide Specificatin Spec Requirements Req #1 Req #2 Req #3 Statewide Test Prcedure 1 Test Prcedure 2 Test Prcedure 3 ITS Prject Specificatin ITS Prject 9.2 Interchangeability f Devices and Appraches t Certificatin Testing One apprach fr an agency t btain interchangeable devices frm a specificatin is t examine the standards and reslve the fllwing, as a minimum: a. Which ptinal bjects a device must supprt b. Specify minimum supprt values fr certain capabilities (i.e. the minimum number f plans in a traffic signal cntrller, the minimum number f phases, size f event lgs, number f fnts supprted, etc) c. Interpret bjects t have a cnsistent implementatin fr the device (e.g., patterntabletype frm NTCIP 1202:1996) When the functinality is clearly described (in a measurable and bservable manner), then devices that use the same bjects t manage the same well-defined functinality will be interchangeable. NYSDOT may cnsider establishing a centralized testing sectin r labratry fr New Yrk State, and a qualified prducts list. 9-2

3 New Yrk State ITS Standards Specificatin Develpment Guide As an alternative, the testing f manufacturers devices may ccur at the manufacturers facilities. 9.3 Testing Methdlgy Prepare the What Needs t be Tested. Only the cmmunicatins interface t a system: Dialgs (Checking fr prper sequence) Message Creatin (Data int Messages) Message Parsing (Messages int Data) Message Delivery (Send / Receipt) - Prtcl Handling f Failures The testing f system cmpliance with the NYSDOT C2C Specificatin may utilize third party sftware r vendr prvided sftware as the testing tl. The prcurement f any third party sftware shuld be crdinated thrugh the Prject Manager and Field Test Manager t ensure versin cntrl and apprpriate tracking f test cnditins. The table belw cntains an utline f what shuld be included in a test plan. Table 9-1. Outline f Field Cntents Identifier (assigned by the Field Test Manager) Intrductin Backgrund Scpe References Test Items Features t be Tested Features Nt t be Tested Apprach Degree f Cmprehensiveness Desired Techniques and Tls Item "Pass/Fail" Criteria Suspensin Criteria and Resumptin Requirements Test Suspensin 9-3

4 New Yrk State ITS Standards Specificatin Develpment Guide Resumptin Requirements Test Deliverables Traceability t Requirements Test Cases Pre-test Interviews Test Prcedures Step-by-step prcedures fr each Cre Functin Test Data Supprting Data and Infrmatin Testing Tasks and Schedule Testing Tasks Schedule Envirnmental Needs Respnsibilities Staffing and Training Needs Risks and Cntingencies Apprvals Develp Test Prcedures The test prcedures prvide step by step descriptins f the tests t be perfrmed. The Dialg Wrksheet presents an excellent starting pint fr develpment f detailed prcedures (See Chapter 5). Starting frm the Dialg Wrksheet allws test plan/prcedures t trace t the use case/requirements test cases t ensure verificatin f the system functinality. The wrksheet allws breaking ut the test prcedures int the fllwing verificatin steps fr each dialg specified: - Verify cmmunicatins exchanges and crrect sequencing f inputs and utputs. - Verify message/mib bject input-utput cmpliance with specificatin XML Schema r ASN.1 definitins. - Verify encding f messages/mib bjects e.g., SOAP fr C2C r OER fr MIB bjects. - Verrify transprt mechanism f messages and MIB bjects e.g., HTTP fr C2C r SNMP fr MIB bjects. 9-4

5 New Yrk State ITS Standards Specificatin Develpment Guide Cnduct the Test Cnduct f the test shall begin when all applicable test plans have been apprved. The test team shuld allw enugh time n-site t cllect all necessary data. Data analysis will nt be perfrmed n site, except when necessary t verify an anmalus cnditin. If an anmaly ccurs it culd be related t the standards r the prduct implementatin. Once the anmalus cnditin is determined NOT t be caused by an errr in the standard then n further reslutin by the test team is necessary. In sme cases, it may be prudent fr the test team t assist the hst agency and/vendr t further reslve the anmaly, but in such cases these results shuld nt be included in the fficial test findings. Test data shuld be captured in a nn-intrusive manner s as nt t interfere with nrmal test site peratins. The test team will lg all actins in a test lg. The frmat f the lg is left up t the discretin f the test team and the Field Test Manager. Hwever the lg must include the date and time f all events s that the captured raw test data can be crss-referenced t specific events Pst Test Analysis The results f the tests shall be analyzed fr prblems/ anmalies. Finally, a pst-test wrkshp shuld be held t brief all stakehlders f the test findings and recmmendatins, and t slicit feedback and cmments fr use in preparatin f the final test reprt fr the tested standards. The Prject Manager and Field Test Manager will be included in the crdinatin and planning f the pst-test wrkshp. 9.4 ITS Standards Testing Tls The sectin abve includes an verview f a methdlgy t ITS standards testing. This sectin describes the type f sftware tls necessary t carry ut ITS standards testing: Center-t-Field Testing Tls Center-t-Field. Because SNMP, ASN.1, and the MIB are pen standards, ne can test field elements and center systems separately. A center simulatr (r client test sftware) can be used t test the interfaces t the sign (withut having t have a prductin center system cnnected t the sign.) A field device simulatr can be used t unit test center-based sftware that will ultimately cmmunicate with a sme field device. The NTCIP standard maps functinal requirements t NTCIP Objects, but mre imprtantly, the standard includes a sectin n Dialgs which indicates which NTCIP Objects must be set and retrieved t enable a specific device functin. NYSDOT can create a standardized statewide MIB. NYSDOT culd then require that a device manufacturer shw that the MIB, running under the SNMP agent f the device manufacturer, is able t respnd t a series f cmmands utlined in a test plan. SNMP and MIBs are used thrughut netwrk cmmunicatins. Fr example, mst netwrk ruters and cmmunicatins netwrk devices are mnitred using SNMP. As a result a number 9-5

6 New Yrk State ITS Standards Specificatin Develpment Guide f cmpanies exist that prvide 3 rd party ff-the-shelf MIB cmpilers and SNMP mnitring tls. The primary tls required fr testing f field devices are therefre: SNMP-based testing sftware (e.g., NTCIP Exerciser, Intelligent Devices Tester) fr testing f client-side (center) and field-side functinality, and MIB Cmpiler t check the validity f the written (ASN.1) frm f the MIB Center-t-Center Testing Tls The center-t-center cmmunicatins test plan prvides a general template fr testing f center interfaces based n the NTCIP XML-based standard this includes testing f dialgs and messages, and the lwer level prtcls (HTTP, FTP, IP, etc.). Testing f the center interfaces specified wuld invlve tls fr evaluating the validity f a sample f XML messages and the transprt mechanism fr mving the XML cntent frm an external center (in this case the test client) t the NYSDOT center sftware being tested. Therefre, the primary tls fr testing f center-based cmmunicatins wuld be: XML Schema Validatin. WSDL Validatin. XML Message Validatin. This is a tl that validates that Applicatin Level Prtcl Validatin. An external center system simulatr can be develped using a thin client web brwser, r simple client scripts that exercise a center s interfaces. This wuld include testing f message sequences defined in the dialgs, and prtcls (HTTP, FTP, etc.). As with the center-t-field case, because the center-t-center standard is based n pen standards that are widely adpted by the IT cmmunity, sftware develpment tls and testing tls are available. 9-6