Conformance 101. February FACE is a Trademark of The Open Group

Transcription

1 Conformance 101 February 2016 FACE is a Trademark of The Open Group

2 Purpose of this Presentation Overview of the FACE Conformance Program and Related Policies and Procedures Audience: New members of the FACE Consortium New participants in the Conformance Subcommittee Other interested parties 2

3 What is FACE Conformance? FACE Conformance is the assessment of a Software Item, known as a Unit of Conformance (UoC), to the applicable Conformance Requirements contained in the FACE Technical Standard Those Applicable Requirements are determined based on the segment and profile selected in the design of the particular UoC Verification of Conformance is conducted utilizing automated test tools and inspection of design and test documents The specific requirements, method of verification, and associated verification evidence is detailed in the Conformance Verification Matrix (CVM) 3

4 What can be Certified as FACE Conformant? Certification is for Units of Conformance (UoC) or UoC Packages Defined on next slide There is No compliance ; things are either certified conformant or not. No FACE certification for entire systems Systems can be comprised completely of Certified UoCs or a mix of Certified UoCs and other software No FACE certification for independent libraries, runtimes, frameworks These can be included in a certification of a larger set 4

5 What is a Unit of Conformance (UoC)? A Unit of Portability (UoP) is Items that fit completely within one of the FACE Architecture portable segments Platform Specific Segment Transport Services Segment Portable Component Segment A Unit of Conformance (UoC) is Items that fit completely within one of the five FACE Architecture segments Platform Specific Segment (also a UoP) Transport Services Segment (also a UoP) Portable Component Segment Operating System Segment (also a UoP) (is not a UoP) I/O Services (is not a UoP) 5

6 What is a Unit of Conformance Package? A Unit of Conformance (UoC) Package is A Collection of UoCs meeting the FACE Standard definition of packages Same Partition Allowable Segments 6

7 What are UoC variants? UoC Variants are versions of a UoC generated to a specific set of functionality and target environment from the same UoC source code. UoC Variants which conform to the Technical Standard may be certified and listed together on a single Conformance Certificate. If the Software Supplier is submitting UoC variants, conformance evidence must detail the conformant variants and their differences. 7

8 Conformance Processes FACE Verification is the act of determining the conformance of an implementation to specification requirements. Verification is handled through an entity known as a Verification Authority (VA), a technical expert on the FACE Technical Standard and Verification process and approved by the FACE Consortium Steering Committee FACE Certification is the process of applying for a FACE Conformance Certificate once verification has successfully been completed. Certification is processed through the FACE Certification Authority (CA) FACE Registration is the process of listing FACE Certified UoCs in a public listing of FACE Certified UoCs known as the FACE Registry. The FACE Registry is accessed from the FACE Landing Page 8

9 Conformance Program and Processes Software Supplier Initiate Verification Initiate Certification Initiate Registration FACE Verification FACE Certification FACE Registration FACE Verification Authority (VA) FACE Certification Authority (CA) FACE Library Administrator (LA) 9

10 Driving Factors Software Supplier is in control Software Supplier initiates the steps No listing of UoCs in process Software Supplier provides information for the Registry (the public listing) FACE Registry does not contain a way to download; Software Supplier must be contacted Software Supplier IP is protected 10

11 Driving Factors No Functional or Performance Testing Interfaces are tested Other Verification Evidence is inspected (evaluated) Functional Testing is assumed as part of other development processes and is not required for FACE Conformance 11

12 Driving Factors Certification is for Portable Software structure The source code and design are certified conformant to the FACE Technical Standard One instance of the object code, compiled for the test environment, is tested for conformance to the FACE APIs and data model Recompiling to a different target does not cause a loss of FACE Certification 12

13 Roles Software Supplier: The Software Supplier is anyone providing software (UoC) to be certified. This may include the original software developer, an integrator, or another entity wishing to certify software developed from another party. 13 FACE Verification Authority: A FACE Verification Authority (VA) is one of several organizations approved by the FACE Consortium to evaluate software against the FACE Technical Standard. The VA is an expert on the FACE technical standard and verification process. The VA conducts or witnesses conduct of the For-the-Record Test, utilizing an approved Conformance Test Tool, and inspects the Verification Evidence.

14 Roles FACE Certification Authority: The FACE Certification Authority (CA) is the singular organization approved by the FACE Consortium that can provide a FACE Conformance Certificate FACE Library Administrator: The FACE Library Administrator manages a listing of FACE Certified UoCs known as the FACE Registry FACE Trademark Licensor: The FACE Trademark Licensor issues the FACE Conformance Certification Trademark for Certified Units of Conformance and Certified Unit of Conformance Packages 14

15 Conformance OV-2 Cer+fica+on Package So9ware Supplier Conformance Cer+ficate Conformance Authori+es Cer+fica+on Authority (CA) Registra+on Shared Data Model Library Administrator Verifica+on Results Package SW Verifica+on Package ITAR SDM Snapshot Registry Verifica+on Authority (VA) Consor+um Documents Tools Public SDM Snapshot Reference Repository Verifica+on Reten+on Repository Verifica+on Results Package Library Repository Requirements Library Repository Requirements and Procedures Documents Consor+um Documents Cer+ficate ID Cer+fica+on Reten+on Repository Library Repository Requirements 15

16 Key FACE References Technical Standard Requirements for the FACE Architecture Conformance Verification Matrix Guidance and Verification Methods Conformance Policy Policy for certification of UoCs Conformance Certification Guide Guidance on the Policy and Program 16

17 Software Supplier Provides The Unit of Conformance (UoC) for Verification Object code, supporting files, and verification evidence Supplied to the Verification Authority (VA) only All UoCs included in a UoC Package Conformance Statement Identification of the Unit of Conformance (UoC) How the Unit of Conformance (UoC) meets FACE Technical Standard requirements Registry Entry Description of the Unit of Conformance (UoC), associated conformant UoC variants, and UoC Package as applicable Contact Information for Obtaining the Certifed Unit of Conformance (UoC) or Certified UoC Package 17

18 VA Provides Verification Statement Indication that the UoC Passed Verification For each UoC comprising a UoC Package, as well for the UoC Package itself Failed Verification is communicated to the Software Supplier specifying what and why failure occurred 18

19 CA Provides Conformance Certificate Indication that the Unit of Conformance (UoC) Passed Certification For each UoC comprising a UoC Package, as well for the UoC Package itself Failed Certification is communicated to the Software Supplier with sufficient detail for the problems to be resolved 19

20 FACE Conformance Program Steps and Processes Software Supplier Conduct Preparation Initiate Verification Initiate Certification Initiate Registration FACE Verification FACE Certification FACE Registration FACE Verification Authority (VA) FACE Certification Authority (CA) FACE Library Administrator (LA) 20

21 Conformance Preparation FACE Conformance Tab FACE Landing Page FACE Reference Repository Software Supplier Supplier Obtains References and Tools FACE Technical Standard Reference Implementation Guide (RIG) Automated Tools, SDK, ITK Conformance Certification Users Guide Conformance Policy Verification Matrix Matrix Users Guide (MUG) Conformance Test Suite 21

22 FACE Library Overview FACE Library Infrastructure Library Administration FACE Consoritum information FACE Events Procurements Main Landing Page Manages conformance and registration workflow VA CoP Library Portal Listing of FACE Certified UoCs Searchable Metadata Registry PR/CR System Reference Repository Verification Retention Repositories Certification Retention Repository Product Repositories VA Site CA Site Vendors Site Problem reports and change requests for Consortium artifacts and tools Dev/Test Tools Business Guide Technical Standard Contract Guide Artifacts submitted for FACE conformance verification / certification Certified FACE Conformant Software and associated artifacts 22

23 Conformance Preparation Identify FACE Criteria Number of UoCs for each: o FACE Architecture Segment o Profile (General, Safety, Safety Extended or Security) o Supporting Libraries for UoC o Test Environment 23

24 Conformance Preparation Supplier provides Verification Evidence o A trace of the FACE requirements to specific documents supporting the requirements o Required for all items in the Tech Standard identified as needing inspection in the Conformance Verification Matrix (CVM) including applicable conditional requirements Verification Needed (Y or N) FACE Segment Technical Standard for the FACE Reference Architecture Edition 1.0 Verification Method Conformance Artifacts (DID or equivalent) N PSS Segment Requirements PSSS 9. All communication with the IOSS shall go through the I/O Services Interface. Test Test Suite SW Supplier Artifact Cross- Reference Verification Notes Conditional Reqs Y Y PSSS 10. Messages communicated through the I/O Services Interface shall be in the format defined in Section D.11. Inspection SDD Y PSSS 11. All components of the PSSS shall use the interface defined in Section 3.11, Section 3.12, or Section 3.13 to access the functions provided by the OSS. Test Inspection Test Suite SAD SDD Inspection is only of Java frameworks or Ada run-times. 24

25 Conformance Preparation Supplier Selects a Verification Authority o List of Approved VAs from the Landing Page o Meets supplier needs o Not limited to Internal Verification o Willing to perform verification for the UoC s applicable FACE Architecture segment, e.g., Operating System 25

26 Conformance Preparation Software Supplier FACE Trademark Licensor FACE Verification Authority (VA) FACE Certification Authority (CA) Supplier Establishes Legal Agreements Conformance Certification Trademark License Agreement with TM Licensor Verification Agreement with Selected VA Certification Agreement with CA 26

27 Conformance Verification Process Software Supplier Initiate Verification Select & Establish Contractual Relationship with VA FACE Verification Authority (VA) Develop SW Verification Package o Verification Agreement o Verification Evidence o Conformance Statement o SW Product Set 27

28 Conformance Verification Process SW Verification Package o Verification Agreement - defines the conformance verification services to be provided by the VA and acceptance by the SW Supplier to provide the required verification evidence and SW Product Set o Verification Evidence - supporting verification documentation submitted by the SW Supplier to provide evidence of FACE Conformance to the applicable conformance requirements of the Technical Standard that are not directly tested by the Test Suite. The verification evidence is organized to correlate with the specific conformance requirements and verification approach contained in the applicable segment of the Conformance Verification Matrix 28

29 Conformance Verification Process SW Verification Package (cont) 29 o Conformance Statement SW Supplier s response to a standard questionnaire, tailored to the appropriate Segment of the Technical Standard, structured to obtain precise identification of the software product and conformance evidence. It includes SW product description documentation to uniquely identify and configuration manage the SW product through the conformance process. The Conformance Statement identifies the specific edition of the Technical Standard against which the SW product is being certified, the applicable set of conformance requirements, the corresponding Conformance Verification Matrix version, and the version of Conformance Test Suite used for verification. If the Software Supplier is submitting UoC variants, a Conformance Statement must detail the conformant variants and their differences.

30 Conformance Verification Process SW Verification Package (cont) o SW Product Set Includes object code representing all source code compiled for the test environment, appropriate data model files, associated information for set-up with interfacing segments, and minimum computer operating environment requirements If verification will include UoC Variants, the Software Product Set must include multiple versions of the object code. The versions of object code files must be representative of all UoC Variants for which verification is sought. The Software Product Set must include an analysis of this coverage. 30

31 Conformance Verification Process Software Supplier Initiate Verification Inspect SW Verification Package FACE Verification FACE Verification Authority (VA) Evaluate the Verification Evidence Conduct/Witness FTR test using Approved Conformance Test Suite Issue Verification Statement Archive Data 31

32 Conduct/Witness the FTR Test Uses the Applicable Approved FACE Conformance Test Suite Witness means watching execution of the FTR test not inspection of test results or test report Evaluates the Object Code through Linking (not executing) the product code Evaluates the Data Model for Conformance (IDL in 1.0) Does no functional testing; only ensures the appropriate FACE APIs are present and correct 32

33 Evaluate the Verification Evidence Inspect the trace from Technical Standard to specific locations in Designs, Requirements, Test Procedures, Test Reports, etc. Ensures the product meets requirements not covered by the Conformance Test Suite (FTR Test) Verification Needed (Y or N) FACE Segment Technical Standard for the FACE Reference Architecture Edition 1.0 Verification Method Conformance Artifacts (DID or equivalent) N PSS Segment Requirements PSSS 9. All communication with the IOSS shall go through the I/O Services Interface. Test Test Suite SW Supplier Artifact Cross- Reference Verification Notes Conditional Reqs Y Y PSSS 10. Messages communicated through the I/O Services Interface shall be in the format defined in Section D.11. Inspection SDD Y PSSS 11. All components of the PSSS shall use the interface defined in Section 3.11, Section 3.12, or Section 3.13 to access the functions provided by the OSS. Test Inspection Test Suite SAD SDD Inspection is only of Java frameworks or Ada run-times. 33

34 Conformance Verification Process Software Supplier Initiate Verification FACE Verification Verification Results Pkg & SW Verification Pkg Verification Retention Repository FACE Verification Authority (VA) 34

35 Conformance Certification Process Software Supplier Initiate Certification Establish Contractual Relationship with CA Verification Results Package FACE Verification Authority (VA) FACE Certification Authority (CA) Submit Legal Agreements o Certification Agreement o Trademark License Agreement CA Requests Verification Result Package from VA 35

36 Conformance Certification Process Software Supplier Initiate Certification FACE Certification FACE Certification Authority (CA) Ensure legal agreements are in place (Certification Agreement and TMLA) Review Conformance and Verification Statements for completeness and correctness 36

37 Conformance Certification Process Software Supplier Initiate Certification FACE Certification Conformance Certificate, Conformance Statement, Verification Statement, TMLA Certification Retention Repository FACE Certification Authority (CA) 37

38 FACE Registration Process Software Supplier Initiate Registration Submit Product Description & Conformance Certificate ID to Library Administrator FACE Library Administrator (LA) 38

39 FACE Registration Process Software Supplier Initiate Registration Request & Receive conformation of Conformance Certificate from CA Populates FACE Registry with Product Description & Conformance Certificate Conformance Certificate ID FACE Certification Authority (CA) FACE Registration FACE Library Administrator (LA) 39

40 Conformance Program and Processes Software Supplier Initiate Verification Initiate Certification Initiate Registration FACE Verification FACE Certification FACE Registration FACE Verification Authority (VA) FACE Certification Authority (CA) FACE Library Administrator (LA) 40

41 Conformance Packages Verification Agreement Verification Evidence SW Product Set Conformance Statement Verification Statement Certification Agreement SW Verification Package X X X Verification Results Package Verification Retention Repository X X X Certification Package Certification Retention Repository X X X X X X X X X TMLA X X Conformance Certificate X X X 41

42 Other Conformance Topics Recertification/Modification of a Certified UoC PR Process Appeals Auditing 42

43 Recertification/Modification of a Certified UoC Section Type of Modification Verification Requirement Certification Requirement Conformance maintenance release Software modification Delta-verification Delta-verification Certification information update New certification release Renamed UoC None Certification information update Modification of FACE Registry information Other modifications None As determined by the VA Certification information update As determined by the CA 43

44 Recertification/Modification of a Certified UoC Conformance Maintenance Releases are the result of modifications made to the UoC because of conformance-related issues that arise following FACE Conformance Certification; for example, conformance-related problems that arise during systemlevel test and evaluation or later during the porting of the UoC. Regardless of whether the problem was due to an improper implementation of a Conformance Requirement or a problem with the Technical Standard, the problem solution required a modification to the UoC. The Software Supplier will provide an impact assessment report resulting from the modification to the Software Verification Package contents and resubmit the Software Verification Package with the changes incorporated. The VA will conduct an assessment on the report and updated submittal, determine whether further information or testing is required, evaluate any additional data or test results, and upon satisfactory results issue a delta-verification assessment report to the CA. The CA will re-issue the FACE Conformance Certificate. 44

45 Recertification/Modification of a Certified UoC Software Modification Releases are the result of any software changes made to the UoC not associated with a conformance maintenance release. The requirement for change could be due to Software Supplier market assessment or customer demand. For these changes, the Software Supplier must submit an impact report on the change(s) and submit a new Software Verification Package. The VA will determine the level of verification required. Upon successful conformance verification, the process will follow the certification process for new UoCs. 45

46 Recertification/Modification of a Certified UoC Renamed UoC If a FACE Certified UoC is to be renamed, with no change to the UoC, the Software Supplier may request such changes at any time by contacting the CA. The Software Supplier will be required to affirm to the CA that there has been no change other than name. Renaming a UoC will additionally require an update of FACE Registry information. Modification of FACE Registry Information If the Software Supplier wishes to modify information in the FACE Registry that has no impact on the conformance of the FACE Certified UoC such as contacts, UoC descriptive text, or other metadata the Software Supplier may request such changes at any time by contacting the LA. 46

47 Recertification/Modification of a Certified UoC Other Modifications Except where specifically stated in this document, any other variant of a FACE Certified UoC which is deemed by the CA to have a material effect on the conformance of the UoC to the Technical Standard constitutes a new UoC, which will be subject to the full conformance verification and certification processes. 47

48 Problem Reporting (PR) Process A Problem Report (PR) may be submitted against any problem encountered with the Technical Standard, FACE Conformance Test Suite(s), or conformance verification and certification processes that inhibit or will inhibit the conformance effort. The PR submitter will utilize the Consortium-wide Problem Report/Change Request (PR/ CR) process. 48

49 Problem Reporting (PR) Process The types of problems that may be reported include: Errors, conflicts, or ambiguities in the Technical Standard. Errors in the test suite(s) used to assess conformance with the specifications; specifically, in the FACE Conformance Test Suite(s), or other test suites referenced by the FACE Conformance Program (if any). Errors, conflicts, or ambiguities in the set-up requirements of the FACE Conformance Test Suite(s) may also be addressed through PRs. Errors, conflicts, or ambiguities in the conformance verification and certification processes; specifically, those related to the Registration process, legal agreements, and completion of the Software Verification Package (set of artifacts to support conformance verification). 49

50 Problem Reporting (PR) Process A submitter may file PRs to address issues that arise for any of these items. The identity of the submitter will be protected throughout this process. The PR is used specifically for the types of errors listed above which are inhibiting the conformance effort. For general questions on the conformance verification and certification processes, running the test suites, or other problems not covered above, the CA can provide information on obtaining further assistance. Issues not resolved through this assistance may be followed-up with a PR. 50

51 Problem Reporting and Approved Corrections Flow Submitter PR/CR Administrator Steering Committee FTG FACE CCB Product CCB Entry INITIATED INITIATED Triage More Information RETURNED REJECTED Provide Feedback Confirm Rejection REJECTED RECOMMEND REJECT Proposal PROPOSED Yes Satisfied No Rejected Approve Correction Schedule Proceed with New Information Publish Correction File Appeal No Satisfied Yes Proceed With Correction 51

52 Appeals Occasions that may give rise to an appeal include, but are not limited to, the following: The Software Supplier disagrees with the resolution of a Problem Report (PR). The Software Supplier disagrees with the CA s grounds for denying the award of certification or the removal of certification. The Software Supplier of a FACE Certified UoC disagrees with a formal notification from the CA of the need to rectify a nonconformance. The incidence of a UoC having been found to be nonconformant will be handled on a case-by-case basis under Conformance Maintenance Release policies. Appeal requests will be made to the Steering Committee in writing within 90 calendar days of issue occasion. 52

53 Appeals Submitter Steering Committee Request Appeal Appeal Granted Yes Appeal Accepted Changes Made No Appeal Denied No Changes Made 53

54 Audits The primary purpose of the audit is to ensure the overall integrity of the FACE Conformance Program and the software contained in FACE Product Repositories. An audit may be conducted on a Software Supplier, a VA, or the CA. Audits will be conducted or directed by the Steering Committee (SC) when deemed necessary. The SC designates an auditor to perform an audit. The auditor may request the applicable information from any of the following entities in order to conduct the audit: Software Supplier Verification Package and configurationmanaged records VA verification records CA certification records 54

55 55 Audits