Challenge to add OpenStack API Validation Framework

Transcription

1 Challenge to add OpenStack API Validation Framework Ken ichi Ohmichi NEC-Soft, Ltd.

2 Agenda OpenStack OpenStack community development OpenStack functions through RESTful API Issues of RESTful APIs Fail-Fast architecture API validation framework Development process Schedule for Havana Page 2 NEC Corporation 2013

3 OpenStack The Open Source Cloud Operating System Controls virtual machines, storage/networking resources. Consists of multiple functional components. Compute: Nova VM image: Glance Identity: Keystone Networking: Quantum (will be renamed) Block storage: Cinder Object Storage: Swift Dashboard: Horizon Metering: Ceilometer Orchestration: Heat

4 OpenStack community development The community is very active. OpenStack community consists of many developers, users and operators. They are cooperating each other on mailing-list, IRC and development tools such as Gerrit. The community releases new version of OpenStack every six months, and the functions increases by each version.

5 OpenStack community development The development routine 1. Submit a bug-report / blueprint to OpenStack Launchpad. 2. git clone necessary source code from OpenStack github. 3. Modify the source code as you like. 4. git commit with the id of bug-report / blueprint. 5. git review for posting a patch into Gerrit system. 6. (Reviewers) review the patch and submit their feedbacks with score on Gerrit. Score +2: Approve Score +1: Looks good do me, but someone else must approve Score -1: I would prefer that you didn t merge this. Score -2: Do not merge 7. Apply reviewer s feedback to your patch, and retry git review for reposting it. 8. (Reviewers) re-review the patch. When core reviewer marks it as +2, the patch is merged to OpenStack github. Congratulations

6 OpenStack functions through RESTful API Control functions with dashboard or command line. These tools uses RESTful API in the background. Dashboard Create vm by pushing here. Control functions through RESTful API directly. EX) We can create a virtual machine by sending API request with following parameters to with POST method. The parameters can be specified with JSON/XML format. { } "server": { "flavorref": "1", "imageref": "70a599e0-31e7-49b7-b f441e862b", "name": "new-server-01"

7 Issue/Solution of RESTful APIs Issue Not all API parameters are completely validated, so some API operations execute without parameter validations. Solution By validating every API parameters just after receiving API request, it makes better OpenStack quality. Because the basic design of OpenStack is Fail-Fast architecture. Page 7 NEC Corporation 2013

8 Fail-Fast Architecture Fail-Fast system does not retry/cleanup if something wrong happens. Ex) If something wrong happens after adding a DB record during one API operation, it does not remove the added record. As the result, incomplete record remains in DB. The antonym is Fail-Safe. The implementation cost of Fail-Safe tends to higher: What should be cleaned up when error handling? What should it do when double errors (something wrong in error handling)? On Fail-Fast system is easy-debugging because developers can know what happens easily. Op-01 add Op-01 add Op-02 Record-01 remains Op-02 del Record-01 removed Op-03 Op-03 Fail-Fast Fail-Safe

9 API validation framework: Goals I have proposed API validation framework to OpenStack community for comprehensive validation. This framework goals are: To validate every API parameter. To return an error response before API operation, if invalid API parameter. To unify the error message format of the response, if the same cause. ex).. is not string.", ".. is not integer." By comprehensive validation just after receiving API request, it makes better OpenStack quality because it can reduce incomplete situations like incomplete record remaining. Merit Page 9 NEC Corporation 2013

10 API validation framework: Validation Point (discussing now) The API validations are executed before each API method. The proposed framework is implemented with jsonschema library. This framework can validate requests of both XML and JSON, because jsonschema handles dict data after JSON/XML deserialization. I heard often Can this framework handle XML request? due to its library name. The answer is Yes, it can. Nova (2) Execute the selected validation Client request JSON/XML Deserialization Router API Validation API Validation API Validation (1) Select a validation and a method corresponding to API API method API method API method (3) Execute the selected method Page 10 NEC Corporation 2013

11 API validation framework: API Schema (discussing now) API schema = API parameter data format. Ex) The API schema of the Change Password API. { 'type' : 'object', 'properties' : { 'changepassword': { 'properties' : { 'adminpass': {'type': 'string', 'required': True}, } } } } Need many API schema definitions because OpenStack has many APIs (hundreds). Page 11 NEC Corporation 2013

12 API validation framework: Merits In addition to the goals, there are two merits of the framework. Clarification of API Parameter Definitions The type and acceptable range and length of each parameter are clarified in the API schema definitions. By defining the schemas of every API, developers will be able to create applications more easily using Nova APIs. Cleaned-up Code The amount of Nova code can be reduced by merging validations and error response methods. Ex) the min_count parameter of create a virtual machine instance API try: min_count = int(str(min_count)) except ValueError: msg = _('min_count must be an integer value') raise exc.httpbadrequest(explanation=msg) if min_count < 1: msg = _('min_count must be > 0') raise exc.httpbadrequest(explanation=msg) 'min_count': {'type': 'integer', 'minimum': 1}, Page 12 NEC Corporation 2013

13 Development process March, 2013 Created some prototype for Nova. Proposed the prototype to OpenStack community. Could listen positive responses. April, 2013 Had a session of OpenStack Design Summit for this framework in Portland, Ore. Could make consensus for its purpose. Could know it is good that the first target of this framework is Nova v3 API. Nova v3 API will be implemented for Havana, and compatibility issues do not happen even if applying strict validation.

14 Schedule for Havana Now we are discussing what is best validation mechanism. We need more time for consensus. We have 4 months until Havana-3 meaning deadline for new code. Make Framework Core Patches. Make API schema Definition Patches. This is very hard work, because Nova has many APIs. Merge by middle of August. Deadline for new code Framework core h1 API schema definition h2 h3 rc1 rc2 rc3 Havana Apr May June July Aug Sep Oct we AD 2013 Page 14 NEC Corporation 2013

15 OpenStack community development is hard work, but contribution to the community is very interesting because the community consists of skillful developers and they give good questions, advices, and answers. I am glad if many people are into OpenStack community. Thanks

16 Page 16 NEC Corporation 2013

17 Related URLs Blueprint Nova API Validation Framework Framework core patch for Oslo Framework core for Nova Nova v3 API How to contribute to OpenStack Page 17 NEC Corporation 2013