Yocto Project Experience: Continuous Integration

Transcription

1 Yocto Project Experience: Continuous Integration Mark Hatle Senior Member of Technical Staff Wind River Edinburgh, Scotland 23 Oct 2013

2 Agenda Our experiences as an OSV, productizing the Yocto Project Software Lifecycle Big-Bang Example Continuous Integration Example Our recommendation 2 Yocto Project The Linux Foundation

3 Productization What does it take to turn the Yocto Project into a commercial product? Yocto Project The Linux Foundation

4 Yocto Project Productization What does an OSVs customer s require? Up-to-date kernel Up-to-date toolchain Up-to-date userspace One or more specific BSP (hardware support) Quality improvements Timely support 4 Yocto Project The Linux Foundation

5 Yocto Project Productization Up-to-date When the Yocto Project release is complete, it is generally considered to be very Up-to-date (nothing older than 6 months) Up-to-date in the customer world is roughly nothing older than one generation, or months Toolchain is at the current community supported version Kernel is at the generally accepted stable version (LTSI or otherwise) 5 Yocto Project The Linux Foundation

6 Yocto Project Productization Hardware Support Customers require the hardware of their choice to be supported. Generally new hardware requires newer versions of the Linux kernel Semiconductor specific optimizations for toolchains, drivers and other components are often required. 6 Yocto Project The Linux Foundation

7 Yocto Project Productization Quality Anything that is released from an OSV needs to be at or better than Open Source quality Requires significant test resources (people and machines) 7 Yocto Project The Linux Foundation

8 Yocto Project Productization Timely support When something doesn t work, the OSV is expected to be the expert on the problem! The OSV must understand the system as a whole The OSV must work with the community to find existing fixes The OSV must work with the community suggest new fixes 8 Yocto Project The Linux Foundation

9 Software Lifecycle How to manage the software lifecycle? Yocto Project The Linux Foundation

10 Software Lifecycle Yocto Project Lifecycle Commercial Product Lifecycle Real World Examples 10 Yocto Project The Linux Foundation

11 Yocto Project Lifecycle 6 month development cycle 4 4 week development milestones 5 th milestone is stabilization Maintenance releases managed for roughly 1 year 11 Yocto Project The Linux Foundation

12 Yocto Project Lifecycle 4-4 week development milestones 12 Yocto Project The Linux Foundation

13 Commercial Product Lifecycle Examples Big-Bang Start with community release Add missing requirements Add new value-add features QA/Verify OSS QA/Verify new components QA/Verify value-add features Work to resolve bugs internally Release to customers Maintain Release (5-10 years) Continuous Integration Work in parallel with community Influence community work Add new value-add features QA/Verify OSS QA/Verify value-add features Work with community to fix bugs Release to customers Maintain Release (5-10 years) Examples assume approx month release cycles 13 Yocto Project The Linux Foundation

14 Big-Bang Lifecycle Example Big-Bang refers to the work Starts with a large amount of community software Need to learn how it works Learn what required product features need to be implemented More of the traditional approach Follow Open Source 14 Yocto Project The Linux Foundation

15 Big-Bang Lifecycle Example Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Yocto Project Update Update EOL Commercial U1 U2 U3 U4 U5 U7 Customer Adoption No opportunity to influence design, must follow Enhancements should be contributed to the next version, and backported Bugs found may or may not have been found by community, may require additional resources to resolve 15 Yocto Project The Linux Foundation

16 Big-Bang Lifecycle Example Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Yocto Project Update Update EOL Commercial U1 U2 U3 U4 U5 U6 At start of commercialization, product components are up to 6 months old By the time of release, it s nearly 12+ months old Has a shelf life of only 6-12 months from release Decision extend shelf-life or uprev? Customer Adoption 16 Yocto Project The Linux Foundation

17 Big-Bang Lifecycle Example Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar cto Project Update Update EOL Commercial U1 U2 U3 U4 U5 Extend shelf life? Customer Adoption Commercial U1 U2 U3 U4 Customer Adoption Update kernel, toolchain, BSPs and other critical elements Backport additional select features 6 months work, only gains 6-12 months Customer Adoption No community help 17 Yocto Project The Linux Foundation

18 Big-Bang Lifecycle Example an-mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar oject Update Update EOL Commercial U1 U2 U3 U4 U5 Customer Adoption Yocto Project Update Update EOL Uprev? Rebase changes (15-45 days) Add commercialization time Commercial Finish date has now slipped back a bit more Still following, not leading U1 U2 U3 U4 Customer Ado 18 Yocto Project The Linux Foundation

19 Commercial Product Lifecycle Examples Big-Bang Start with community release Add missing requirements Add new value-add features QA/Verify OSS QA/Verify new components QA/Verify value-add features Work to resolve bugs internally Release to customers Maintain Release (5-10 years) Continuous Integration Work in parallel with community Influence community work Add new value-add features QA/Verify OSS QA/Verify value-add features Work with community to fix bugs Release to customers Maintain Release (5-10 years) Examples assume approx month release cycles 19 Yocto Project The Linux Foundation

20 Continuous Integration Lifecycle Example Continuous Integration refers to tracking and contributing to the community development Work with the community on development Learn capabilities and feature deficits as development continues Ability to influence the community by discussing requirements and/or providing patches Ability to monitor OSS quality over a longer period of time Requires resources to follow the community Lead with the OSS community 20 Yocto Project The Linux Foundation

21 Continuous Integration Lifecycle Example Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Yocto Project Commercial Update Update EOL U1 U2 U3 U4 U5 U6 Customer Adoption Ability to identify issues and work with the community to resolve them Enhancements can be contributed during development Bugs can be filed with the community and worked on as a group 21 Yocto Project The Linux Foundation

22 Continuous Integration Lifecycle Example Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Yocto Project Commercial Update Update EOL U1 U2 U3 U4 U5 U6 Customer Adoption During commercialization components are current By the time of release, it s only 6-7 months old Has a shelf life of months from release No reason to extend the shelf-life! 22 Yocto Project The Linux Foundation

23 Continuous Integration Lifecycle Example Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar cto Project Commercial Update Update EOL U1 U2 U3 U4 U5 U6 Customer Adoption Uprev! Yocto Project Update Update EOL Yocto Project Update Update EOL No rebase required Ramp up developers Commercial U1 U2 U3 U4 U5 U6 Customer Adoption 23 Yocto Project The Linux Foundation

24 Real World Examples Big-Bang Took a large product team 6 months to commercialize Required 6 one month cycles to add enhancements and backport upstream features First cycle was devoted to investigation Required significant developer resources Release was approx time of next YP release 24 Yocto Project The Linux Foundation

25 Real World Examples Big-Bang Extended life Required 6 months development to update kernel, BSPs, toolchain and other customer essential systems Required roughly the same development effort as a new product Release occurred at appox time of EOL of the base Yocto Project release 25 Yocto Project The Linux Foundation

26 Real World Examples Big-Bang Uprev Requires 45 days (of one engineer) to update the tree 2 Yocto Project releases. This 45 days simply enabled the main development team. Projected to required 6 months of development to commercialize and add new features, QA, etc. Release was now after the next YP release 26 Yocto Project The Linux Foundation

27 Real World Examples Continuous Integration Work done in parallel with the community. Able to ramp up a small team to full team over the course of development. As bugs were found, many filed with the community and fixed in a timely manner. (Many critical bug fixes were submitted to the community.) As missing features were identified, worked with the community to implement functionality. 27 Yocto Project The Linux Foundation

28 Real World Examples Continuous Integration Required 1 full time resource to manage integration and tracking of the community. Resource was the go-to person for questions about community quality, bug triage, etc. Continuous uprev averaged every 1-2 weeks for the first 4 milestones. Bug fixes and QA took most of the two week time. (Longer than we expected.) Unexpected change in the 5 th milestone caused a single 4 week integration cycle. Expect weekly uprevs after product release for point fixes. 28 Yocto Project The Linux Foundation

29 Real World Examples Continuous Integration Estimated to be the same amount of improvement and commercialization required Smaller teams required, as less unexpected reactionary work was required 29 Yocto Project The Linux Foundation

30 Recommendation What should you do? Yocto Project The Linux Foundation

31 Recommendations Semiconductor Mfg Kernel/BSP support big-bang address customers wanting a stable approach continuous integration keep changes timely, and ready to release for the next stable release May depend on chip market, release schedules and customer demand 31 Yocto Project The Linux Foundation

32 Recommendations OSV Use continuous integration. Quicker time to market, more up-to-date software, and longer shelf-life. Should allow time to sync to semiconductor and customer requirements. 32 Yocto Project The Linux Foundation

33 Recommendations ISV / Application Developers Follow the YP versions your customers need. Most likely to follow the stable release, but for large applications the continuous-integration model may make sense. 33 Yocto Project The Linux Foundation

34 Recommendations Device Developers Look at what your needs are. If you don t need work-in-progress features, it s better to start with a stable release! If you expect to be updating the OS over the life of the product, continuous integration may be useful. 34 Yocto Project The Linux Foundation

35 Thank you for your participation!