Yocto Project Experience: Continuous Integration Mark Hatle Senior Member of Technical Staff Wind River Edinburgh, Scotland 23 Oct 2013
Agenda Our experiences as an OSV, productizing the Yocto Project Software Lifecycle Big-Bang Example Continuous Integration Example Our recommendation 2 Yocto Project The Linux Foundation
Productization What does it take to turn the Yocto Project into a commercial product? Yocto Project The Linux Foundation
Yocto Project Productization What does an OSVs customer s require? Up-to-date kernel Up-to-date toolchain Up-to-date userspace One or more specific BSP (hardware support) Quality improvements Timely support 4 Yocto Project The Linux Foundation
Yocto Project Productization Up-to-date When the Yocto Project release is complete, it is generally considered to be very Up-to-date (nothing older than 6 months) Up-to-date in the customer world is roughly nothing older than one generation, or 12-18 months Toolchain is at the current community supported version Kernel is at the generally accepted stable version (LTSI or otherwise) 5 Yocto Project The Linux Foundation
Yocto Project Productization Hardware Support Customers require the hardware of their choice to be supported. Generally new hardware requires newer versions of the Linux kernel Semiconductor specific optimizations for toolchains, drivers and other components are often required. 6 Yocto Project The Linux Foundation
Yocto Project Productization Quality Anything that is released from an OSV needs to be at or better than Open Source quality Requires significant test resources (people and machines) 7 Yocto Project The Linux Foundation
Yocto Project Productization Timely support When something doesn t work, the OSV is expected to be the expert on the problem! The OSV must understand the system as a whole The OSV must work with the community to find existing fixes The OSV must work with the community suggest new fixes 8 Yocto Project The Linux Foundation
Software Lifecycle How to manage the software lifecycle? Yocto Project The Linux Foundation
Software Lifecycle Yocto Project Lifecycle Commercial Product Lifecycle Real World Examples 10 Yocto Project The Linux Foundation
Yocto Project Lifecycle 6 month development cycle 4 4 week development milestones 5 th milestone is stabilization Maintenance releases managed for roughly 1 year 11 Yocto Project The Linux Foundation
Yocto Project Lifecycle 4-4 week development milestones https://wiki.yoctoproject.org/wiki/yocto_1.5_schedule 12 Yocto Project The Linux Foundation
Commercial Product Lifecycle Examples Big-Bang Start with community release Add missing requirements Add new value-add features QA/Verify OSS QA/Verify new components QA/Verify value-add features Work to resolve bugs internally Release to customers Maintain Release (5-10 years) Continuous Integration Work in parallel with community Influence community work Add new value-add features QA/Verify OSS QA/Verify value-add features Work with community to fix bugs Release to customers Maintain Release (5-10 years) Examples assume approx 12-18 month release cycles 13 Yocto Project The Linux Foundation
Big-Bang Lifecycle Example Big-Bang refers to the work Starts with a large amount of community software Need to learn how it works Learn what required product features need to be implemented More of the traditional approach Follow Open Source 14 Yocto Project The Linux Foundation
Big-Bang Lifecycle Example Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Yocto Project Update Update EOL Commercial U1 U2 U3 U4 U5 U7 Customer Adoption No opportunity to influence design, must follow Enhancements should be contributed to the next version, and backported Bugs found may or may not have been found by community, may require additional resources to resolve 15 Yocto Project The Linux Foundation
Big-Bang Lifecycle Example Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Yocto Project Update Update EOL Commercial U1 U2 U3 U4 U5 U6 At start of commercialization, product components are up to 6 months old By the time of release, it s nearly 12+ months old Has a shelf life of only 6-12 months from release Decision extend shelf-life or uprev? Customer Adoption 16 Yocto Project The Linux Foundation
Big-Bang Lifecycle Example Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar cto Project Update Update EOL Commercial U1 U2 U3 U4 U5 Extend shelf life? Customer Adoption Commercial U1 U2 U3 U4 Customer Adoption Update kernel, toolchain, BSPs and other critical elements Backport additional select features 6 months work, only gains 6-12 months Customer Adoption No community help 17 Yocto Project The Linux Foundation
Big-Bang Lifecycle Example an-mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar oject Update Update EOL Commercial U1 U2 U3 U4 U5 Customer Adoption Yocto Project Update Update EOL Uprev? Rebase changes (15-45 days) Add commercialization time Commercial Finish date has now slipped back a bit more Still following, not leading U1 U2 U3 U4 Customer Ado 18 Yocto Project The Linux Foundation
Commercial Product Lifecycle Examples Big-Bang Start with community release Add missing requirements Add new value-add features QA/Verify OSS QA/Verify new components QA/Verify value-add features Work to resolve bugs internally Release to customers Maintain Release (5-10 years) Continuous Integration Work in parallel with community Influence community work Add new value-add features QA/Verify OSS QA/Verify value-add features Work with community to fix bugs Release to customers Maintain Release (5-10 years) Examples assume approx 12-18 month release cycles 19 Yocto Project The Linux Foundation
Continuous Integration Lifecycle Example Continuous Integration refers to tracking and contributing to the community development Work with the community on development Learn capabilities and feature deficits as development continues Ability to influence the community by discussing requirements and/or providing patches Ability to monitor OSS quality over a longer period of time Requires resources to follow the community Lead with the OSS community 20 Yocto Project The Linux Foundation
Continuous Integration Lifecycle Example Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Yocto Project Commercial Update Update EOL U1 U2 U3 U4 U5 U6 Customer Adoption Ability to identify issues and work with the community to resolve them Enhancements can be contributed during development Bugs can be filed with the community and worked on as a group 21 Yocto Project The Linux Foundation
Continuous Integration Lifecycle Example Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Yocto Project Commercial Update Update EOL U1 U2 U3 U4 U5 U6 Customer Adoption During commercialization components are current By the time of release, it s only 6-7 months old Has a shelf life of 12-24 months from release No reason to extend the shelf-life! 22 Yocto Project The Linux Foundation
Continuous Integration Lifecycle Example Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar cto Project Commercial Update Update EOL U1 U2 U3 U4 U5 U6 Customer Adoption Uprev! Yocto Project Update Update EOL Yocto Project Update Update EOL No rebase required Ramp up developers Commercial U1 U2 U3 U4 U5 U6 Customer Adoption 23 Yocto Project The Linux Foundation
Real World Examples Big-Bang Took a large product team 6 months to commercialize Required 6 one month cycles to add enhancements and backport upstream features First cycle was devoted to investigation Required significant developer resources Release was approx time of next YP release 24 Yocto Project The Linux Foundation
Real World Examples Big-Bang Extended life Required 6 months development to update kernel, BSPs, toolchain and other customer essential systems Required roughly the same development effort as a new product Release occurred at appox time of EOL of the base Yocto Project release 25 Yocto Project The Linux Foundation
Real World Examples Big-Bang Uprev Requires 45 days (of one engineer) to update the tree 2 Yocto Project releases. This 45 days simply enabled the main development team. Projected to required 6 months of development to commercialize and add new features, QA, etc. Release was now after the next YP release 26 Yocto Project The Linux Foundation
Real World Examples Continuous Integration Work done in parallel with the community. Able to ramp up a small team to full team over the course of development. As bugs were found, many filed with the community and fixed in a timely manner. (Many critical bug fixes were submitted to the community.) As missing features were identified, worked with the community to implement functionality. 27 Yocto Project The Linux Foundation
Real World Examples Continuous Integration Required 1 full time resource to manage integration and tracking of the community. Resource was the go-to person for questions about community quality, bug triage, etc. Continuous uprev averaged every 1-2 weeks for the first 4 milestones. Bug fixes and QA took most of the two week time. (Longer than we expected.) Unexpected change in the 5 th milestone caused a single 4 week integration cycle. Expect weekly uprevs after product release for point fixes. 28 Yocto Project The Linux Foundation
Real World Examples Continuous Integration Estimated to be the same amount of improvement and commercialization required Smaller teams required, as less unexpected reactionary work was required 29 Yocto Project The Linux Foundation
Recommendation What should you do? Yocto Project The Linux Foundation
Recommendations Semiconductor Mfg Kernel/BSP support big-bang address customers wanting a stable approach continuous integration keep changes timely, and ready to release for the next stable release May depend on chip market, release schedules and customer demand 31 Yocto Project The Linux Foundation
Recommendations OSV Use continuous integration. Quicker time to market, more up-to-date software, and longer shelf-life. Should allow time to sync to semiconductor and customer requirements. 32 Yocto Project The Linux Foundation
Recommendations ISV / Application Developers Follow the YP versions your customers need. Most likely to follow the stable release, but for large applications the continuous-integration model may make sense. 33 Yocto Project The Linux Foundation
Recommendations Device Developers Look at what your needs are. If you don t need work-in-progress features, it s better to start with a stable release! If you expect to be updating the OS over the life of the product, continuous integration may be useful. 34 Yocto Project The Linux Foundation
Thank you for your participation!