STRATEGIC & INNOVATIVE PRACTICES PORTFOLIO, PROGRAMS & PROJECT (PPP) MANAGEMENT FOR REDEFINING INDIA

Transcription

1 Title Reimagining Large Scale Beta Programs Theme STRATEGIC & INNOVATIVE PRACTICES PORTFOLIO, PROGRAMS & PROJECT (PPP) MANAGEMENT FOR REDEFINING INDIA Keywords Beta, Large Scale, feedback, innovative practices, process Abstract At Adobe Digital Marketing Business Unit, Beta Programs are strategic to the Business. They involve more than 1000 registered Enterprise Customers participating almost round the year. The challenge is to keep a large community of customers and partners engaged and committed so that we get the desired KPI's met. This paper will talk about the Reimagining approach taken by Beta Program Management team to overhaul the effectiveness and engagement for the program. The paper will highlight issues observed in running large scale Beta programs and how they were mitigated by ensuring a buy-in from multiple stakeholders. The whole program is broken down into 4 phases Planning, Recruitment, Execution and Closing and issues and mitigation plan is provided for each of these four phases. The learnings from here can be applied to any company trying to make their public or private Beta programs successful. The paper will lastly also provide best practices for laying down Beta Program Life Cycle (and how it should be nested within the product release cycle).

2 Table of Contents Introduction... 3 Problems with Running Successful Beta... 3 Beta Program Overview... 5 Planning Phase... 6 Recruiting Customers... 7 Execution... 7 Closure... 9 Tiered Structure of the Beta Program Role of Enablement (Field Teams) in Beta program Best Practices Conclusion References... 18

3 Introduction Adobe is essentially a Product Development company and therefore Beta programs are its lifeline. They blend well with the Lean approach of developing in small increments and validating often with the customers. Having a thriving and effective Beta programs are critical to the success of Adobe products and in essence success of its customers. In terms of categorization there are two types of Beta programs Implementation ( which require code changes by customers) and non-implementation Beta s which do not require any code changes by external participants. This paper explores the experience of running very large scale Beta programs (private) and the learning we have derived from them. A private Beta is one which is a closed or an invite only Beta program. This paper covers B2B private Beta programs. B2C private Beta are not covered. A large Beta in the context of Adobe is one where we have more than a 1000 customers and partners participating at the same time. We also do public Beta (open to all) however public Beta are outside the scope of this paper. Problems with Running Successful Beta During the course of running Beta program over the last few years following issues were observed (from surveys, team feedback, data collected). Later part of the paper will talk about the ground up reimagining and process which was put in place to address most of these issues. The issues are categorized under phases (the phases are described in more detail later) Planning Phase Beta plan not committed as part of project plan commit Field teams not aware of Beta plan Recruitment Phase Keeping Beta participation bar low No participant curation done on ongoing basis

4 Execution Phase Low motivation of Beta participants Low engagement of Beta participants No bidirectional visibility on what's happening on Beta issues reported Poor response rate from internal teams on responding to forums Customers had issues installing on-prem components themselves Beta cycle starting late overlapping with end game Language barriers Closing Phase Success criteria of the program not defined upfront Sudden engagement drop at closure of the program

5 Beta Program Overview Project releases are typically once in 3 months for SaaS and 6 months for On-Prem. There are additional monthly minor releases. Beta programs are aligned with major releases. UI Betas (for SaaS offerings) are typically 6 week long (Figure 1) whereas implementation betas which require some implementation effort at customer/partner side are typically 12 weeks long (Figure2) Figure 1 Figure 2

6 In terms of Beta lifecycle it consists of 4 distinct phases. Here is a description of these 4 phases of Beta. These four phases are now defined in a way that addresses some of the problem areas described earlier. Planning Phase This is the phase where product team brainstorms the next Beta program. Following aspects are deliberated in this phase a. What are the objective of doing Beta Different people in the team have different opinions or takeaways from a Beta program. It is always recommended that teams write down what they want to achieve in the given Beta. Over the years we have settled down to following as the typical objectives of doing Beta program Solicit Feedback on New Features Feedback on Usability Issues Participants get early access to software Enablement of field (customer facing teams) Collect Enhancement Requests Gather Participants References/Quotes/Collaterals for sales b. Program Overview Is the description of Beta itself. Product Managers are typically asked to describe what the Beta program is all about. c. Product Overview Next is the description of the product if it is the first launch or if it is a follow on release and what are the customer benefits being made available with the release. d. Roles and Responsibilities of various stakeholders need to be defined and agreed upon upfront. e. Criteria for Success needs to be defined such that at the time of program exit we have a benchmark available to say whether or not program objectives were met. f. Define how Beta support process will work g. Define Beta schedule and deliverables h. Define what is customer/partner (external participant) motivation to participate i. Legal review Each Beta cycle some change happens so it is always good to review existing disclosure/confidentiality of information templates with legal and get their explicit sign offs.

7 Recruiting Customers Some amount of curation and matchmaking is required to get the right customers into the Beta program. Competitors should not be able to get in and at the same time sales team should be able to build a strong pipeline based on Beta participation. Customer needs to be mapped to Backlogs. Product Managers should be able to define 2/3 customers for Epics they would like feedback to be received on. This allows for tighter follow ups and hi-touch engagement with these select few even though program is available for hundreds of others Product Managers, Sales, Enablement, Marketing, Engineering can recommend participants for Beta programs. Product Management takes the final call on who gets the access to the program. Stage 1 Beta Program Managers prepare the initial list of participants from the previous release engagement data + add participation requests which came in late into the cycle + any new licensed customers and hand it over to Product Managers Stage 2 Product Managers map the customers to release themes based on customer insights (what features they use, what enhancements request they have logged in the past, which release themes customers are most likely to benefit from) Stage 3 Product Marketing/Product Management validate the list before the registration invites go out to weed out competitors, customers who explicitly asked to be dropped from Beta. The set left after Step 3 is the set of invitees for the Beta program. During the course of the program it is still possible that folks can get added. Execution This is the phase where actual Beta program is delivered a) Kick-Off Call All Beta participants (yes we maintain transparency) are invited to Beta kick off call. This is an expectations settings call where expectations from both sides are set and mechanics of the program are discussed. b) External Forums Forums are created for external participants to post questions/enhancement requests

8 c) Beta Calls Beta calls are done with the participants on weekly basis. Following items are covered in the call o Introduce new features and give a demo o Field feedback on previous weeks testing assignment o Assign new features to participants for coming week d) Beta loads may be uploaded on the forums (in case of on-prem betas). Each Beta load accompanies a set of documentation and release notes. If a feature is available for the first time, product teams also upload screen share videos demo ing the feature to customers. This greatly helps the understanding e) Assignments Assignments are posted every week/alternate week which ask external participants to test and provide feedback on specific workflows. This ensure teams get the best out of customers and there is no ad-hoc ness in the Beta testing process. f) A beta blitz is an opportunity for the Adobe product team and internal testers to come together in a dedicated working session and test the beta product. This ensures that the product going out for the external beta has most frequent and obvious issues already discovered. A test plan is circulated to the participating teams before the blitz starts. Questions can be asked over the phone and in the Adobe Connect chat pod and bugs and feature enhancements can still be submitted through the Beta Feedback Tool. The product teams are available on a dedicated basis to answer questions. g) Bi-directional Visibility is provided both to external participants as well around status of issues/bugs reported so participants get a comfort feel that their issues are being listened to and being acted upon. h) Measuring Participant Engagement during the course of program, infographics are created to demonstrate participant engagement. An example is provided in Figure 3

9 Figure 3 i) Pre and Post Call Surveys Surveys are conducted before every beta call to test the effectiveness of previous call and to determine if any course correction needs to be made during the Beta calls. Similarly Exit surveys are done to prepare better for the next beta call. Pre-post call surveys get better responses because of the captive audience and also because Beta content is still fresh on the minds of participants. Closure This is the program wrap up phase. Active beta calls have ended and Beta Program Management team starts preparing for the following wrap up activities 1. Exit surveys are conducted at the end of Beta program to determine Gap Analysis. Gap Analysis determines external participant s satisfaction against important elements of the program. A sample Gap Analysis is provided in Figure 3 and Figure 4

10 Figure 3

11 Figure 4 2) Since different customers are at different stages of testing/engagement the Beta closure cannot be a cold closure. For this purpose when the Beta program closes Enablement program takes over these customers till they are successful in adopting new features coming in the release. 3) Retrospective survey is done with internal stakeholder 4) A Beta Exit Report is prepared which included findings from external participant s feedback and internal stakeholder feedback. Tiered Structure of the Beta Program Over the years we have realized that one size fit all does not work. Different stakeholders of the program need to be treated differently during the course of beta lifecycle. Therefore now we have designed a 3 tier structure for the program

12 Tier 1 These are internal Adobe teams who are not part of the product team which is delivering beta but are either internal customers of the product or have some stake in making the product successful with customers. These participants get the first access to beta program. For Tier 1 quality bar is relatively lower and focus is to get early feedback on feature/documentation/usability before external world gets their hands on the product. Tier 2 These are the set of External participants who had good engagement score in the previous cycle. These external participants (customers and partners) showed willingness to test early builds and provided good feedback. These customers and participants are provided the next level of access and are also referred to as Hi-Touch customers. Product Engineering likes being deeply engaged with this set. They are not only good sounding boards but are willing to act as references to get new deals. Tier 3 These are the set of External participants who want a stable release before they want to spend any cycles on testing and their engagement in previous cycle was not too great. Yet they are important because they provide upsell/cross sell opportunities. They are brought in when the product is near Feature Complete and UX complete for the release. Role of Enablement (Field Teams) in Beta program Problem Field and partners do not have deep-technical knowledge about the new product/ solution releases soon enough to be able to do customizations on the product/ reference site and build quick Proof of Concepts. Need There is a need for technical deep-dive and learning content for field, training team and partners on the new product releases. This is required to ensure they are comfortable in taking the new product through demo/ customization/ implementation for customers. There is a need to ensure the solution to enablement problem is scalable and content can be updated incrementally as well.

13 Solution Recruit the right people for beta program and bring them in the beta program in multi-tiered fashion (as part of Tier 1) as described previously. These are the people whose regular job requires them to be on top of what the new product will be. We need to bring such people on the product release beta program so that they can lay their hands on the early builds and provide feedback to the product in works. It also enables beta participants to submit their enhancement requests (which product management can align with the roadmap and provide a view on that). The beta program is the first step in ensuring we start a strong engagement with our participants. We need to ensure that the engagement continues and doesn't end with wrapping up of beta program. Participant recruitment is extremely important here as unmotivated participants will not be able to add value. Best Practices Access to Test Scenarios It is very helpful for the beta testers to have access to a set of test scenarios to guide them through testing the new features of the release. This can be as high level or detailed as is available when the beta program starts. Quality Engineering can be a great resource for this material. Documentation A link or PDF of the beta documentation is needed prior to going to beta. Major updates to the documentation should be communicated regularly via the forums/beta calls to external and internal participants. Quality documentation directly impacts the motivation and engagement of partners. Beta Calls Beta calls should be held with the group of testers on a schedule deemed appropriate for the release. Typically they are weekly, but based on the length of the beta cycle, they may be every other week until the release date nears then move to weekly. The product team should attend every beta call or delegate to someone to represent them. At times, based on participation or feedback needed by the product team, it may be necessary to hold 1:1 calls with the individual beta testers.

14 Get commitment from external participants Make Beta programs coveted for participants. Participants must demonstrate commitment in terms willingness to commit time upfront and then their participation and engagement is monitored Keeping Participants engaged through the course of Beta program Provide them with clear instructions on what to test, what has changed, what s new, what s fixed, etc. Provide them with any workarounds, known issues, etc. Respond to their incidents within 48 hours. Follow up on their incidents and let them know what has happened with them. Follow up on feature requests and let them know what has happened with them. Regular conference calls. Product Training/spotlights. Communication, communication, communication! Beta Dashboards We have our own in-house Beta Dashboards which pull out data from relevant sources and present in a format which provides a periodic picture of where we are with the program. An example is provided below

15 Beta Feedback Tool We have Beta Feedback buttons incorporated in the UI of the products so that customers can send the feedback/issues directly from the product interface

16 A Beta Program is successful when Beta Testers used the product as they would in their unique testing environment so as to avoid the issue in their live environment once the product is released. The beta testers logged any bugs encountered and were able to create feature requests for Product Management to consider for future releases which could benefit the product as a whole. All expected new features are present and the product performs the function it was intended to perform. Development is able to fix a majority, if not all, the higher priority reported issues prior to General Availability. Marketing has a pool of customers to use as reference sites. Overall, the product is cleaner and more stable than it was prior to going to beta.

17 Conclusion One size does not fit all when it comes to Beta programs. Every company is unique in terms of its employees, culture and its customers. However knowing what and how other companies have successfully delivered can be an inspiration/act as reference/used as a framework/used as template. Choice is yours on what you want to take away and change what might not have been working. Here is a quick summary of how Beta programs have been reimagined 1. We now have multi-tiered Beta Programs ( to address engagement issue) 2. We now have Customer facing internal teams part of the Beta programs 3. We now have dedicated Beta Program Managers who are responsible for End to End execution and success of Beta programs 4. We now Nest Beta programs within product release and Beta plans are signed off during project release commit Beta programs are now first class citizens 5. We now provide Bi-directions visibility to participants 6. We have quantitative measures in place ( e.g. Gap Analysis) to determine areas of improvement 7. We have improved the timing of the Beta programs by starting them early and not taking them too close to the end game 8. We now do hosted Beta wherever possible to reduce customer friction and ramp-up time 9. We now have a participant curation process, which raises the participation bar high and identifies few hi-touch customers who will provide the most valuable return and will be most likely to be engaged 10. We now have clearly defined success criteria during planning so that at the time of exit the team can be objectively measured Areas still needing improvement all said and done there are still a few areas which need to be addressed Language Barrier we still to address the issue that customers in geographies like Japan have lesser Beta engagement as they do not understand and read English very well. We do have localized documentation but that is available later in the cycle. We are planning to have Evangelists help us out in the next cycle to address this concern and also to have the documentation localized earlier in the cycle.

18 References Internal company content management system