PERFORMANCE METRICS FOR THE IT SERVICES PORTFOLIO

Transcription

1 Bulletin of the Transilvania University of Braşov Series I: Engineering Sciences Vol. 4 (53) No. - 0 PERFORMANCE METRICS FOR THE IT SERVICES PORTFOLIO V. CAZACU I. SZÉKELY F. SANDU 3 T. BĂLAN Abstract: The etrics for the Software as a Service odel exists ostly fro the financial perspective and it is a though task to easure the contribution of the services that are part fro a portfolio inside this odel. During the paper, several indicators are introduced, based on the project anageent techniques, to quantify the contribution of each service to the portfolio benefits and goals. This kind of approach reveals the services relations inside the portfolio, with the possibility to estiate each service contribution to portfolio revenue in a defined point of tie. Key words: Software as a Service, key perforance indicators, etrics.. Introduction The Software as a Service (SaaS) paradig is one of the IT trend that is already ipacting the copanies daily business and represents a software delivery odel that allows copanies to rent an application under a service shape fro another copanies like Service Providers. The payent of such services can have different strategies such as onthly or yearly fee or service per usage fee. This concept becoes sustainable due to the Internet propagation which now perfors as a big IT platfor available everywhere, advancing of the virtualization technologies in close relation with the trend towards open source software and standardization and ultiate because of the pressure in the copanies to cut the IT related costs. In this landscape, for a Service Provider (SP), it is iportant to track the perforance of its services using different perspectives. Thus, tools are use in this regard, to understand in a tiely anner the eaning of perforance for each service fro the business portfolio and to react as quickly as possible to the ost urgent and iportant trends and probles. The indicators are one of the SaaS perforance etrics used to deterine service perforance but ost of the are financial- and arketing- based. The ai of this article is to introduce a set of key perforance indicators () which take into account the perforances of each service fro the SP portfolio in such a way to easure the generally strategic perforance of the services portfolio, covering the benefits that are necessary to achieve the SP goals. The recognition of the services perforance variance that ipacts the overall service portfolio strategy is possible by using the indicators proposed in this paper, that allow the ipleentation of quick action plans to attenuate the variance. Sieens Progra and Syste Engineering Braşov. Faculty of Technical and Huan Sciences, Sapientia Hungarian University of Transilvania. 3 Faculty of Electrical Engineering and Coputer Science, Transilvania University of Braşov.

2 00 Bulletin of the Transilvania University of Braşov Series I Vol. 4 (53) No. - 0 The SaaS indicators introduced in this article are based on the services portfolio strategy perspective and they are split in three levels: quantifying the realization degree of the services portfolio benefits, quantifying the degree of the services portfolio goals and quantifying of the services portfolio revenue taking into account the benefits and the goals of the portfolio. The paper has been organized in such a way that the first section contains an overview of the existing SaaS etrics and soe of the services portfolio benefits; the second section introduces the theoretical approach that is the basis of the realization of the key strategic perforance indicators for SaaS services portfolio; third and forth section contain the strategic perforance indicators developent and explanations with concrete results and the last section presents the conclusions and further developents in this area.. SaaS Metrics and Benefits Overview The SaaS odel allows the usage of the IT infrastructure in an optiized way, fro technical and business point of view, due to the efficient usage of IT equipent and other operational resources []. This gives the possibility to the SPs to aintain lower fees for the tenants that are using the services. Thus, SPs have to track the perforance of their services taking into consideration the financial aspects, the benefits that their services bring to the organization and the copletion degree of the services goals [7]. The services can be grouped in portfolios based on different criteria like arket targets, application types, and addressed industries. Also, the services can deliver different types of benefits which have to be aligned to the SP strategy. Benefits have to be agreed and planned before the service creation and the portfolio aggregates all benefits delivered by each service that is part of it. The benefits of each service contribute to the SPs organization and should give the arket copetitive advantages and the desired arket place [6]. Sae as in project anageent, the role of a services portfolio is the verification of the expected benefits to be delivered as the services as it has been planned... Overview of existing SaaS Metrics One of the SaaS etric generally used by the SPs in their effort to track the services perforances is the Churn Rate [8], [9]. This etric represents in percentage the users rate in cancelling a service over a specific period of tie [8]. If the value of this etric is high, it eans that there are probles with the service users satisfaction or with the service renewals eaning that the nuber of the new custoers is less that the ones which cancel the service. Another etric that is in a direct relation with the Churn Rate is the Recurring Revenue of the service [8]. It represents the service incoe oney fro the custoers that are using and paying the service in a specific tie interval like onths, quarters or years [9]. The Average Recurring Revenue is a derivation fro the previous etric and represents the total recurring revenue on average per custoer [9]. Related to the service developent and running, one of the etrics used is the Cost of Service per custoer [8]. It represents the average recurring costs per custoer that are needed to develop and aintain the service including the necessary infrastructure and SP personnel costs to run the service. Metrics related to custoer acquisition costs depend on what the SP needs to

3 Cazacu, V., et al.: Perforance Metrics for the It Services Portfolio 0 follow up. There are service etrics to deterine the costs related to arketing activities for custoer acquisition, to calculate sales expenses or to estiate the return of acquisition investent [8]... SaaS Benefits Overview One of the service benefits is the tie to arket of a service that is deployed on an existing infrastructure as well as the access of the service at a global level [7]. Another benefit is the predictability of the service usage growth based on real tie onitoring. In a direct relation, the revenue flow of the services can be estiated ore accurate because of the SaaS payent odel which is generally based on the service subscription ethods. SPs have aggregated operating environents which have as a benefit the lower level of the operations costs using also standardization and autoation of the services IT related activities. The SaaS odel offers the possibility to the copanies to enter on the new arket segents represents one of the SaaS strategic benefits that it is translated in the respective services portfolio benefits. 3. Indicators and Critical Success Factors The represents a quantitative etric related to the expected perforance of a process, accoplishent of the objectives and it is in a direct relation with the critical success factors [3]. Copleentary to a, there has to be a goal that represents the desired level of the perforance, defined before the easureent [5]. The critical success factor represents the essential eleent necessary to ensure the successful copetitive perforance for the organization [4]. By using this approach, the identification of the issues that ipact the strategy of the services portfolio is done. Before defining the s, it is essential to perfor an analysis of the services portfolio objectives and their critical success factors. Most of the services s are financially related or based on a specific perforance of the services. Because of this, the evaluation of the services portfolio has soe liitations since the success factors and the objectives of the portfolio are not evaluated. Based on this, only financial related forecasts can be done for the services portfolio which not all the tie ay reflect the SP strategic objectives and targets. For exaple, a SP has a services portfolio with three related services, two services are perforing fro econoic point of view and one not. But, the last service is very strategic for the copany, because it brings ore benefits that the other two on longer run. With only the financial perspective, the last service has to be cancelled but the relations inside the services portfolio give the possibility to evaluate all the variations of the services perforances that can effect the achieveent of the defined portfolio objectives. 4. Definition of Strategic Key Perforance Indicators for Services Portfolio In 00, Hynuk S. and Benoit R. fro Canada have published an article related to the easureent of the projects portfolio strategic perforance, where they have introduced several indicators that were correlated with the projects contribution to the benefits and goals of the portfolio []. Based on the critical success factors, in the article were defined several steps to reach the point to deterine the contribution of the projects to the realization of the portfolio objectives []. First step is to deterine the objectives of the portfolio based on the strategy and the needs of the copany. Second step is to analyze the key benefits that are essential to reach the portfolio goals. Third step is to establish

4 0 Bulletin of the Transilvania University of Braşov Series I Vol. 4 (53) No. - 0 the relation between the benefits and each project fro portfolio as well as with the portfolio objectives. The relationship is established using percentage values to describe the contribution of the projects to the portfolio key benefits. Sae procedure is used to find out the benefits contribution to the portfolio objectives []. In the sae way as in the project anageent, the objectives and the benefits of the services portfolio are derived fro the Service Provider s copany strategy. Based on the described steps and research, the authors of this paper have adapted and introduced new indicators to estiate the contribution of the services to the SPs portfolio. As in the projects case, the contribution of a service to the corresponding portfolio is defined as in the next forula []: C SO n B ( C C ). () SB BO The eanings of the paraeters are related to the described steps as follows: C SO - contribution of service S to the objective O, based on n portfolio benefits, C SB - relative contribution of service S to the benefit B, C BO - contribution of benefit B to the objective O. Restrictions have to be applied, since proportional values are used []: C SO S n B ( ), () ( ). (3) C BO The both forulas are applicable for all objectives O fro the services portfolio []: C SB S ( ). (4) The forula is applicable for all the portfolio benefits B based on the contribution of all services to the respective benefits. Based on the presented forulas and steps, there is the possibility to identify, according to portfolio objectives and benefits, which service is perforing inside the portfolio as estiated and which service needs special attention like actions plan that should stabilize it or even to cancel it in tie. Also, for the services portfolio it is iportant the easureent and estiation of each service contribution to the portfolio revenue fro the benefits and objectives perspective. Based on the Recurring Revenue introduced in the previous paragraphs, easured for the each onth, the authors of the paper define the contribution of the service S to the Monthly Recurring Revenue, as it is in the next forula: C SMMR p ( C O SO C OMMR ). (5) The significance of the new introduced paraeters is: C SMMR - contribution of service S to the Monthly Recurring Revenue, based on p portfolio objective, C OMMR - relative contribution of objective O to the Monthly Recurring Revenue. The restriction that has to be applied for each estiated Monthly Recurring Revenue is: C SMMR S ( ). (6) This new introduced diension is used in the definition of the s that show the status of the services portfolio based on decisive factors, agreed deviations and strategic viewpoint [3].

5 Cazacu, V., et al.: Perforance Metrics for the It Services Portfolio 03 The perforance level of the services and the contribution to the benefits and objectives of the portfolio are described in the next two indicators, derived fro project anageent research area. Thus, the level of the benefit B realization for a service portfolio expressed by the B, where the perforance etric of the service S that contributes to the benefit B is in direct relationship with the contribution of the service S to the benefit B, is expressed in the next forula: B ( S SB C SB ). (7) The value of the B is interpreted as a rate of the perforance with intervals between 0 and, which eans that the benefit B is under the required target; above it eans that the key perforance of B is over achieved and exactly eans that the key indicator reached the target. The rate expression of the achieveent capacity of the portfolio objective O based on the key perforance indicator of the benefit B and contribution of the portfolio benefit B to the objective O is shown in the next forula: O n B ( C ). (8) B The interpretations of the O values are the sae as for the B. Additionally to these key indicators adapted for the services area, the authors introduced other new indicators which show the status of the portfolio perforances, based on which evaluations and reactions to itigate any ipact of the risks can be done in a tiely anner. The key perforance for the portfolio represents the contribution of the each service S to the each objective O in a direct BO relation with the realization easureent of each objective O expressed by O : Porfolio p S O ( C ). (9) The interpretation of this can be done according to the nuber of objectives, but the target is reached when the value of this indicator is equal with the nuber of objectives p. What is below this value is in the risk area of the portfolio and depending on the nubers and root cause of the probles that can be deterine also fro the previous forulas, actions have to be taken. A key perforance indicator can be established also for the Monthly Recurring Revenue in a direct relation with the services portfolio objectives perforances and the relative contribution of the portfolio objectives to reach the Monthly Recurring Revenue. This indicator is presented in the next forula, where all the paraeters where previously introduced. MMR p O O SO ( C ). (0) O OMMR The value of MMR is interpreted as a rate, where the eans that the targeted perforance was accoplished, what is ore than it eans that fro the strategic point of view the MMR perforance is in the opportunity area, and what is in the 0 and interval is interpreted as a risk zone, where the achieveent of portfolio perforance is endanger by the objectives perforances. Since a is easured during tie intervals and the service portfolio evolves, review of each indicator has to be done regularly to check the validity of it. The last proposed indicator is the deviation of the service contribution to the portfolio Monthly Recurring Revenue, which

6 04 Bulletin of the Transilvania University of Braşov Series I Vol. 4 (53) No. - 0 is calculated as the argin between the estiated contribution of the service S to the MMR, C SMMR and the actual contribution of the service S to the actual MMR, A SMMR, in a certain point of tie: SMMR C A. () SMMR SMMR If the value of the is negative, then the actual contribution of the service to the MMR is bigger than estiated and if positive, the actual contribution is less than the forecast. But, the services being part of the portfolio, even if the actual contribution of a service is bigger than estiated will ipact another service fro portfolio which will have less contribution than the forecast. 5. Results and Discussions Deonstration of the theoretical approach is done based on a portfolio that has four services, three key benefits and two strategic portfolio objectives defined. Based on different techniques, the contributions of the services to the portfolio benefits (Table ) as well as the contribution of the benefits to the strategic portfolio objectives (Table ) are done in the beginning of the services deployent phase. Services to Benefits Table Contribution B B B 3 S S S S Benefits to Objectives Table Contribution O O B B B After the consensus is obtained, the next step is to calculate the contribution of each service to each portfolio objective like in the next exaple based on forula : C SO The table below contains all the results: Services to Objectives Table 3 Contribution O O S S S S The deterination of the objectives contribution to the Monthly Recurring Revenue in this step is done subjectively (Table 4) and has to be odified during the tie according to the portfolio strategy. Objectives to Each MRR Table 4 Contribution MRR M MRR M O O Based on the forula 5, the next step consists in the calculation of each portfolio service contribution to the MRR, for the two onths M and M : C , S MMR _ M C , S MMR _ M C , S3 MMR _ M C , S4 MMR _ M C , S MMR _ M C , S MMR _ M C , S3 MMR _ M C S4 MMR _ M

7 Cazacu, V., et al.: Perforance Metrics for the It Services Portfolio 05 After the values estiation, when the real values of the portfolio services contribution to the each MRR are received due to the services running in tie (Table 5), the deviation for each service contribution to the each MRR is perfored using the proposed SMMR and forula. The Figure and Figure show the deviations in percentage for each portfolio service contribution to the revenue together with the coparison between real and estiated contribution rate. Plan vs Actual Values 0,5 0,45 0,4 0,35 0,3 0,5 0, 0,5 0, 0,05 0 P P P3 P4 CS MMR MRR M Diff 30,00% 5,00% 0,00% 5,00% 0,00% 5,00% 0,00% -5,00% -0,00% -5,00% Fig.. Services Portfolio Evolution in the Month with high deviation values % Deviation: Plan vs Actual Services to Each Actual MRR Table 5 Contribution MRR M MRR M S S S S Plan vs Actual Values 0,5 0,45 0,4 0,35 0,3 0,5 0, 0,5 0, 0,05 0 P P P3 P4 30,00% 5,00% 0,00% 5,00% 0,00% 5,00% 0,00% -5,00% -0,00% -5,00% % Deviation: Plan vs Actual , S MMR _ M , S MMR _ M , S3 MMR _ M , S4 MMR _ M , S MMR_ M , S MMR _ M , S3 MMR _ M S4 MMR _ M The Table 6 represents the rate values for the perforance indicator of each portfolio service contributing to each benefit. Services to Benefits Table 6 B B B 3 S. 0.3 S S S CS MMR MRR M Diff Fig.. Services Portfolio Evolution in the Month with noral deviation values The values fro Table 6 are used to calculate the s for the portfolio benefits (forula 7) and objectives (forula 8) as it is shown next: B B B 3 O O , , , , , , After the first su is calculated, Portfolio has the value:

8 06 Bulletin of the Transilvania University of Braşov Series I Vol. 4 (53) No. - 0 Portfolio The target value of the in this case is, which eans that all the services fro the portfolio are at 00% realization of the target in relation with the benefits and objectives of the services portfolio. In this case, the target is not achieved, the value being in the portfolio risk area. The sae case applies also for the perforance indicators of the MMR (forula 0), calculated for onths, where the values are under, eaning that they are in the risk zone: MMR _ M MMR _ M , For all the defined s, thresholds can be deterined in such a way to define alars and contingency plans. 6. Conclusions The paper presents the developent of etrics for the Software as a Service odel. The etrics are focused on the easureent of the strategic contribution of the services to the portfolio benefits and objectives, having as starting point the project anageent techniques. Based on the new introduced s, the decisions to run further a service fro the strategic point of view can be done by analyzing the s values. In this way, the Service Provider anageent can apply action plans to itigate the risk or to use the opportunities detected in this way. The next steps for the authors are to incorporate these s in the services portfolio dashboard of the SaaS fraework that collects all the necessary inforation and displays the in a real tie anner, with triggers and alars. Acknowledgeents The authors wish to thank to all their colleagues fro Sieens PSE that gave their support in producing results for this paper. References. Garbea, A., Sisak, F., Perniu, L.: Service Oriented Architecture a Proise to the Future. In: Bulletin of the Transilvania University of Braşov, (00), Vol. 3 (5), p Hynuk, S., Benoit, R.: Measuring Portfolio Strategic Perforance Using Key Perforance Indicators. In: Project Manageent Journal, Wiley and PMI, 00, p Parenter, D.: Key Perforance Indicators (): Developing, Ipleenting, and Using Winning s, nd Edition. NJ. Wiley, Rockart, J.F.: Chief Executives Define Their Own Data Needs. In: Harvard Business Review 57 (979) No, p Sinclair, D., Zairi, M.: Effective Process Manageent through Perforance, Part III. In: Business Process Reengineering & Manageent Journal (995), MCB UP, p Willias, D, Parr, T.: Enterprise Prograe Manageent: Delivering Value. NY. Palgrave Macillan, Zhang, Q., Xinke, L.: Coplexity Metrics for Service-Oriented Systes. In: IEEE Explore (009), p *** SaaS Metrics - A Guide to Measuring and Iproving What Matters. Available at: Accessed: *** SaaS Metrics Guide to SaaS Financial Perforance. Available at: NuDGD. Accessed: