1 Twards Supprting the Adptin f Sftware Reference Architectures: An Empirically-Grunded Framewrk Silveri Martínez-Fernández Universitat Plitècnica de Catalunya Jrdi Girna, , Barcelna (Spain) ABSTRACT A Sftware Reference Architecture (SRA) allws rganizatins t reuse architectural knwledge and sftware cmpnents in a systematic way and, therefre, t reduce csts. SRAs mainly appear in rganizatins in which the multiplicity f sftware systems (i.e., sftware systems develped at multiple lcatins, by multiple vendrs and acrss multiple rganizatins) triggers a need fr life-cycle supprt fr all systems. Thus, SRAs are very attractive when rganizatins becme large and distributed in rder t develp new systems r new versins f systems. In return, rganizatins face the need t analyze the return-ninvestment (ROI) in adpting SRAs, and t review these SRAs in rder t ensure their quality and incremental imprvement. The gal f this research is t envisage an empirically-grunded framewrk that supprts rganizatins t decide n the adptin f SRAs and its subsequent design and suitability fr the rganizatin purpses. It helps rganizatins t harvest and arrange relevant evidence frm the wide spectrum f invlved stakehlders and available infrmatin and dcumentatin in SRA prjects. Such a framewrk is being shaped thrugh an actinresearch apprach between ur research grup and everis, an IT cnsulting firm. Categries and Subject Descriptrs D.2.9 [Sftware Engineering]: Management cst estimatin, life cycle, prductivity D.2.11 [Sftware Engineering]: Sftware Architectures dmain-specific architectures Keywrds Sftware reference architecture, empirical sftware engineering 1. INTRODUCTION 1.1 Backgrund Every sftware system has a [cncrete] sftware architecture : the sftware architecture f a system is the set f structures needed t reasn abut the system, which cmprise sftware elements, relatins amng them, and prperties f bth . Nwadays, the size and cmplexity f sftware systems, tgether with critical time-t-market needs, demand new sftware engineering appraches t design sftware architectures . One f these appraches is the use f a Sftware Reference IDESE'13 Baltimre, Maryland USA. Cpyright wned by authr. Architecture (SRA) that allws t systematically reuse knwledge and elements when develping a cncrete sftware architecture  . A SRA becmes, then, the baseline fr many sftware systems, as depicted in Figure 1. The purpse f SRAs is t serve as guidance fr the develpment, standardizatin, and evlutin f diverse sftware systems . This is pssible because SRAs are abstract enugh t allw its usage in differing cntexts . Figure 1. SRAs are the baseline f many sftware systems. 1.2 Mtivatin It has been claimed that SRAs reduce the cmplexity f hardware and sftware architecture by systematically reducing envirnmental diversity [...], enables greatly increased speed and reduce peratinal expenses as well as quality imprvements due t lwered cmplexity, greater investment and greater reuse . Thus, IT rganizatins that lack architecture and cnfiguratin standards [...] have higher csts and less agility that thse with enfrced standards . Accrding t their expected benefits, SRAs have becme widely studied and used by researchers and practitiners. There are many examples f SRAs f different types. On the ne hand, there are SRAs that target a technlgical dmain (als called platfrmriented SRAs ) such as The Open Grup Standard fr SOA Reference Architecture, which is a blueprint that prvides guidelines and ptins fr making architecture, design, and implementatin decisins when adpting a service-riented apprach t infrmatin technlgy . There are als SRAs frm academia t slve well-knwn technlgical prblems (e.g., sftware testing tls ). On the ther hand, there is anther type f SRAs that fcus n a specific business dmain. These SRAs can either target many rganizatins (whse sftware systems share the business dmain), r target a specific single rganizatin (which aims t standardize r facilitate the develpment and maintenance f its wn sftware systems). An example f a SRA that targets many rganizatins is AUTOSAR , which fcus n the autmtive dmain and is being used fr many car manufacturers, and suppliers in rder t standardize the sftware in mdern vehicles.
2 An example f SRAs fr a single rganizatin is the SRA fr NASA s Earth Science Data Systems, which facilitates and hmgenizes the develpment f this type f systems . In this thesis wrk, we pay special attentin t this last type f SRAs, thse that are designed fr and adpted in a single rganizatin. They appear in rganizatins where the multiplicity f sftware systems (i.e., systems develped at multiple lcatins, by multiple vendrs and acrss multiple rganizatins) triggers a need fr lifecycle supprt fr all systems . In this cntext, SRAs are very attractive when rganizatins becme large and distributed  in rder t develp new systems r new versins f systems. 1.3 Prblem Statement Despite SRAs are cnsidered a highly relevant strategic asset by industry and academia, there are still sme gaps that hamper its successful adptin. While cmprehensive methds have been defined fr the analysis f cst and benefits (e.g., CBAM ) and the review (e.g., ATAM ) f (cncrete) sftware architectures, SRAs have received relatively less attentin in literature. The reasn fr this can be prbably traced in an assumptin that these methds fr sftware architectures are directly applicable t SRAs. But in practice, as Angelv et al. pint ut , practitiners face additinal difficulties in wrking with SRAs. It is due t the specific features f SRAs with respect t sftware architectures , such as the need f a high up-frnt investment, their generic nature and high level f abstractin, the wide grup f stakehlders that they invlve, r the risks frm the instantiatin in the rganizatin s prtfli f sftware systems. This situatin triggers specific questins that have nt been addressed yet. Specifically, tw majr issues that need further research have been identified: 1. Lack f supprt fr deciding n the adptin f SRA. There is a shrtage f appraches t precisely evaluate the benefit f architecture prjects  in rder t take infrmed decisins abut adpting a SRA in an rganizatin. Thus, managers and executives lack f supprt t analyze whether it is wrth t invest n the adptin f a SRA in their rganizatin and t calculate its return-n-investment (ROI). 2. Lack f supprt fr SRA design and review. On the ne hand, in spite f research n the elements that shuld cmpse a SRA  and methds abut hw t design them , there is little evidence and supprt abut hw they are actually put frward in practice. On the ther hand, althugh there are several evaluatin methds fr SRA review , they have been hardly applied in the industrial practice. One ptential reasn culd be that the rganizatins find them expensive t apply and hard t be custmized and selected fr their specific needs and practices ; especially because there is n supprt fr identifying the real factrs that might apply in their specific rganizatins (e.g., sme methds suggest t cnsider the imprtant quality attributes fr the rganizatin, but which are such attributes fr the rganizatin?). Therefre, in such situatins, practitiners face difficulties that jepardize the success f the SRA prject. In this cntext, we greatly believe that the availability f evidence abut cst and benefits f real SRA acquisitin prgrams and SRA design and review industrial experiences wuld serve as a basis fr articulating a framewrk t supprt rganizatins and practitiners t face bth prblems. The paper is structured as fllws. Sectin 2 describes previus wrk t cpe with these tw issues. Sectin 3 shws the gal f this thesis. Sectin 4 presents the empirically-grunded framewrk that aims t accmplish such gal. Sectin 5 explains the actin-research initiative in which the framewrk is being shaped and validated, and Sectin 6 the threats t validity. Sectin 7 shws the current status f this research and future wrk. Finally, Sectin 8 expses the pints in which advice wuld be mre valuable. 2. RELATED WORK This sectin presents firstly related wrk fr calculating the ROI f architecture-centric appraches; and secndly related wrk abut reviewing SRAs. 2.1 SRA Ecnmics Althugh there is a lack f research in evaluating the ecnmic viability f SRA adptin, there is a strng base f research in related areas that culd be adapted with this purpse. One the ne hand, given the reusable nature f Sftware Prduct Lines (SPL), the ecnmic mdels that have been prpsed in this area fr identifying its csts and benefits can be adapted t SRA ecnmics. Hwever, ur analysis f these mdels led us t identify that they mainly fall shrt in: Validatin in industry. Very few [ecnmic mdels fr prduct line architecture] actually have been used as a basis fr further develpment r adpted in industry . Thus, there is a clear need fr many mre empirical studies t validate existing mdels . Easy adptin f mdels in industry by identifying realistic metrics t cllect and reprt. It is difficult fr the practitiners t evaluate the usability and usefulness f a prpsed slutin [ecnmic mdels fr prduct line architecture] fr applicatin in industry . N guidelines exist t fully peratinalize the mdels in practice . On the ther hand, there is als related wrk in ther areas. Fr instance, ecnmics-driven sftware architecture analysis methds (e.g., CBAM ). Hwever, existing prpsals d nt specifically aim at making an investment analysis f the adptin f an architecture-centric prgram. SRA adptin is actually a sub area inside their generic decisin-making cntext. Furthermre, ther wrks have addressed aspects as the quantificatin f the benefit frm architecture prjects that imprve quality attributes , and the uncertainties assciated with early lifecycle cst estimatin . At a lwer level, generic sftware metrics like design structure matrix (DSM), culd als be adequate fr calculating the cst and benefit factrs f SRA adptin and make mre cmplete mdels. As a result, the intended thesis uses and tailrs knwledge in these mentined areas fr its wn purpses. 2.2 Review f SRAs The sftware architecture f a sftware system is an early result f the develpment cycle that helps t identify and address imprtant quality aspects, such as system s perfrmance, security, reliability, and maintainability, befre the implementatin f the system is started . Therefre, it is a strategic apprach fr reducing cmplexity, csts and risks. As a result, t ensure architectural quality is an elusive gal. Thus, the practice f evaluating sftware architectures (i.e., the prcess f evaluating whether suitable aspects have been addressed) has matured, with well-
3 knwn methds such as ATAM , which helps stakehlders understand the cnsequences f architectural decisins with respect t the sftware system s quality attribute requirements. Existing methds fr sftware architecture evaluatin have been previusly applied fr SRAs, such as in . Hwever, the sftware engineering cmmunity rarely adpts the methds and techniques available t supprt disciplined architecture review prcesses . Fur pssible reasns fr this are: As Ali Babar et al. pint ut, we think that there remains a need fr systematically accumulating and widely disseminating evidence abut the factrs that may influence the selectin and use f different methds, techniques, and tls fr architecture evaluatin . Evaluatin methds d nt include evidence abut relevant aspects f SRAs, such as which quality attributes are relevant in this type f prjects. Evaluatin teams need t have the visin frm all stakehlders (e.g., prject managers, sftware architects, develpers, etc.). This is nt always supprted by evaluatin methds, leading t prblems while cnducting architectural reviews. Each f these stakehlders has a vested interest in different architectural aspects, which are imprtant t analyze and reasn abut the apprpriateness and the quality f the reference architecture . There is a lack f recent research that prpses means t evaluate reference architectures . These fur issues can be addressed by cllecting evidence abut the factrs that influence the selectin f evaluatin methds, studying the imprtant review criteria and the interest f essential stakehlders. Fr this reasn, mre and mre empirical studies t supprt SRAs thery are starting t be cnducted, as . 2.3 What is it needed? This state-f-the-art drve us t: The frmulatin f an ecnmic mdel fr SRAs built upn: Cst and benefit factrs frm prduct line architecture mdels that are easy-t-apply by the industry. The gal is t prvide guidelines t fully peratinalize the mdel in practice. The gap f SRA ecnmics inside the sftware architecture decisin-making cntext. Generic sftware metrics that can quantify new cst and benefit factrs. The study and cllectin f evidence abut relevant aspects t supprt the design f SRA and the use f architecture evaluatin methds. In , we identified six qualitative relevant aspects (verview, requirements, architectural decisins, business qualities, methdlgy and technlgies) which we take as a primary input fr their further refinement based n the evidence frm rganizatins. 3. RESEARCH OBJECTIVES This sectin expses the gal f this thesis: t supprt rganizatins t decide n the adptin f SRAs and its subsequent design and suitability fr the rganizatin purpses. As stated abve, such a gal will be dealt with by expliting real evidence. It is divided int tw Research Questins (RQ), which respectively deal with the tw prblems stated in Sectin Research Questin 1 The creatin and maintenance f cmplex sftware systems invlves making a series f business-critical architecture design decisins. Imagine that yu are the CIO f an rganizatin with a wide prtfli f sftware systems. Yu have read abut the expected benefits that a SRA may bring t yur rganizatin, e.g., standardizatin f cncrete sftware architecture f systems, greater reuse, shrter time-t-market, reduced csts, reduced risk, supprt fr system develpment at multiple lcatins, by multiple vendrs and acrss multiple rganizatins, and s n. Therefre, yu are cnsidering adpting an existing r new SRA t create and maintain yur rganizatin s sftware systems. Hwever, hw d yu knw if it is wrth fr yur rganizatin t invest n the adptin f a SRA? This questin culd be answered by making a business case with the help f an ecnmic mdel fr SRAs. In the SRA cntext, an ecnmic mdel is needed t help making business cases. An ecnmic mdel shuld take int accunt csts, benefits, risks, and schedule implicatins. An ecnmic mdel t perfrm cst-benefit analysis n the adptin f SRA is a key asset fr ptimizing architectural decisin-making. Reifer defines a business case as the materials prepared fr decisins makers t shw that the idea being cnsidered is a gd ne and that the numbers that surrund it make financial sense . That is, business cases enable t justify investments in technlgy. Spending in the adptin f a SRA withut a previus and trustwrthy analysis seems t be reckless and can lead t a disaster. This triggers the statement f the RQ 1: RQ 1. Hw can rganizatins be supprted t quantitatively analyze the up-frnt investment n the adptin f a SRA? The bjective f the RQ 1 is t prvide guidelines t supprt rganizatins t quantitatively analyze if it is wrth t adpt a SRA. Such an bjective cnsists f cnstructing an ecnmic mdel that supprts the decisin f adpting a SRA. This analysis ptimizes the decisin-making prcess when studying whether t make the strategic mve t SRA in an rganizatin. This research questin is divided int fur sub-research questins. The mtivatin f each sub-research questin is as fllws. First f all it is needed t understand the cntext in which SRAs are adpted (RQ 1.1). Secnd, t identify the data that can be easily cllected in the industry t quantify the csts and benefits f SRAs (RQ 1.2). Third, t define the actual csts and benefits implied by SRA adptin (RQ 1.3). Finally, t make the business case fr the adptin f SRAs (RQ 1.4). RQ 1.1: Hw are SRAs used in practice? RQ 1.2: Which available data d rganizatins have t quantitatively calculate the csts and benefits f adpting a SRA in an rganizatin? RQ 1.3: Which are the cst and benefit factrs f acquiring a SRA in an rganizatin? RQ 1.4: Hw is it pssible t calculate the ROI f the adptin f a SRA in an rganizatin? 3.2 Research Questin 2 Intrducing a SRA int an rganizatin nt nly invlves making a decisin cnsidering the afrementined prductivity issues, but als invlves the analysis f risks, nn-risks, benefits and tradeffs. Whereas prductivity is actually measured in terms f effrt/cst and ecnmic benefits, architectural quality is usually estimated in relatin t eliciting implicit and explicit requirements
4 f the different stakehlders affected by the develpment f the system. Nevertheless, bth views are necessary t achieve a cmprehensive analysis f the system. T help rganizatins t cpe with architectural quality during the design and review prcesses f SRAs, we prpse t accumulate real evidence abut the relevant aspects fr SRAs frm key stakehlders (e.g., which quality attributes they mainly enfrce). This triggers the statement f the RQ 2: RQ 2. Hw can rganizatins be supprted t deal with architectural quality frm its wn industrial evidence? The bjective f the RQ 2 is t supprt practitiners t deal with architectural quality by prviding them a way t gather its wn industrial evidence abut the aspects that matter fr their SRA. Such gathered aspects are aimed t be used t feed any existing architectural evaluatin methd. Specifically, this research questin is divided int five sub-research questins. These RQs address relevant aspects fr the design and review f SRAs: business qualities (RQ 2.1), elements (RQ 2.2), requirements (RQ 2.3), architectural decisins (RQ 2.4), and supprtive technlgies (RQ 2.5) respectively. RQ 2.1: Hw different stakehlders perceive the ptential benefits and drawbacks f SRAs? RQ 2.2: Which are the elements that cmpse a SRA and what is their ptential reuse acrss dmains? RQ 2.3: Which quality attributes des a SRA enfrce? RQ 2.4: Hw are architectural decisins taken and dcumented in SRA prjects? RQ 2.5: Which supprtive technlgies (i.e., methdlgies, tls) are currently being used in SRA prjects? The next sectin presents an empirically-grunded framewrk that aims t prvide means t answer these RQs. 4. RESEARCH APPROACH Empirical research is a way f gaining knwledge by means f direct and indirect bservatin r experience. One f the bjectives f Empirical Sftware Engineering is t gather and utilize evidence t advance sftware engineering methds, prcesses, techniques, and tls. This thesis prpsal fsters the cnductin f empirical studies as a way t incrementally build up SRA thery. The next sectin describes the expected cntributin f this thesis: a Framewrk fr SRA Analysis and Review. 4.1 An Empirically-Grunded Framewrk T accmplish the gal f this research, we plan t devise a framewrk by prviding prcedural guidelines fr setting up and carrying ut empirical studies. The framewrk is cmpsed f an assrtment f empirical studies that wuld help rganizatins t deal with RQ1 and RQ2. Each empirical study fits int ne f the three steps fr empirical research suggested by Whlin et al. : understand, evaluate and imprve. The main idea is that it is in mst cases impssible t start imprving directly and that empirical studies can be cmplementary and supprt each ther (e.g., results frm a preceding study can be used t crrbrate r develp further these results). The framewrk explicitly deals with the understanding and evaluatin steps. The imprving step is achieved by iteratively applying the evaluatin step and cnsidering the lessns learned. The studies shuld be cnducted sequentially. Figure 2 describes the studies that cmpse the framewrk. The rws indicate the step in which the study is being applied whereas the clumns shw the RQ that the study appraches. As shwn by Figure 2, the framewrk is cmpsed f fur studies, tw fr each RQ: RQ 1 is supprted by tw studies: A Survey t check existing value-driven data in rganizatins. It aims t prvide supprt r guidelines t check existing value-driven data in the rganizatin in rder t perfrm a quantitative evaluatin. An Ecnmic mdel t calculate the ROI f adpting a SRA. It aims t prvide an ecnmic mdel t calculate the ROI f adpting a SRA. RQ 2 is supprted by tw studies: A Survey t understand the impact f using a SRA. It aims t prvide supprt r guidelines t understand the impact f using a SRA in the rganizatin in rder t perfrm a qualitative evaluatin. An architectural evaluatin methd specific fr SRA. The abve survey helps t prvide supprt fr the selectin f an architectural evaluatin methd fr SRA and easy its cnductin with infrmatin frm key stakehlders. As explained in Sectin 2.2, currently, there exist evaluatin methds. Therefre, the framewrk just enables practitiners fr the smth applicatin f such existing methds. Figure 2. Empirically-grunded framewrk t supprt rganizatins n SRA adptin, design and review.
5 5. ACTION-RESEARCH PROJECT This research has its rigin in an nging actin-research initiative amng ur research grup and everis, a cnsulting cmpany. Actin research is learning by ding - a grup f peple identify a prblem, d smething t reslve it, see hw successful their effrts were, and if nt satisfied, try again . As a result, the afrementined framewrk is being devised by applying the actin-research cycle in everis: 1) diagnsis f a prblem, 2) examinatin f ptins t slve the prblem, 3) selectin f ptins and executin, 4) analysis f the results, and, 5) repeat fr imprvement. As a cnsulting cmpany, everis ffers slutins fr big businesses (e.g., banks, insurance cmpanies, public administratin, utilities, and industrial rganizatins) that need t manage a wide prtfli f sftware systems that share a specific-dmain. Given the cmplexity f the resulting infrmatin systems, which integrate bespke applicatins with cmmercial packages, these systems need high-quality sftware architecture. This is the service that rganizatins hire t everis. The slutin prvided by everis is based n the adptin f a SRA in the rganizatin, frm which cncrete sftware architectures are derived and used in a wide spectrum f sftware systems. In this cntext, everis cmmissined ur research grup tw main tasks (respectively aligned with ur RQs): t calculate the ROI that rganizatins get after adpting a SRA. and t gather evidence t supprt the design and review f SRAs fr their clients. Precisely, the architecture grup f everis experienced difficulties t cpe with the tw issues that we are cping with in the RQs. As a cnsequence, we are applying the studies envisaged in RQ 1 in rder t calculate the ROI derived frm SRAs that everis created (r plan t create) fr rganizatins. On the ther hand, t supprt them t achieve architectural quality, we plan t cnduct the studies f RQ 2. T d s, it is necessary t cntact SRA s stakehlders . In everis, three essential rles are distinguished: sftware architects that cperatively wrk t figure ut a SRA t accmplish the desired quality attributes and architecturally-significant requirements f the client rganizatin; architecture develpers that are respnsible fr cding, maintaining, integrating, testing and dcumenting the SRA s sftware cmpnents; and applicatin builders that take reusable cmpnents frm the SRA and instantiate them t build cncrete sftware architectures fr sftware systems. 5.1 Framewrk Shaping and Validatin The expected result f this thesis is the empirically-grunded framewrk afrementined (see Figure 2). The framewrk will be incrementally cnstructed based n the actin-research apprach in everis. everis' results will be suitably packaged with the aim f being applied in similar rganizatins. Furthermre, fr its shaping and validatin, the research is divided in tw stages: the frmative and the summative stages. During the frmative stage, we will cnduct empirical studies 1, 2 and 3 (detailed belw) in everis. As their cnductin advances, their feedback will cntribute t incrementally shape and package btained results. The summative stage will take place nce the framewrk has been adequately imprved, shaped and packaged. The primary rle f this stage will be t validate the final versin f the framewrk with practitiners. The next three sub-sectins respectively describe hw the studies f the framewrk have been designed and are being cnducted in the actin-research initiative with everis. 5.2 Empirical Study 1 Objectives f this study. The bjective f this survey is t identify the quantitative infrmatin that can be retrieved frm past prjects in rder t feed the ecnmic mdel (see Empirical Study 2 belw). The main perceived ecnmic benefits n the use f SRAs are the cst avidance in the develpment and maintenance f systems due t the reuse f sftware elements and the adptin f best practices f sftware develpment that increase the prductivity f develpers. The ecnmic mdel needs this data t define and calibrate the parameters t calculate the ROI f adpting an SRA in an rganizatin. Methd. Explratry surveys with persnalized questinnaires applied t relevant stakehlders t find ut the quantitative data that has been cllected in SRA prjects and applicatin prjects. Sampling. A sample f 5 everis SRA prjects and 5 applicatins built upn such SRAs have been selected, and their respective sftware architects and applicatin builders. Apprach fr data cllectin. We use nline questinnaires t ask sftware architects and applicatin builders abut existing infrmatin in past prjects fr calculating cst avidance frm SRA reuse in applicatins. The questinnaire is cmpsed f yes-n questins asking if specific metrics are available fr SRA prjects. Questinnaires enable the additin f cmments and metrics if desired by the interviewees. Data Analysis Methds and Techniques. The data analysis cnsists f cunting in hw many prjects a specific metric is available. Further details f this apprach have been reprted in . 5.3 Empirical Study 2 Objectives f this study. T remain cmpetitive, rganizatins are challenged t make infrmed and feasible value-driven design decisins. Hwever, there is a lack f supprt fr evaluating the ecnmic impact f these decisins with regard t SRAs. This damages the cmmunicatin amng architects and management, which can result in pr decisins. This empirical study analyze whether it is wrth investing in a SRA with the help f an ecnmic mdel. Methd. A case study in which REARM , which is an ecnmic mdel fr SRA adptin, is applied. Sampling. A sample f 2 everis SRA prjects and 2 applicatins built upn such SRAs have been selected. Apprach fr data cllectin. Results frm the Empirical Study 1 revealed that the data available in rder t calculate csts and benefits are effrt and sftware metrics . We cllect these metrics, which are presented in , frm tw types f tls. On the ne hand, a time tracking tl (e.g., JIRA , Redmine ) t cllect the invested effrt frm training, develpment and maintenance activities. Keeping track f activities is cmmn in practice fr prject management and auditing.
6 On the ther hand, tls that calculate sftware metrics t analyze the benefits that can be fund in the surce cde. Fr instance, SnarSurce  ffers tl supprt fr btaining general sftware metrics such as LOC, dependencies between mdules, technical debt, and percentages f tests and rules cmpliance. Data Analysis Methds and Techniques. Fr analyzing the utput f the ecnmic mdel we apply analysis techniques fr business case, such as breakeven analysis, cst-benefit analysis, payback analysis and sensitivity analysis . We have cnducted this empirical study in a public administratin. Results have been published in . 5.4 Empirical Study 3 Objectives f this survey. The purpse f this survey is t understand the impact f using SRAs fr designing the cncrete sftware architecture f the applicatins f a client rganizatin. This is a descriptive survey that measures what ccurred while using SRAs rather than why. With this survey we want t incrementally increase the evidence abut key aspects that really matter t the SRA being analyzed. These key aspects crrespnd t the five sub-rqs f RQ 2. Methd. Descriptive surveys with persnalized questinnaires applied t relevant stakehlders t gather their perceptins and needs. Sampling. The target ppulatins f this survey are SRA prjects and SRA-based applicatins executed by everis. A sample f 9 representative everis SRA prjects in client rganizatins was selected. In these prjects we plan t cntact three essential stakehlders: sftware architects, SRA develpers and applicatin builders. All these prjects were frm Eurpe (seven frm Spain). Apprach fr data cllectin. On the ne hand, semi-structured interviews are used fr sftware architects. The reasn f using interviews is that these rles have higher knwledge than the ther rles abut SRA, s we want t cllect as much infrmatin as pssible frm them. Prir t the interviews, questinnaires are delivered t cllect persnal infrmatin abut the interviewee and t infrm him/her abut the interview. On the ther hand, nline questinnaires are used fr SRA develpers and applicatin builders, since mst f their questins are abut supprtive technlgies and their respnses can be previusly listed, simplifying the data cllectin prcess. The cmplete versin f the prtcl and the questinnaires is available at survey-prtcl.pdf. Data Analysis Methds and Techniques. T perfrm data analysis, we apply qualitative analysis methds . Our research team held several discussin meetings during and after data cllectin. Fr aviding bias, the everis team did nt participated n these meetings (we cntacted them just fr serving as intermediaries fr appraching t the respndents whenever we needed clarificatins). We incrementally prcessed the manual transcriptins f all interviews and autmatically gt the data frm the nline questinnaires. We based ur analysis n grunded thery techniques  such as cnstant cmparisn and crss-case analysis . These techniques are well-fitting in situatins where the researcher des nt want t use pre-cnceived ideas, and instead is driven by the desire t capture all facets f the cllected data and t allw the prpsitins t emerge frm the data. The data analysis cnsisted f tw steps. First, analysis is driven by cding pieces f data as the cnstant cmparisn methd requires . We first read the interview transcripts and the data frm the questinnaires and attach a cding wrd t a prtin f the text a phrase r a paragraph. The cdes are selected t reflect the meaning f the respective prtin f the interview text t a specific RQ. Secnd, we perfrm crss-case analysis  t see the different views frm multiple stakehlders ver the answers with the same cde. We cluster all pieces f text that related t the same cde t analyze it in a cnsistent and systematic way. Finally, t interpret the results, we plan t hld a meeting with the everis team in rder t discuss and imprve ur understanding f the results. By the time f writing this paper, preliminary results abut the RQ 2.1 have been reprted in . 6. THREATS TO VALIDITY This sectin discusses pssible threats t validity f the design f the framewrk in the actin-research prject with everis. It is presented in terms f cnstruct, internal and external validity as well as reliability, as prpsed in . It als emphasizes the mitigatin actins used. 6.1 Cnstruct Validity T strengthen this aspect we have perfrmed a rigrus planning f the study and established a slid prtcl fr data cllectin and data analysis. First f all, t start devising the framewrk, it became necessary t previusly identify relevant aspects t assess SRAs. Hwever, a cmmnly accepted set f criteria t assess SRAs des nt exist . Thus, we identified imprtant aspects t assess SRAs ut f practice and ut f the literature in . The framewrk envisages these aspects as a primary input fr their further refinement during its frmative and summative stages. The clse invlvement f the everis team in the research planning and design is being vital t the suitable cnstructin and develpment f the data cllectin instruments (i.e., the ecnmic mdel s metrics, the interview guides and the questinnaires). In additin, these instruments have been pilted and enhanced t ensure their effectiveness. Given the invlvement f the everis team n the study, we were aware f the imprtance f including specific mitigatin actins fr evaluatin apprehensin by ensuring the aggregated presentatin f the respnses and their cnfidentiality. 6.2 Internal Validity One f the main relevant decisins that directly affected the sampling apprach is that we decided t first chse everis prjects and then participants that cvered the rles we were lking fr. In this way, we ensured that each participant wuld fcus his/her answers n the cntext f the defined prject. This wuld allw a better interpretatin and assessment f cntextual infrmatin. It wuld therwise have been very difficult t interpret certain SRA influential factrs related t the nature f the prjects. We are aware that sme pssible biases may be related t this strategy, fr instance that the everis team chses the mst successful prjects as sampling. T minimize this, we explained them the imprtance f having a representative
7 sampling f the prjects they perfrm in rder t btain reliable data. Regarding the individuals, there is always the pssibility that they frget smething r d nt explicitly state it when they are asked abut it. T reduce this risk: 1) in the case f the interviews, we discussed sme ptential tpics that might be mitted by the respndents, and paid particular attentin t them during the interviews in rder t ask fr clarificatins if necessary; 2) in the case f the nline questinnaires, we designed them in such a way that the respndent culd add additinal cmments and has t answer all the crrespnding questins while he/she culd cmplete the questinnaire at any time, s it gives them the pssibility f cnsulting registries and dcumentatin in case he/she needs t remember smething; 3) in all cases we perfrmed triangulatin by adding questins aimed t cnfirm the crrectness f the answers. We put frward several mitigatin strategies. First, recrding and transcribing all interviews cntributed t a better understanding and assessment f the data gathered. Secnd, t reduce the ptential researcher bias, several meetings were held amng the researchers in rder t discuss the results. Third, althugh the access t the tls, surce cde and dcumentatin is prvided by the everis team, the cllectin f metrics and data frm them is dne by the researchers. 6.3 External Validity As it was mentined in Sectin 1.2, SRAs are widely recgnized in the industry, and ther rganizatins present a very similar cntext t everis. As a cnsequence, we think that it culd be pssible t bserve similar experiences in prjects and cmpanies with similar cntexts. As Seddn et al. suggests: if the frces within an rganizatin that drve bserved behavir are likely t exist in ther rganizatins, it is likely that thse ther rganizatins, t, will exhibit similar behavir . As future wrk, as part f the summative stage, we plan t replicate the framewrk in similar cntexts t everis. We present the results frm the studies with a detailed explanatin f SRA prjects cntext as well as the methds and materials used. This is essential t allw the replicatin f the empirical studies f the framewrk fr ther SRA vendrs and acquisitin cmpanies with similar cntexts in rder t crrbrate the results and being able t generalize the results. Mrever, we acknwledge that several ther factrs may influence SRA prjects and their stakehlder s perceptins (as rganizatinal prcesses and plicies, resurces, cultural issues, etc.). We, therefre plan t design the framewrk in a way that ther factrs can be included at the cnvenience f the rganizatin that is applying it. 6.4 Reliability In rder t strengthen this aspect we addressed the validity f the study. Besides the strategies abve, we maintained a detailed prtcl, cnducted the survey tasks (data cllectin and analysis) by at least tw researchers, spent sufficient time with the study, and gave sufficient cncern t the analysis f all respnses. Such analysis was subsequently discussed with the everis team t imprve the understanding and cntextualizatin f the cnclusins. Finally, in case f the existence f related thery papers r empirical studies, we study hw the results frm ur actin-research apprach supprt r refute previus hypthesis and add new empirical-based prpsitins. 7. CURRENT STATUS AND FUTURE OF THIS RESEARCH S far we have cmpleted the fllwing activities f the frmative stage t shape the framewrk: Preliminary design f the framewrk  based n the state-f-the-art and the state-f-the-practice. Cnductin f the Empirical Study 1 . Cnductin f the Empirical Study 2 . Executin f the Empirical Study 3 and analysis f ne grup f questins (related t benefits and drawbacks ). We plan in the immediate future t carry ut the analysis f the remaining grups f questins frm the Empirical Study 3. Finally, we will package the final research results and validate them in the summative stage f the framewrk. This evaluatin cnsists f the replicatin f the Empirical Studies 1, 2 and 3. Organizatins analyzing whether t make the strategic mve t SRA adptin and rganizatins that face the design and review f SRAs will benefit frm this framewrk. 8. ISSUES OF DISSERTATION There are several issues f which I wuld like t get feedback at the dctral sympsium, althugh any feedback wuld be welcme: Are there any ther relevant aspects t SRAs that culd be addressed in future studies? Are there any mre ways t imprve the data cllectin and analysis? Hw t ensure external validity fr similar cntexts? Hw is it pssible t generalize/package the results withut bias? 9. ACKNOWLEDGMENTS This wrk has been supprted by Cátedra everis and the Spanish prject TIN C REFERENCES  Ali, M.S., Babar, M.A., Schmid, K.: A cmparative survey f ecnmic mdels fr sftware prduct lines. In: Sftware Engineering and Advanced Applicatins, SEAA 09. pp IEEE (2009)  Angelv, S., Grefen, P., Greefhrst, D.: A framewrk fr analysis and design f sftware reference architectures. Infrmatin and Sftware Technlgy 54(4), (2012)  Angelv, S., Trienekens, J.J., Grefen, P.: Twards a methd fr the evaluatin f reference architectures: Experiences frm a case. In: ECSA, pp Springer (2008)  Angelv, S., Trienekens, J.J., Kusters, R.: Sftware reference architectures - explring their usage and design in practice. In: Sftware Architecture. Springer (2013)  AUTOSAR: (2013) [Online; accessed 10-Jul-2013]  Babar, M.A., Bass, L., Grtn, I.: Factrs influencing industrial practices f sftware architecture evaluatin: an empirical investigatin. In: QSA, pp Springer (2007)
8  Babar, M.A., Grtn, I.: Sftware architecture review: The state f practice. Cmputer 42(7), (2009)  Bass, L., Clements, P., Kazman, R.: Sftware architecture in practice, 3 rd Editin. Addisn-Wesley Prfessinal (2012)  Carriere, J., Kazman, R., Ozkaya, I.: A cst-benefit framewrk fr making architectural decisins in a business cntext. In: Sftware Engineering, 2010 ACM/IEEE 32nd Internatinal Cnference n. vl. 2, pp IEEE (2010)  Clements, P., Kazman, R., Klein, M.: Evaluating sftware architectures. Addisn-Wesley Reading (2001)  Clutier, R., Muller, G., Verma, D., Nilchiani, R., Hle, E., Bne, M.: The cncept f reference architectures. Systems Engineering 13(1), (2010)  Crbin, J., & Strauss, A. (Eds.). (2008). Basics f qualitative research: Techniques and prcedures fr develping grunded thery. Sage.  Fergusn, R. et al: "Quantifying uncertainty in early lifecycle cst estimatin (QUELCE)." SEI Technical Reprt, paper 706 (2011).  Gallagher, B.P.: Using the architecture tradeff analysis methd t evaluate a reference architecture: A case study. Tech. rep., DTIC Dcument (2000)  Galster, M., Avgeriu, P.: Empirically-grunded reference architectures: a prpsal. In: Prceedings f QSA. pp ACM (2011)  Galster, M., Avgeriu, P., Tfan, D.: Cnstraints fr the design f variability-intensive service-riented reference architectures an industrial case study. Infrmatin and Sftware Technlgy (2012)  Graaf, B., Van Dijk, H., van Deursen, A.: Evaluating an embedded sftware reference architecture -industrial experience reprt-. In: CSMR Ninth Eurpean Cnference n. pp IEEE (2005)  Glass, R. L. (1994). The sftware-research crisis. Sftware, IEEE, 11(6),  JIRA. (2013) [Online; accessed 14-Jul-2013]  Khurum, M., Grschek, T., Peterssn, K.: Systematic review f slutins prpsed fr prduct line ecnmics. Strategic Decisin Supprt fr Sftware Intensive Prduct Management p. 64 (2008)  Martínez-Fernández, S., Ayala, C., Franch, X., Martins, H.: Benefits and drawbacks f reference architectures. In: 7th Eurpean Cnference n Sftware Architecture (ECSA) (2013)  Martínez-Fernández, S., Ayala, C., Franch, X., Martins, H.: A framewrk fr sftware reference architecture analysis and review. In: 10 th Wrkshp n Experimental Sftware Engineering (ESELAW) (2013)  Martínez-Fernández, S., Ayala, C., Franch, X., Martins, H.: REARM: A reuse-based ecnmic mdel fr sftware reference architectures. In: 13th Internatinal Cnference n Sftware Reuse (ICSR) (2013)  Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded surcebk. Sage.  Muller, G., van de Laar, P.: Researching reference architectures. In: Views n Evlvability f Embedded Systems, pp Springer (2011)  Nakagawa, E.Y., Antnin, P.O., Becker, M.: Reference architecture and prduct line architecture: A subtle but critical difference. In: Sftware Architecture, pp Springer (2011)  Nakagawa, E.Y., Oquend, F., Becker, M.: RAMdel: A reference mdel fr reference architectures. In: WICSA. pp IEEE (2012)  NASA: ESDS Reference Architecture fr the Decadal Survey Era. ESDS%20Reference%20Architecture%20v1.1.pdf (2012). [Online; accessed;10-jul-2013]  Oliveira, L.B.R., Nakagawa, E.Y.: A service-riented reference architecture fr sftware testing tls. In: Sftware Architecture, pp Springer (2011)  Redmine. (2013) [Online; accessed 14-Jul-2013]  Reifer, D.J.: Making the sftware business case: Imprvement by the numbers. Addisn-Wesley Prfessinal (2001)  Rbsn, C. (2002). Real wrld research: A resurce fr scial scientists and practitiner-researchers (Vl. 2). Oxfrd: Blackwell.  Schmid, K.: An initial mdel f prduct line ecnmics. In: Sftware Prduct-Family Engineering, pp Springer (2002)  Sctt, D.: Gartner hype cycle fr real-time infrastructure. (2012), [Online; accessed 1-Mar- 2013]  Seaman, C.B.: Qualitative methds in empirical studies f sftware engineering. IEEE Transactins n Sftware Engineering, vl. 25(4), pp , (1999).  Seddn, P., Scheepers, P.: Twards the imprved treatment f generalizatin f knwledge claims in IS research: drawing general cnclusins frm samples. EJIS, pp. 1-16, (2011).  Snar Surce. (2013) [Online; accessed 14-Jul-2013]  The Open Grup: SOA Reference Architecture. https://www2.pengrup.rg/gsys/jsp/publicatins/public atindetails.jsp?publicatinid=12490 (2013).  Whlin, C., Höst, M., Henningssn, K.: Empirical research methds in sftware engineering. In: Empirical Methds and Studies in Sftware Engineering, pp Springer (2003)  Yin, R. K. (2009). Case study research: Design and methds (Vl. 5). Sage.
The Capacity Develpment Results Framewrk A strategic and results-riented apprach t learning fr capacity develpment The Capacity Develpment Results Framewrk A strategic and results-riented apprach t learning
Teacher s Manual fr the wrld s mst ppular LMS Jaswinder Singh Hw t Use Mdle 2.7 2 Hw t use Mdle 2.7, 1 st Editin Teacher s Manual fr the wrld s mst ppular LMS Jaswinder Singh 3 This bk is dedicated t my
A Call fr Clarity: Open Questins n the Scpe f FDA Regulatin f mhealth A whitepaper prepared by the mhealth Regulatry Calitin December 22, 2010 Authrs Bradley Merrill Thmpsn Epstein, Becker & Green P.C.
SECURITY GUIDANCE FOR CRITICAL AREAS OF FOCUS IN CLOUD COMPUTING V3.0 INTRODUCTION The guidance prvided herein is the third versin f the Clud Security Alliance dcument, Security Guidance fr Critical Areas
Finding the Way: A Discussin f the Swedish Migrant Integratin System Finding the way: A discussin f the Swedish migrant integratin system OECD 2014 1 July 2014 Finding the way: A discussin f the Swedish
A Beginner s Guide t Successfully Securing Grant Funding Intrductin There is a wide range f supprt mechanisms ut there in the funding wrld, including grants, lans, equity investments, award schemes and
A Plan t Transfrm the Empire State s Medicaid Prgram Better Care, Better Health, Lwer Csts M U L T I - Y E A R A C T I O N P L A N TABLE OF CONTENTS Intrductin... page 3 Health System Redesign in New Yrk:
Frm Data Mining t Knwledge Discvery in Databases Usama Fayyad, Gregry Piatetsky-Shapir, and Padhraic Smyth Data mining and knwledge discvery in databases have been attracting a significant amunt f research,
Springer Texts in Statistics Gareth James Daniela Witten Trevr Hastie Rbert Tibshirani An Intrductin t Statistical Learning with Applicatins in R Springer Texts in Statistics 103 Series Editrs: G. Casella
N Unsafe Lift Wrkbk Cver and Sectin Break image prvided curtesy f Arj Canada Inc. Table Of Cntents Purpse f this wrkbk... 2 Hw t use this wrkbk...3 SECTION ONE A Brief Review f the Literature...5 SECTION
Click4it Wiki - Tlkit Mst Significant Change Step by Step Step 1: Starting and raising interest A. It may help t use ne f the fllwing metaphrs t explain the MSC: Newspaper: Newspapers are structured int
Eurpean Medicines Agency Lndn, 20 Nvember 2008 Dc. Ref. EMEA/149995/2008 COMMITTEE FOR MEDICINAL PRODUCTS FOR HUMAN USE (CHMP) GUIDELINE ON SAFETY AND EFFICACY FOLLOW-UP - RISK MANAGEMENT OF ADVANCED THERAPY
Number 13 March 1999 R ELIEF AND R EHABILITATION NETWORK 13 RRN newsletter Imprving aid plicy and practice in cmplex plitical emergencies In this issue... Articles... 1 Cdes f Cnduct: Wh Needs Them?...
Please cite this paper as: Mickleit, A. (2014), Scial Media Use by Gvernments: A Plicy Primer t Discuss Trends, Identify Plicy Opprtunities and Guide Decisin Makers, OECD Wrking Papers n Public Gvernance,
Develping a Cmmunicatin Plan- Aim Aim The aim f a cmmunicatin plan states the verall gal f the cmmunicatin effrt. Determining the Aim Ask yurself r yur team what the verall gal f the cmmunicatin plan is.
Smallhlder Oil Palm Prductin Systems in Indnesia: Lessns frm the NESP Ophir Prject Idsert Jelsma, Ken Giller and Thmas Fairhurst Plant Prductin Systems, Plant Sciences Grup, Wageningen University, Wageningen
Hw t Write Prgram Objectives/Outcmes Objectives Gals and Objectives are similar in that they describe the intended purpses and expected results f teaching activities and establish the fundatin fr assessment.
New Zealand s Climate Change Target Our cntributin t the new internatinal climate change agreement - summary f cnsultatin respnses Disclaimer All reasnable measures have been taken t ensure the quality
2010 LIFE AFTER STUDY Internatinal students settlement experiences in New Zealand DOL11566 NOV 10 Authrs Angie Wilkinsn, Statistics New Zealand Paul Merwd, Department f Labur Anne-Marie Masgret, Department
RISING TO THE CHALLENGE Re-Envisining Public Libraries RISING TO THE CHALLENGE Re-Envisining Public Libraries A reprt f the Aspen Institute Dialgue n Public Libraries by Amy K. Garmer Directr Aspen Institute
Nt in Cully: Anti-Displacement Strategies fr the Cully Neighbrhd Prepared fr Living Cully: A Cully Ecdistrict June 2013 Nt in Cully: Anti-Displacement Strategies fr the Cully Neighbrhd June 2013 Acknwledgements
GENERAL EHS GUIDELINES: INTRODUCTION Envirnmental, Health, and Safety General Guidelines Intrductin The Envirnmental, Health, and Safety (EHS) Guidelines are technical reference dcuments with general and
30 th Street Area Master Plan September, 2012 City f Bise, Idah Prepared by Bise City Planning & Develpment Services & Capital City Develpment Crpratin Cnsultants HDR Leland Cnsulting Grup Parametrix 30th