Effect. of the Document Quality. of the Project Start Architecture. on Project Success. An exploration. Master thesis

Transcription

1 Effect of the Document Quality of the Project Start Architecture on Project Success An exploration Master thesis University: Radboud University, Nijmegen, The Netherlands Curriculum: Architecture in the Digital World (2006) Author: Ir Ernst de Boer Student number: Supervisor: Prof. Dr H.A. (Erik) Proper Second reader: Dr S.J.B.A. (Stijn) Hoppenbrouwers Date: 17 March 2009 Version: 1.1 (final)

2 Copyright Ernst de Boer. Parts of this thesis may be reproduced for scientific or educational reasons without prior approval. For all other reasons, prior written approval is needed before reproduction. In all reproductions including scientific reproductions, references to the author and this thesis must be included. Informing the author when references or reproductions are made is much appreciated. Effect of PSA document quality on project success page 2 of 53

3 Abstract Enterprise architecture is a maturing field that helps organisations align their internal organisational structure, processes and information technology to their strategy. One of the instruments used in this alignment is the project start architecture document (PSA). Changing the organisation and its information technology is mainly realised by projects which may or may not be successful. This thesis explores the relationship between PSAs and project success. The exploration builds upon existing knowledge about project success and applies this knowledge to the PSA. The concept that links project success with the PSA emerges from literature to be tangibility. Using a small number of case studies, we validated this expected relationship between the PSA and tangibility. Next, we were able to determine in which cases a PSA is useful and when it is not, the important factors being the type of project and the project management style. Finally, we identified a number of relevant document quality characteristics that make a PSA a good one. Effect of PSA document quality on project success page 3 of 53

4

5 Samenvatting Enterprise architectuur is een zich ontwikkelend vakgebied dat organisaties helpt hun interne organisatiestructuur, processen en informatietechnologie op de bedrijfsstrategie af te stemmen. Een van de instrumenten daarbij is de projectstartarchitectuur (PSA). Het werkelijk veranderen van de organisatie en de informatietechnologie wordt voornamelijk door projecten gerealiseerd. Dit onderzoek verkent de relatie tussen PSA's en project succes. We gebruiken daarvoor bestaande kennis over projectsucces en passen deze kennis toe op het gebruik van de PSA. Het concept dat projectsucces verbindt met de PSA blijkt in de literatuur tastbaarheid te zijn. Door middel van een klein aantal case studies kon deze relatie tussen de PSA en tastbaarheid worden gevalideerd. Vervolgens bleken we in staat om aan te geven in welke gevallen een PSA wel en niet zinvol is. De belangrijkste factoren daarbij blijken het type project en de projectmanagementstijl te zijn. Tevens waren we in staat een aantal relevante kwaliteitsaspecten te herkennen die een PSA een goede PSA maken. Effect of PSA document quality on project success page 5 of 53

6

7 Table of Contents Abstract... 3 Samenvatting Introduction Theory Projects, project success and architecture Project success model Project success and PSA PSA contents and the contents quality Summary Research execution Research strategy Sources for research Results Results per project Results per reader Cross checks and open questions Summary Conclusions Tangibility scale Tangibility Quality characteristics Open questions Summary Impact on PSA usage and writing in practice Limitations Reflection Reflection on the research Suggestions for future research References Appendix 1 Project success and stakeholders Appendix 2 Generic PSA contents template Appendix 3 Project type measurement (controllability) Appendix 4 Tangibility measurement Appendix 5 PSA quality characteristics measurement Appendix 6 Interview script Appendix 7 PSA open questions Appendix 8 Raw data: tangibility measurements Appendix 9 Raw data: cross checks and open questions Appendix 10 Raw data: quality characteristics measurement Effect of PSA document quality on project success page 7 of 53

8

9 1 Introduction Many projects fail. This is also the case for IT projects, even though there is extensive literature on project management and its effect on project success. The usual ingredients of the recipes to improve project success are project management control instruments, management style and people. To further improve the success rate of IT projects, other expertise fields are contributing as well. One of those areas is Enterprise Architecture, which is about aligning an organisation s structure and behaviour with its strategy and goals. One of the instruments that emanate from that field is the so-called Project Start Architecture or Project Solution Architecture (PSA). Several organisations now require such a document before starting the realisation phase of an IT project. PSAs supposedly make projects more successful. However, this has never been verified or falsified. This research explores that claim. Another question is which aspects of a PSA influence the chances on project success. One of the consequences of not having this knowledge is that neither effectiveness nor efficiency of the PSA is guaranteed. This research explores which aspects are relevant and which are not for effective PSAs, hence creating the opportunity to reduce the efforts needed for creating PSAs. Some of the conclusions can also help project managers and architects to use the PSA better, ultimately leading to more successful IT projects. Finally, the results can be used as a starting point for follow-up research. The steps leading to the knowledge sought are: first, from literature, the conceptual relation between the PSA and project success is deduced. Going into more detail, a relationship between PSA contents and its contents quality, and the factor(s) influencing project success is derived. A third step consists of exploring the relationships between the different aspects of PSA contents quality and the relevant factors for project success, by means of interviews and scoring tables. For this purpose, a number of potential PSA users are interviewed about actual projects at ASR Netherlands 1. Based on these interviews, the relationships are explored between different content aspects and the relevant factors influencing project success. The thesis ends with concrete conclusions and suggestions for future research leading to an even more effective and efficient use of the PSA instrument. 1 Research started at one of the subsidiaries of Fortis: Fortis Insurance Netherlands. During research execution, Fortis sold parts of the company. ASR Netherlands is the continuation of Fortis Insurance Netherlands. This did not influence the research. Effect of PSA document quality on project success page 9 of 53

10

11 2 Theory 2.1 Projects, project success and architecture Most organisations use information technology (IT) to support their business processes, which are necessary to realise products or services. In an ever changing world, these business processes, or the products or services they realise, change continuously. This in turn makes it necessary for IT to change as well, thus creating the changed support that is needed. Some of these changes can not be carried out by the line organisation and need to be managed by a project. In this thesis, a project is defined as a management environment that is created for the purpose of delivering one or more business products according to a specified business case (CCTA, 1999, p. 7). When such a project includes changes in IT, it is called an IT project. In this research, only IT projects are considered. Many of these IT projects are not successful (Ernst & Young, 2007), which poses a problem for the organisation: the goal is not realised and the project is thus a waste of resources. Several definitions and understandings of project success exist (Shenhar, Dvir, Levy, & Maltz, 2001). Traditionally, project success used to mean that the project had realised its results on time, within budget and to specification. Currently, another way of looking at project success prevails which is oriented more towards creating business value: does the project satisfy relevant stakeholders? (van Aken, 2002; Pinto & Slevin, 1988b). This research uses the latter view in which the relevant stakeholders are limited to the endusers and business executive. The definition of a project is in line with this view. The reasons why this strong reduction in relevant stakeholders is possible in this research, are given in Appendix 1. Today, the first view is sometimes called project management success (de Wit, 1988) as opposed to project success, in this thesis defined as the level to which the project results satisfy the end-users and business executive. A lot of research has been done on increasing project success rates by project management techniques, a clear example provided by a recent special issue of the International Journal of Project Management (Khazanchi & Reich, 2008). Applying the knowledge from that research area has done a great deal of good, but project success can not be realised by project management techniques alone. For instance: the level of confidence that customers and users have in the project manager is important for project success, but this can not be managed easily (Xia & Lee, 2005). Other scientific areas therefore try to contribute to more successful projects as well. Enterprise architecture is one of these areas. The theory and practice of Enterprise Architecture (van den Berg, Luijpers, van Steenbergen, & Wagter, 2005), (Lankhorst, 2005) introduces the so called Project Start Architecture or PSA. It is a document that is created during project initiation for projects that contain IT changes, and is supposed to achieve higher project success rates. It is used in two ways: as a summary of all relevant principles, guidelines, reference architecture etc. that are relevant for the project and which the project then needs to use when crafting a solution; or, taking into account the principles, guidelines etc. and the project objective, as a description of the solution itself. This research explores the link between the PSA and project success. In this thesis, only the second type of PSA is therefore relevant. Many definitions of Enterprise Architecture exist, for an overview see Op 't Land, Proper, Waage, Cloo, & Steghuis (2008). Because the research is about a specific use (specification of results) of a specific artefact (PSA) in a specific context (project), it is not important to know in which theoretical frameworks this usage occurs and in which frameworks it does not. Effect of PSA document quality on project success page 11 of 53

12 Figure 1. Place of PSA in Prince2 To illustrate in which phase the PSA is positioned, the relationship between the PSA and the project phases is shown. Since one of the most commonly used project management methods for IT projects is Prince2 (CCTA, 1999), this method is used for the illustration. The use of a PSA is not confined to any specific project management technique though. In Prince2 terminology, the PSA is a basis for writing the project plan (project initiation document or PID) and creating a product breakdown structure: the PSA explains what will change; the PID uses this to plan how. As mentioned above, van den Berg & Steenbergen (2004) claim that PSAs always contribute to project success. The authors explain this by reasoning that the PSA gives more clarity about the deliverables, thereby giving the project manager better possibilities to manage the project. This has no scientific background though. They propose a standard PSA template for all projects. An example of such a template is included as Appendix 2. Other authors (Felix & Harrison, 1984) also found the need for solid (start) architectures in complex, distributed systems development projects some time ago, yet they don't go into detail on the demands on such an architecture. In the day-to-day practice of the organisation studied, PSAs are not regarded to always contribute to project success neither project managers nor architects see the direct relation between the use of PSAs and project success that is claimed by van den Berg & Steenbergen. The question arises: when does a PSA contribute to success, and when does it not; the next question being: what factors contribute to success and which factors do not? To summarise, no scientific research has been done on the effect that the PSA has on project success, yet a one-size-fits-all template is provided to be used in all cases. To improve project success rates, knowledge is needed on both the PSA creation process and the PSA contents. This knowledge has not been made explicit yet. This research creates some of that explicit knowledge and shares the knowledge with the architectural and project management communities. Effect of PSA document quality on project success page 12 of 53

13 In order to keep this research feasible, the scope is limited. One of the restrictions is that the PSA creation process is left out and focus is on the resulting document only 2. A second limitation is introduced by focusing only on the use of the PSA in the project initiation phase, although the PSA can be used in other phases as well. 2.2 Project success model Before going deeper into the question of what a PSA should contain to achieve optimal positive effect on project success, a model must be in place to align project success (in the value sense) with relevant aspects of hard core project management techniques. Such a model is created by van Aken (2002). His work is summarised in this section. He uses the concept of project success as defined in section 2.1 and adds an overview of all factors influencing that success, based on a thorough literature review. Van Aken has been able to conclude that three relevant dimensions exist that can be inhibited by projects and that influence project success. These dimensions of projects and their respective influences on project success are shown in the conceptual model (Figure 2). The dimensions are: - controllability (van Aken, 2002, pp ); a combination of the complexity and the tangibility of the project; - working style (van Aken, 2002, pp ); the management style of the project manager; - use of instruments (van Aken, 2002, pp ). Figure 2. Conceptual model of project success (van Aken, 2002, p. 129) The associated conclusions on project success are (2002, p. 149): 1) to achieve project success, structuring is desired in projects that are barely controllable, but not in controllable projects; 2) a goal oriented style of working affects project success favourably to a high degree; 3) when work is goal-oriented, the project manager can work successfully with only a few instruments; 4) the more instruments used, the less chance of project success. The numbers are changed into positive (+) or negative (-) effects in the following figure. Connections not mentioned do not have a significant effect. 2 Another reason is that some good practices on the PSA creation process are already shared among architects, although no formal publications are known. Effect of PSA document quality on project success page 13 of 53

14 This thesis builds upon van Aken s work. The limits from his research therefore also apply to this research: the results are valid only for projects in organisations over 50 people, and that are organised with a separate project manager, business executive, and project team (no small projects). 2.3 Project success and PSA In order to determine the effect of a PSA on project success, all possible influences in the model must be investigated: what is the possible effect of a PSA on the respective influence? This is shown in the following figure which extends the previous model: PSA? Structured working style Controllability: - tangibility + / / - + if contrasted, - if the same few Project success instruments needed Goal oriented working style - + Instrument use Figure 3. Conceptual model of PSA contents effects on project success The following subsections examine each of these influences. The two types of working style are discussed together Effect of PSA on instrument use Van Aken gives a good overview on possible types of instruments (2002, pp ). The PSA can be considered to be an instrument and therefore affects instrument use positively. Since van Aken s end conclusion is that instrument use is not a major contributing factor but it will reduce project success if too many instruments are used too rigidly, the conclusion can be drawn that, through the factor instrument use, the PSA is not a major contributing factor to project success: however, too much emphasis on the PSA will reduce project success chances. Further exploration of the effect of PSA on instrument use is not going to help improve project success and therefore is kept out of scope in the remainder of this research Effect of PSA on working styles Working style (both goal oriented and structured) is important for project success, especially in combination with the controllability of a project (van Aken, 2002, p. 149). Working style is solely based on the project manager s style of working. Although the working style can result in the use of a PSA, the reverse is not true: PSAs have no direct effect on the working style. The project manager may choose to change his working style though because a PSA is available. This is not considered to be an effect of the PSA itself on project success and the relationship will not be explored further. The effects are shown as dotted lines in Figure Effect of PSA on controllability Uncontrollable projects can be managed to realise success by applying a structured working style. For controllable projects, it is wise to apply a less structured style. Changing uncontrollable projects into controllable projects therefore influences project success it will improve project success as long as it is complemented by the right Effect of PSA document quality on project success page 14 of 53

15 project management style. Since a PSA can influence controllability directly, a PSA can affect project success. In controllable projects, the project members can commit themselves to results, while in the less controllable projects only a commitment to do ones best can be made. Assuming that project managers and executives prefer results above efforts, a controllable project is preferred for project success Summary of the PSA effects The influence of the PSA on controllability of the project is the only relevant factor for project success. At the same time, the PSA must be used as little as possible as a project management instrument because that use will only reduce chances on project success. Since controllability is that important, the following subsection describes the term more thoroughly Controllability Projects can be distinguished by type. Van Aken uses the Dutch word grijpbaarheid as the discerning variable. He translated this word as governability. However, in today s world the word governability is commonly used in a more political sense. This research chooses to change the translation of the Dutch word grijpbaarheid to controllability. The original definition of controllability can then be translated into the degree to which the project manager can influence both contents and course of a project. The controllability of a project is a weighed sum of (van Aken, 2002, p. 76); see Appendix 3 for more detail: - 1 x number of disciplines involved; - 2 x tangibility: Concrete, physical end products are highly tangible and can be specified in much detail, yet low tangible end products are hard to specify, and - 3 x numbers of parties involved; In this thesis, the solution is assumed not to change from the bare fact that a PSA is created the PSA just describes the solution and makes the design decisions more explicit. As a consequence, the number of disciplines and the number of parties involved are fixed. This results in only one variable that can be changed by the PSA: tangibility. Van Aken chose to define tangibility indirectly by providing a scoring tool to measure tangibility which is part of Appendix 4. A dictionary defines tangibility as capable of being understood and evaluated, and therefore regarded as real or capable of being given a physical existence (Webster's, 1990). 2.4 PSA contents and the contents quality The question to be answered is now: what characteristics of the PSA influence tangibility? Literature (e.g. van den Berg et al., 2005) tells that completeness and accuracy of a PSA are important. For these, templates are available that tell what must be described, e.g. Zammuto (n.d.). The how question is never answered, e.g. if the PSA is easily understood, apart from a thesis by Veltman van Reekum in She created an instrument for measuring the quality of Enterprise Architecture products. However, she built this instrument on software, information and data quality instruments which seems to be the wrong starting point: when considering the PSA, it is the quality of the description itself that is relevant, not the quality of what is described 3. Another difference is the purpose of the measurement tool: she developed a method to determine quality as such of all kinds of enterprise architecture products, whereas the goal of this research is 3 Nevertheless, the solution that is described by the PSA must be correct a good description of a bad solution will not make a project successful. Effect of PSA document quality on project success page 15 of 53

16 to determine the effect of the different quality characteristics of one of the architecture products on one of the project s characteristics. This thesis assumes that the how question is very important though: the tangibility of the project as felt by the project members is based upon reading the PSA, and reading involves more than contents. This research explores which quality aspects are indeed important (both what and how related) and therefore should be given attention when creating a PSA, and which aspects can safely be ignored when writing a PSA. The different quality aspects of (technical 4 ) documents in general, and therefore of a PSA, can be captured by nine characteristics (Hargis et al., 2004). A scoring list is available in their book and reproduced in Appendix 5. The nine quality characteristics are, grouped by three main characteristics: - Easy to use task orientation accuracy completeness - Easy to understand clarity concreteness style - Easy to find organization retrievability visual effectiveness. All nine characteristics are scored by four to nine questions each. There is no reason to expect or assume that one or more of the characteristics are irrelevant in influencing tangibility. On the other hand, If a particular quality characteristic shows to be very relevant, it is possible to explore that particular characteristic more in depth. For example, if completeness shows to be very important, the elements that are most important for completeness can be checked using the list in Appendix Summary This ends the theoretical background to this research. In this chapter, it was shown that PSAs can have an influence on project success. Focusing on the project initiation phase, the factors and prerequisites that play a role were discussed. Two of those factors are further explored in the practical research part: tangibility and the quality characteristics influencing it. These two will give some insight into the questions: when can a PSA have a positive effect on project success, and what characteristics make this happen. 4 The term technical applies to process, product and organizational models too, which are also part of PSAs. Since these models are still hard, the whole PSA is considered to be of a technical nature. Effect of PSA document quality on project success page 16 of 53

17 3 Research execution The execution of the practical part of the research gives the data to answer the question: What are the relationships between the different quality characteristics and tangibility? How this question is answered is the subject of this chapter. 3.1 Research strategy For this explorative study, a small number of the same experiments are chosen. The project initiation phase is imitated in this experiment in a controlled way: a person, a typical user of PSAs, is given the standard set of project documents that are normally available at project initiation. The person is given some time to read this information, in order to simulate a real project. To determine this period, the person is asked how much time he 5 usually takes to read the initial project documentation. The time the user usually takes to read a PSA is also asked. Having read the project documentation for this predetermined period, the person is interviewed about the tangibility of the project, using a questionnaire. Next, the person is given the actual PSA of the project. The reader is asked to use the aforementioned amount of time to read this PSA. After reading, the tangibility is determined again and the reader is interviewed about the perceived document quality, again using a questionnaire. Finally, some open questions are asked for cross checking the validity of the answers and possible new clues. The extended interview script is included as Appendix 6. To be able to analyse both variations in readers and in PSAs, the same persons are asked to read different PSAs, and different persons are asked to read the same PSAs. To keep the research feasible within a limited time frame, three people are interviewed and three PSAs are analysed. This results in 27 contact moments: three projects, three readers and three contact moments per reader / project combination. An important consideration is to make sure the reader does not have knowledge about the project studied or the functional domain the project is about. If this happens to be the case anyway, another project is selected. E.g., if a project is about life insurance and one of the readers would be or would have been practicing in this field, another project in another field is selected. The next section gives the project and reader selection criteria in more detail. Alternative approaches have been considered but cast away as not feasible: better separation of the different effects (project / reader / PSA) could be realised by creating several PSAs for the same project and comparing these PSAs on their effect on tangibility by having the same person read the different PSAs; to correct for the organisational bias, more readers in more organisations could be interviewed. In other types of research or follow-ups, these approaches can be valuable. 3.2 Sources for research How to determine tangibility with and without the PSA? There is a scale available in van Aken s book which is used with minor adaptations. The scale is reproduced in Appendix 4. The score is a number between 1 (very low) to 5 (very high). How to determine the quality characteristics levels? This research is about only relative effects: which quality characteristics are the most important? An absolute and objective scale is therefore not a must. The subjective opinions of the readers suffice as long as they score consistently through all measurements. Interviews and the questionnaire from Appendix 5 (Hargis et al., 2004) 5 I expect the reader to understand that the pronoun he can refer to a woman. Effect of PSA document quality on project success page 17 of 53

18 are used for this purpose. The scale goes from 1 ( among the worst ) to 5 ( can be used as a model ). The assumption made is that the interviewees will apply the same (personal, subjective) scale during research. A cross check is introduced to find a shifting norm, if present. This is done during the interview for the third project by asking again some of the same questions for the first project, as remembered by the interviewee. If the reader now happens to assign different values to the first project than he did earlier, the interviews for the first and project must be repeated. Which projects to select? For comparison reasons, it is important to use projects that only have tangibility as varying factor, so the projects must be comparable in the number of disciplines involved and the number of parties involved (simple versus complex). Measuring these values objectively is realised by listing possible parties and disciplines, asking all readers and the researcher to state which parties and disciplines were/are involved, and counting them. If the numbers differ considerably, another project is chosen. The list used to determine number of parties and number of disciplines is part of Appendix 3. The projects are all started recently (2007 and later). Projects don t need to be finished yet since the projects successes are not relevant for this study. This approach does allow for follow-up research which can look into the actual project success and project working style. All projects are selected from the same organisation, where access to all PSAs is assured. Which people to interview? Since this research is not about the effect of using a PSA as such, interviews are held with people that are familiar with PSAs in general and usually are involved in project execution. For practical reasons, the readers come from the same organisation as the projects. Care is taken they are not familiar with the projects that are the objects of the PSAs. Otherwise, tangibility changes before and after reading the PSAs can not be measured truthfully. Using people from the same organisation also guards against the bias of differences in generic organisational knowledge, architecture process knowledge and PSA template experience. Effect of PSA document quality on project success page 18 of 53

19 4 Results This chapter gives a short characterisation of the three projects and the main measurement results. Conclusions follow in Chapter Results per project As one of the prerequisites for project selection and comparison, all projects must have a comparable number of disciplines and number of parties involved. This is indeed the case (6 or 7, and 9-11 respectively). The exact scores are irrelevant and therefore omitted in the following subsections. They are included in Appendix 8 - Appendix Project 1: SAP The first project is Real Estate on SAP. The project aims to replace the current (legacy) software set for one of the company s subsidiaries with SAP. The subsidiary manages and develops assets in real estate (housing, shops, offices, land). Integration with other applications in the company s application environment (data warehouse, archiving, document management, geographical information system, to name some) is an important aspect. At the start of the PSA writing, many documents on SAP implementation details were already available. The readers in this research were allowed to browse through all documents and take with them what they wanted to read. All consciously and realistically chose to read only the project initiation document (PID). After reading this PID, the scores for tangibility were 3.3, 2.7 and 3.3 respectively. After reading the PSA, the scores changed to 4.3, 3.3 and 3.3. All readers found this PSA a very good addition to the PID, especially since the PID only focused on project management and hardly on how things were to be solved on a technical level. The PSA gave clear answers and good direction. Although one of the readers did not like one of the fundamental technical choices, it was clear to him that this choice was final and all consequences were taken into account. The PSA scored also well on the clarification of the quality norms and the amount of knowledge needed for project members. Out of the three PSAs, this one scored highest on the quality aspects easy to use and easy to understand Project 2: web site The second project is about an addition to a web site. The insurance company studied traditionally works with brokers (in Dutch: assurantietussenpersonen), and does not sell to or communicate with consumers directly. To lower the administrational overhead for these brokers, other projects have already realised small web elements that these brokers can use on their web sites to allow their current and new customers to calculate premiums and to buy so-called simple products, like car insurance. The goal of this project is to realise a similar solution on the public web site of the insurance company itself. These new customers and their applications for new insurance policies must be routed to one of the brokers. The project has a very challenging deadline. After reading the project documentation, the readers scored mixed on tangibility: twice 2.7 and once 4.0. After reading the PSA, the values changed to 4.3, 3.3 and 2.0. The last value was explained by the reader with the words I was confused after reading the PSA. Effect of PSA document quality on project success page 19 of 53

20 All readers found the PSA useful in giving clear directions to the project members and were convinced the PSA had helped the project, even though it was clear to them that the PSA had only been given formal approval when the project was almost finished. They were sure that the unfinished and informal PSA had been used during the project as if it was formally approved. This was even true for the reader who had said he was confused. The PSA scored high on the clarification of the quality norms. The PSA scored low on accuracy and completeness, which led to a low score on understanding. Its quality scored average on both content and format. The PSA was directed at project members who were already familiar with the subject; the readers were not Project 3: accounts payable This project is aimed at enhancing the process regarding accounts payable. At the time of the project start, the current process was a manual process in which a paper copy of the invoice and an accompanying slip were distributed by the financial department to the department that can state whether the invoice should be paid or not. The signed slip is then sent back to the financial department, processed manually and the invoice is paid. The new process replaces this paper flow by an electronic flow, and automates the processing step. No changes take place in the process of reading the invoices and entering them into the accounts payable system: these steps continue to be manual. The workflow process of approving invoices is part of the accounts payable system, in which a link to the scanned image of the invoice is made available. The project had seen three unsuccessful attempts before, due to discussions on technical issues during project execution. A PSA had not been created for those attempts. A PSA has been created for this (4 th) attempt 6. After reading the project documentation but before reading the PSA, the readers scored this project the highest on tangibility (average 3.9). After reading the PSA, one of the readers surprisingly lowered his score somewhat, while the other scores remained unchanged and stayed to be the highest (average 3.8). Asking whether they thought the PSA had improved the chances on project success, readers one and three thought the PSA would indeed have benefited the project because the technical choices were clearly made (although none of the readers liked the choices), while reader two found the PSA useless. This conclusion was in line with his scoring: reading the PSA had not changed any of his tangibility scores. Interestingly, the other readers had not, or not significantly, changed their scores either but still deemed the PSA useful for the project. The PSA quality scored lower than the other PSAs on understanding and retrieving. Although the relative scores are consistent for all three readers, the variance within the answers is very high. 4.2 Results per reader Reader one raised his tangibility scores for the first and second PSAs to 4.3. The third PSA received a final lower score of 3.0. In the tangibility questionnaire, the reader especially liked the clear results and the good explanation of the quality norms in the first PSA. The second PSA was valued about the same but with the critical verbal note in the open questions that the PSA was not very 6 As a matter of fact, the project finished successfully in this fourth attempt. Effect of PSA document quality on project success page 20 of 53

21 good regarding the quality of the contents. The values measured for the quality characteristics were consistent with this remark, although the correlation is not strong. The reader was not impressed by the third PSA: almost all tangibility measurement scores were lowered, only the knowledge needed to execute the project was a bit less (positive). This was clearly visible as well in the quality characteristics measurement, which showed particularly low scores on content. The cross-check questions corroborate the results. The second reader also found PSAs #1 and #2 of the same quality and a substantial improvement to the original set of project documentation. Most scores were raised with the emphasis on the quality requirements 7. Interestingly, the third PSA does not improve the score on tangibility, yet the project s tangibility receives the highest score. As mentioned above, the reader thought this PSA was useless. When checked with the quality attributes scores, it is observed that the reader found this PSA not directed towards him (task orientation scores). The third reader did not score any project s tangibility higher after reading the respective PSAs. As a matter of fact, most of his scores did not change after reading the PSAs. Only the quality requirements and the knowledge needed improved for PSAs #1 and #2, while the direct tangibility question was awarded higher for the third PSA. The latter score was explained by the reader: it came from one single word in the PSA which had made him understand the only thing unclear to him before reading the PSA. For the second project, the reader even lowered the tangibility score. He explained this by telling: I m confused. I miss the context and the relations, the alignment between the different parts. The same was true when I had read the project documentation, but I had expected to find it in the PSA. On this basis, it can be reasoned that the score should be corrected to 3 (unchanged). Expectations play a role, apparently. 4.3 Cross checks and open questions In search for the quality characteristics that are most important for a PSA that enhances project success chances, the interview ended with open questions on this subject. The same questions were also used to verify the consistency between the other measurements and conclusions. The questions are listed in Appendix 7. In this section, first the answers to the question What contributed to the clarification? (The PSA as a document, not the contents as such) are given. They give some insight into the softer aspects of PSAs and the readers appreciation of the PSAs. The answers are presented per reader since this appreciation is expected to be a personal preference. Reader 1: The other source documents clarified the most. (web site) Especially the infrastructure pictures and the Archimate figure. (SAP) Reader 2: The sentence from this message, the ( ) application is initiated answered the only question I had after reading the initial project documentation. (accounts payable) I only started to understand when I saw the pictures. (web site) The contents are not good enough, so the form is irrelevant. (web site) How wonderful the PSA template was used. (serious remark on SAP) 7 Quality requirements are a well-known term in architecture. They are mostly non-functional in nature. They answer questions on e.g. portability, availability and time-to-market of changes. In software, they are also known as software quality attributes and defined in ISO standard Effect of PSA document quality on project success page 21 of 53

22 This is a good PSA: it shows the solution and explains why it was chosen. The issues are mentioned, respected and a choice is made explicit and explained. (SAP) Reader 3: The structure of the PSA and the tables. (SAP and web site) Process scheme; the relationship -application mentioned in 4.2 (of the PSA). I missed the link application process and the consistency. (accounts payable) Next are all answers to the question: Do you think this PSA increases the quality of the solution and the chances on project success? Why? Why not?. They are presented per PSA. SAP: No alternatives are shown, only the preferred solution. So I think it directed the project well, but I do not know if the solution is the best one. (reader 1) The solution is shown well, but I don t agree with it! (reader 3) Web site: These two chapters (infrastructure and application) were clear about the solution, but no rationale was given. It probably helped the project in delivering, but I don t know about the quality of the solution. (reader 1) It lists the issues and gives answers, so it does increase project success chances. But I don t think it enhances the quality of the solution. (reader 2) No alternatives are given and no reason why this solution is chosen. So it enhances project success, but I don t know about the quality of the solution (reader 3) Accounts payable: Although the PSA gives answers, it does not give the rationale so discussions may arise during execution. If the document is the result of discussions with these people, then it is all right.. (reader 1) Not really. (reader 2) The solution is shown well; it will certainly enhance project success. (reader 3) The third question is: Which (content) elements have the biggest effect on the score? Keep in mind that this question does not play a role in answering the research questions of this thesis but is only asked for exploration of possibilities for follow-up research. This question is answered less specific by the readers. They all said it was the PSA as a whole. One interviewee pinpointed a specific table (functionality vs. software modules, i.e. relation between business process and application), but this was not supported by the other readers for the same PSA. Another reader said that one specific sentence made the biggest difference in understanding. This sentence explained a specific relationship between application and process, as did the previous answer. The answers to the previous question may also be used (with caution) to provide some insight in the relevancy of the different content elements. 4.4 Summary With one disputable exception, the tangibility of all projects after reading the PSAs, were at least 3. Readers did not always agree but trends were clearly visible. The scores on quality requirements benefited most from the PSAs. The overall document quality characteristics scores are in line with the tangibility scores and the open questions. Effect of PSA document quality on project success page 22 of 53

23 5 Conclusions 5.1 Tangibility scale The scale determined by van Aken (2002) is adapted to this research, as explained in detail in Appendix 4. Summarized, the planning and team questions are omitted. Question B, how is the project to be executed, is a special one. This aspect was expected to be important, since PSAs not only explain the final situation (solution) but also show the roadmap, e.g. intermediate stages. All three projects studied were however about new processes, applications and infrastructure, so no intermediate stages were present. The question was therefore interpreted by all readers as a planning question. All PIDs and other source documents somehow contained this planning information, while, logically, the PSAs didn t. Therefore, the readers scored this question higher or equal before reading the PSA than after, or answered that the PSA hadn t changed anything in their understanding because the PSA is not about planning. Considering this, question B was not used for analysis and conclusions. Question E is not about tangibility, yet is important for the quality of the solution itself. During the first test interview, questions A ( clear result ) and K ( I do not know what the result will be ) appeared to be the same question, but asked using a positive and negative sentence structure respectively. The second question was removed and the relative weight of the score on question A was doubled. Staying as close to van Aken as possible, the formula becomes: Tangibility = 2 x clear results + 1 x dependency on external factors. As a cross check, it was also tested whether tangibility measurement would be different when excluding question I. The differences in the answers on question C are only minimal (2 answers +1, one answer -1) and the net effect on the tangibility is hardly present (3.37/3.55 becomes 3.44/3.67). In a qualitative sense, nothing changes. The formula did therefore not need to be changed and question C was kept included. 5.2 Tangibility As expected, the readers experienced and scored tangibility differently, probably because of differences in knowledge, role and experience. This confirms that the scale is subjective. However, the overall picture from the measurements was consistent among readers, which confirms that tangibility measurement can be used as an instrument. The measurement itself only takes 5-10 minutes, which makes the instrument usable in practice as well. A summary of the tangibility scores is pictured in Figure 4 below. Two reasons were uncovered for the reader-project combinations in which the tangibility did not rise: 1. the tangibility is already high; 2. the PSA is expected to give more clarity than it actually does. If tangibility is indeed as important as expected, the PSA needs to be more tailor-made. It needs to take into account the already present tangibility and give more emphasis to the areas that are still unclear. Other researchers have already answered what areas should be covered in the PSA. This research shows that, depending on the knowledge and information already available, one should elaborate only on those subjects where knowledge is to be gained. In the PSAs studied, the quality requirements are an example of these needs, as are the relationship between business process and applications. The current PSA template in this organisation and in literature is suitable for this tailoring, but from this small sample it is clear that authors are not aware of this need or do not have the necessary competences to act upon this knowledge. Effect of PSA document quality on project success page 23 of 53

24 Tangibility before and after PSA reading 5,0 4,5 tangibility score 4,0 3,5 3,0 2,5 2,0 1,5 1,0 0,5 Before After 0,0 r1 SAP r2 SAP r3 SAP r1 web site r2 web site r3 web site r1 accounts payable r2 accounts payable r3 accounts payable reader & project Figure 4. Tangibility changes for all reader / project combinations 5.3 Quality characteristics The first analysis is the separation of content (what) and format (how). For each project, the scores of the three readers are averaged. This is allowed since the readers have different norms, but these norms are constant. This was checked a couple of times by asking the readers during the interview for the third project: what is your score for this question for PSAs #1 and #2?. The answers were consistent. The overall scores for the PSAs were 3.6, 3.4 and 3.3. For the following paragraphs, one must bear in mind that the number of content questions was only 13 against 49 format questions. When only the answers to the content related characteristics are taken into account, the scores are 3.9, 3.4; 3.4. For the format related questions, the scores were 3.5, 3.4 and 3.3 respectively. Figure 5 summarizes these scores, Appendix 10 contains all scores. The coloured bars show the average scores; the vertical lines denote the minimum and maximum scores. The results for format are pretty close to each other. The question arises whether the readers are relatively indifferent to the format elements of the PSAs and thus score all questions about the same, or that the PSAs are comparable in quality. Might it be that the template has an important influence on the format score? For that purpose, all questions are separated into template related and not template related. Keep in mind that this is another viewpoint than content or format: some questions can be content as well as format related, see Appendix 5. When these template questions (content as well as format related) are not taken into account, the PSAs score almost equal on quality (3.50, 3.37 and 3.37 respectively, n=37). When only the template related questions are taken into account, the scores become 3.79; 3.57; 3.21 (n=16). The scores for these template related questions stay almost the same when differentiated between the content related template questions and the format related template questions and therefore neglected further. Figure 6 shows the integral scores. Effect of PSA document quality on project success page 24 of 53

25 5,0 4,5 4,0 3,5 3,0 2,5 SAP w eb site accounts payable 2,0 1,5 1,0 overall score format only content only SAP 3,58 3,50 3,89 w eb site 3,43 3,42 3,44 accounts payable 3,32 3,28 3,44 Figure 5. Quality characteristics: content and format separated 5,0 4,5 4,0 3,5 3,0 2,5 SAP w eb site accounts payable 2,0 1,5 1,0 overall score template only non-template SAP 3,58 3,79 3,50 w eb site 3,43 3,57 3,37 accounts payable 3,32 3,21 3,37 Figure 6. Quality characteristics: template and non-template separated From these scores the conclusion is drawn that the non-template related questions are even more evenly answered. The PSAs only clearly scored differently on the aspects of how the information was presented (from the viewpoint of the reader or not), and if Effect of PSA document quality on project success page 25 of 53