Joint Application Testing (JAT)

Transcription

1 Presentation Paper Bio Bio Return to Main Menu Return to Main Menu P R E S E N T A T I O N F7 Friday, Nov 12, 1999 Joint Application Testing (JAT) Wim Blommaert International Conference On Software Testing, Analysis & Review NOV 8-12, 1999 BARCELONA, SPAIN

2 Joint Application Testing th Anniversary Wim Blommaert Eurostar 1999 Software Testing Services Copyright 1999 ps_testware - Joint Application Testing - 1

3 5th Anniversary Software Testing Services Joint Application Testing Better, Cheaper, Faster Copyright 1999 ps_testware - Joint Application Testing - 2

4 Introduction Wim Blommaert ps_testware Copyright 1999 ps_testware - Joint Application Testing - 3

5 Agenda Some facts Joint Application Testing A classical gap in testing The engineering model About communication User involvement Interviews & workshops Kernel teams Convincing IT and Users Portfolio analysis Examples Copyright 1999 ps_testware - Joint Application Testing - 4

6 5th Anniversary Software Testing Services Some Copyright 1999 ps_testware - Joint Application Testing - 5

7 Y2K Functional testing Critical No knowledge 187 No documentation IT Business Testers No reusable tests 1/8/98 1/12/98 1/2/99 1/4/99 1/6/99 Copyright 1999 ps_testware - Joint Application Testing - 6

8 Quality Quality Experience with test planning Lap Time (Weeks) Cost (Mandays) Time Quality (Business Link) Money classical finished Y2K Finished classical planned Copyright 1999 ps_testware - Joint Application Testing - 7

9 How did we do this IT and USERS joining together in testing To provide high quality software, on time, within budget JAD JAT By essentially filling the testing Gap Using super efficient communication techniques Copyright 1999 ps_testware - Joint Application Testing - 8

10 A classical gap in testing (1/3) Strategic choices Requirements V-model Audit test Right Acceptance application test / trust Logical design Technical design Program design Functional System Correctness test Integration test Technical Correctness Module test Coding Coding Testing Copyright 1999 ps_testware - Joint Application Testing - 9

11 A classical gap in testing (2/3) IT : Technically correct User : Is this what I need Should I use it IT User Module & Integration Testing System Testing Acceptance Testing Copyright 1999 ps_testware - Joint Application Testing - 10

12 A classical gap in testing (3/3) Consequences System tests are sometimes done twice (overlap) More often system tests are not done properly System tests are often technical or not detailed enough Copyright 1999 ps_testware - Joint Application Testing - 11

13 The engineering model (1/2) This SW car is approved by by factory IT QA QA How nice. Let s try it. IT User Module & Integration Testing System Testing Acceptance Testing Copyright 1999 ps_testware - Joint Application Testing - 12

14 The engineering model (2/2) Acceptance testing becomes a test drive serving two purposes Assuring that the right software is built Build confidence Copyright 1999 ps_testware - Joint Application Testing - 13

15 About communication (1/2) What are the priorities What shall we test What is it supposed to do... I ll tell you IT Communication is essential User Copyright 1999 ps_testware - Joint Application Testing - 14

16 About communication (2/2) What are the priorities What shall we test What is it supposed to do... IT Building consensus quickly is essential Copyright 1999 ps_testware - Joint Application Testing - 15

17 User involvement Strategic choices Requirements V-model Audit test Acceptance test Logical design JAD Technical design System test JAT Integration test Program design Module test Coding Coding Testing Copyright 1999 ps_testware - Joint Application Testing - 16

18 5th Anniversary Software Testing Services Using super efficient communication techniques Copyright 1999 ps_testware - Joint Application Testing - 17

19 Interviews What do you need to know No consensus building needed Easy 10 Commandments Bonus Copyright 1999 ps_testware - Joint Application Testing - 18

20 Workshops(1/4) We need to build consensus Consensus building needed Moderated and Structured Difficult Copyright 1999 ps_testware - Joint Application Testing - 19

21 Workshops (2/4) A meeting to quickly build consensus based on groups moderated interactive clear objectives defined scope limited in time agenda agreed results Copyright 1999 ps_testware - Joint Application Testing - 20

22 Workshops(3/4) Facilitation Organisation Establish and maintain an environment that enables super efficient communication and consensus building Rules Copyright 1999 ps_testware - Joint Application Testing - 21

23 Workshops (4/4) Facilitator Result Format Does not participate in discussion Copyright 1999 ps_testware - Joint Application Testing - 22

24 Kernel teams (1/3) People participate in more than one phase of the V-model The purpose is to improve the transfer of knowledge Sometimes a trade off has to be made between independence and optimal communication Copyright 1999 ps_testware - Joint Application Testing - 23

25 Kernel teams (2/3) Strategic choices Requirements V-model Audit test Acceptance test User Logical design JAD Technical design Program design System test Integration test Analyst Module test Coding Copyright 1999 ps_testware - Joint Application Testing - 24

26 Kernel teams (3/3) Strategic choices Requirements V-model Audit test Acceptance test User Logical design Technical design System test JAT Integration test Program design Analyst Module test Coding Copyright 1999 ps_testware - Joint Application Testing - 25

27 Convincing IT and Users (1/6) I don t have time to do a lot of testing (System) Testing is not my job We have to do a lot of tests ourselves Acceptance testing becomes a test drive, Quality is provided by the factory. Testing will be done by professionals. User Copyright 1999 ps_testware - Joint Application Testing - 26

28 Convincing IT and Users (2/6) I do not trust IT to do my tests IT does not know the business You will fully participate in the System Tests providing knowledge You will still do an acceptance test User Your acceptance test will be used as input for the system test Copyright 1999 ps_testware - Joint Application Testing - 27

29 Convincing IT and Users (3/6) I already have a test set. We will just do the work all over again Your test set will be fully re-used when possible and tests will be added to provide more depth if needed Your test set is still needed for the acceptance test User System tests and Acceptance tests have different goals, are different tests Copyright 1999 ps_testware - Joint Application Testing - 28

30 Convincing IT and Users (4/6) I already have a good test set Your test set will be fully re-used when possible and tests will be added to provide more depth if needed How can the users help us They know the business. They will provide resources. IT Copyright 1999 ps_testware - Joint Application Testing - 29

31 Convincing IT and Users (5/6) Shouldn't the users test the functionality Do you expect your car to turn the right degree when you turn the wheel What s in it for me The users will gain confidence in the SW The relationship with the users will improve You will get extra resources IT Copyright 1999 ps_testware - Joint Application Testing - 30

32 Convincing IT and Users (6/6) So I am no longer responsible for the functionality Yes, you are but now you can do a better job IT Copyright 1999 ps_testware - Joint Application Testing - 31

33 Sco pe Test Planning Plan Design Tes t Development Follow-up Build Tes t Tes t execution Repair Ret est Portfolio Analysis Pragmatic Approach WHAT Resources, commitment H L L Interviews Confirmation workshop P1 Interviews P3 Need for consensus Workshops P2 P4 Workshops Straw man model Interviews H Copyright 1999 ps_testware - Joint Application Testing - 32

34 Example (1/3) Y2K Projects V-model Requirements Logical design User (business knowledge) Technical design Program design Test coach (test knowledge) Acceptance test System test JAT Integration test Developer (application knowledge) Module test Coding Copyright 1999 ps_testware - Joint Application Testing - 33

35 Example (2/3) New Project V-model Requirements Acceptance test WHAT Logical design User TST System test Technical design Program design Coding Integration test Module test Copyright 1999 ps_testware - Joint Application Testing - 34

36 Example (3/3) New Project V-model Requirements Acceptance test User Logical design TST System test Technical design Integration test Program design Module test Coding Copyright 1999 ps_testware - Joint Application Testing - 35

37 5th Anniversary Software Testing Services Questions Copyright 1999 ps_testware - Joint Application Testing - 36

38 Wim Blommaert Wim Blommaert has eleven years of IT experience mainly in a banking environment. During six years he worked at a large data centre as a capacity planner. Subsequently, he spent two years leading a development methodology group supporting +/- 800 developers. Wim Blommaert was made responsible for setting up a large test organisation for a big European bank. As a Y2K test manager, he finally managed their Y2K test program. Wim Blommaert joined ps_testware in June 1998 as a management consultant. In this capacity, he was involved in setting up and managing Y2K programs and test organisations for different companies (including large banking organisations and production environments). In 1999, Wim became Service Delivery Co-ordinator for ps_testware managing a large number of projects in the Benelux Countries. 1

39 Joint Application Testing Wim Blommaert ps_testware nv Tiensesteenweg 329 B-3010 Leuven Tel Fax ps_testware BV Stephensonweg 14 NL-4207 HB Gorinchem Tel Fax Abstract The JAT concept was not something I developed overnight. The concepts had been lingering in my mind for at least five years. They then started to surface out of pure necessity. Only after I had seen most of them at work in practice, I had the time to bring them all together in the concept of JAT. In my previous job, I supported developers being responsible for the development methodology in a large IT department. It was in that capacity that I came in contact with the concepts of Joint Application development (JAD). I followed courses, wrote guidelines and finally became a JAD tutor. I was also the facilitator in numerous workshops. The concept of JAD focuses on the co-operation of developers and users when developing software. Most of the time, it is associated with Rapid Application Development (RAD) but this is mainly due to lack of understanding in this area. You don t need to apply RAD to use the JAD concept. The key technique is what is called a workshop or a JAD session. A curious fact (which I could only appreciate later when I got involved in testing) was that in the entire story of JAD not a single word was spent on testing. When I got involved in testing four years ago, I was so busy that thoughts about JAD again dozed off. Until 1998 when I got heavily involved in setting up and running Y2K test projects. These Y2K test projects were facing the following problems: Extreme time pressure 2

40 Low user involvement Lack of test resources Lack of knowledge regarding the applications that had to be tested Confronted with these issues, I started to implement a number of ideas that, in retrospect were things we had used when applying JAD but now applied to testing. So it was obvious (at least after a while) to call the concept JAT referring to the fact that it is JAD applied not to the left hand side of the V-model but to the right hand side. It s this concept of JAT that is explained in this document. If applied properly you can benefit from it just as you can from JAD. Since I borrowed a lot of the ideas of JAD it should not surprise you that occasionally I will refer to JAD to clarify certain elements. Enjoy your reading. Key words: Joint Application Testing (JAT), Joint Application Development (JAD), System Testing, User involvement, testing gap, Engineering model, Kernel teams, Portfolio analysis, Super efficient communication, Interviews, Workshops. 3

41 1. Objectives This paper supports the following objectives: The reader will understand the concept of Joint Application Testing (JAT). The reader will be able to start using the JAT concepts to his or her advantage. The reader will be provided with a number of practical tips. The reader will know how to convince IT and Users to participate in JAT. This paper does NOT support the following objectives: The reader will not be able to execute workshops. 2. Intended Audience. People that might benefit from this presentation include: System Testers who want to co-operate intensively with the user community. Users involved in testing. Managers who need to convince both IT and Users to co-operate in testing. 4

42 3. Joint Application Testing (JAT) 3.1 Some facts In this chapter, I hope to illustrate the benefits of JAT if applied properly. I will do this by providing some objective information on projects to which it was applied. Y2K In one company, we managed to successfully test (in a structured way) 20 applications for Y2K in few months time using JAT. Users were always involved in these projects providing at least 20 % of the resources. The applications that were tested were selected based on following criteria: They were business critical No documentation was available No or very limited knowledge was available No reusable tests existed It was perhaps a doubtful honour to test only these applications but the results were nevertheless very good. The number of Y2K errors found in these applications was 10 times higher compared to (less difficult) applications that were tested in a non-structured way without using the concept of JAT. After the users had been involved in these tests (following the concepts of JAT) they had no more need to do any tests of their own. This was not the case for the other applications on which both IT and users executed tests. Test planning In another company, we had been defining what to test without using any of the JAT techniques in the planning phase of a test project. Everyone realised that this took too much time and consumed a lot of resources. Alternatives, however, were not presented When we started a Y2K project with a size comparable to the new projects in the same company we had been doing, we started using JAT techniques since we were confronted with the problems that were listed above (time pressure). The same task (defining what to test) took only 25 % of the time and 20 % of the resources. Now even when we take into consideration that these projects were not exactly the same size and that they already existed (legacy systems), these numbers were very significant. When the planning for the next new project had to be made, we felt sure enough to offer the customer a fixed price contract in which, compared to the previous project, both lap time and cost were reduced by a factor 3. These reductions were made possible by the introduction of JAT. Remarkable was that not only the lap time and the cost were reduced but that the quality of the work also increased. Whereas in the first project the decision of what to test had been made mainly from a technical point of view, in the Y2K project these decisions were taken from a business point of view. This was made possible by the fact that the users (following the concept of JAT) were heavily involved in this phase of the project. 5

43 This proves that by the introduction of JAT was it not only possible to reduce cost and time but also to increase quality. It should not have come as a big surprise since this is exactly what is promised if you apply Joint Application Development (JAD) properly. 3.2 How it was done When I finally thought about the concept of JAT, I came up with three main elements that had helped us to realise the improvements described above. We were able to reduce time and cost and to increase quality by: IT and Users joining together in testing. Essentially filling what I call the testing gap. Using super efficient communication techniques. I will explain each of these topics in the next chapters. Now to announce that one is going to improve in the three classical dimensions of software development (time, cost, quality) seems odd or even unbelievable since most people assume that a gain in one (or two) of the dimensions has to be at the expense of the other one(s). If you are familiar with the concepts of JAD, this should be less surprising because when JAD is applied properly, it is supposed to produce a benefit in all three dimensions. Further lecture will hopefully clarify why this is possible. Keep in mind that the benefits of JAD and / or JAT can only be high when they are applied properly. I have seen a lot of projects in which the application of these concepts backfired on the inexperienced team. In short, applying them is not easy but can be very rewarding. 6

44 3.3 A classical Gap in testing If one looks at the V-model (The Gartner Version also used at ps_testware) simply put the test phases have the following purpose: Build phase test phase Purpose Who tests Define Requirements Acceptance testing Build the right product Users Build Confidence Functional Analysis System testing Functional correctness Technical Analysis Integration testing Technical correctness IT Program Design Module testing Coding Now more often then not IT people are mainly interested in technical correctness focussing heavily on module & integration testing. IT clearly sees these tests as being part of their responsibility. Users on the other hand are more focussing on acceptance testing. They would like to assume that both technical and functional correctness was already achieved at a lower level. Users will normally (although I have seen cases where they to my surprise did not) assume a natural responsibility for acceptance testing. The problem occurs in the middle where it is not so clear who should take the lead. Who is responsible for system testing For some the answer is trivial but reality shows it is not. As a result of this confusion sometimes system tests are done double once by IT and then repeated by the users. Some people even suggest that users should exactly repeat the systems tests made by IT and call them acceptance tests. This redundancy costs a lot of money. Often systems tests are not done properly because both IT and Users see it as the responsibility of the other party to do them resulting in none being done. Or IT executes Module & Integration tests and calls it systems tests. The same often goes for the users whom all to easily promote their acceptance tests to system tests. Even if one party clearly takes ownership of the system tests problems often occur. If IT takes ownership the system tests they execute are often too technical in nature (instead of functional). If the users assume the role of systems testers these tests are often not done detailed enough because the users are already assuming a certain level of functional correctness or are more thinking about executing an acceptance test. In reality the end result is very often a gap in the right hand side of the V-model or in testing. 7

45 3.4 The engineering model When we want to plug this gap in the V-model, the first answer of course is who should take responsibility for the system tests. My answer to this question is, what I call, the engineering model. The engineering model is something we all use frequently but as a reader it might sound very unfamiliar. The engineering model is related to the way commodity goods like cars, VCR s, PC s are made. When you buy a car, VCR, PC, the only type of tests you perform as a user are acceptance tests. You will certainly not do any technical tests 1. Will you do system tests These tests are related to functional correctness. When I buy a car I assume that it is fully technical and functional operational. I will not risk stepping in this car when this is not the case. The only thing I normally do is an acceptance test, a test drive in fact. There are only two things I should be worried about: Is this the car I want, need, asked for (size, colour,) Do I have enough confidence in this car to buy it All the other things should be the concern of the factory. Why should we not want to act the same way in the case of software We all know that the word software engineering does not yet provide the same guarantees we are used to as far as the production of cars is concerned. But decent testing should help to make progress in this area. If one thinks about software in terms of the engineering model, it is clear that system testing is the responsibility of IT and that acceptance testing can be reduced to a test drive on condition that system testing is done properly The purpose of acceptance testing becomes: To assure that the right software is built (based on the user requirements) To give the users confidence in the fact that they will be able to do their business with this piece of software. 3.5 About communication What is the key factor to success in software engineering and in testing Having the smartest programmers Having the best testers In fact, when you take some time to consider it, the most important issue is communication. Being able to communicate effectively and efficiently is of the utmost importance. In fact what is produced is pure information (unless you consider the disc on which the software can be put). In a lot of cases, communication is in a lot of cases the cause of failure in projects. Getting priorities mixed up, implementing the wrong functions, testing parts of the software that hardly support the business, not testing important business functions. That is why both JAD and JAT focus heavily (perhaps even exclusively) on the improvement of communication in IT projects. 1 I know that some of you in fact do (more as a hobby) but I m quite happy to use my car without having to take it apart. I would never be able to get it assembled again anyway. 8

46 One aspect of communication is getting consensus. When more than one person is involved in a project (by definition the case) getting consensus becomes important, consensus on priorities, test strategies, Getting consensus can be very time and resource consuming (or even impossible) without the right techniques. This is why both JAD and JAT also concentrate on techniques for rapid consensus building. In example 2 (test planning) of chapter 3.3, it took the company extremely long to define what to test. This was mainly a problem of getting consensus. People who do not know techniques that allow for rapid consensus building get used to endless discussions. In the end, it is the manager who has to decide resulting in no consensus at all. 3.6 User involvement in JAD and JAT In classical development it is often assumed that user involvement is minimal once the user requirements have been defined since IT can take over from there. In a lot of cases that I have seen users involvement was not even clear in this first phase (defining user requirements) IT took over right from the start. The concept of JAD is based on a strong co-operation between IT and users. The effect is that users get involved very early in the development process and stay involved at least until functional analysis (possibly even later). Unless the user interface is defined in the more technical phases, user involvement in these phases is often not needed. The key element in this co-operation are workshops (or JAD sessions) during which both IT and users participate at least until the processes, data and user interface have been defined. If you look at the right hand side of the V-model (testing), the same ideas are still valid. In classical testing the users would receive the software fairly late in the process. They are then expected to accept the software using an acceptance test. Almost never, have I seen users who got involved in the tests IT did, not even when JAD was used. So even when people use JAD effectively, involving the users more on the left hand side of the V-model it is not so obvious to do the same thing on the right hand side. If you use the concept of JAT the users do get involved in the tests executed by IT. How this can realised will be explained in the next chapters. This user involvement will mainly focus on the system tests. Although IT will drive these system tests, users will get involved heavily. The benefits of doing this have already been mentioned. The resulting system tests will be of a higher quality reducing the need for acceptance testing. Next to this, the users will gain additional confidence in the product because they get involved in these system tests. This will again reduce the need for acceptance testing. User involvement in the more technical tests (module and integration tests) is normally not needed. 9

47 4. : using super efficient communication techniques 4.1 Interviews Interviews are very useful when you do not have to build consensus in a group of people. E.g. When you need information from a specialist, interviews are relatively easy to execute (at least when compared to workshops). Nevertheless there are some rules that can help when interviewing people. I call them the ten commandments of a good interview. Make sure you know the answer to the following questions. 1. SETTING: why do you want to interview this person 2. SCOPE: what do you want to know 3. DETAIL: what do you specifically want to know from this person 4. RELATIONSHIP: what type of relationship do you want with this person in the future 5. RESULT & WRAP UP: what are you going to do after the interview 6. INTRODUCTION: do you have an introduction to the interview 7. STRUCTURE: are you going to structure the interview 8. SCENARIO: what would be a good scenario for the interview 9. TIME: how much time do you allocate for the interview 10. LOGISTICS: Are there any practical points of attention (room, drinks, ) 4.2 Workshops Workshops are a completely different ball game. Within the scope of this text, it is not possible to discuss the issues related to workshops in any level of detail. I will therefore limit myself to the main issues. Workshops are extremely useful when consensus has to be reached within a group of people. Since we are talking about the concept of JAT, these groups will typically include both developers and users. Without the proper use of workshops, getting consensus can take forever or will possibly never be reached. A workshop is lead by a moderator or facilitator. This is one of the key elements of workshops. It s very hard to be a good moderator in a workshop. People often underestimate this. Especially when the groups are big (12 people has to be considered the maximum) and the participants have very different opinions, a workshop can easily become a facilitator s nightmare. Workshops are very structured. There is a fixed set of rules that the participants will have to follow (try getting everyone to be exactly on time and to turn of their mobiles). At first some people might argue that the workshop is too formal but once the participants get used to these rules and start appreciating the efficiency of communicating in this way they will easily accept them. Often I ask people if they use workshops. All too often the answer is yes. When I then make a further inquire into how this is done it turns out that what they call workshops does not fit the following definition. A workshop is a meeting used to build consensus quickly and: is based on groups is moderated is very interactive has clear objectives has a defined scope 10

48 is limited in time is built around an agenda produces agreed results is formal Some of these so-called workshops turn out to be nothing more than inefficient informal chats. Some people even tell me they have two moderators in their workshops. Or they tell me that after the workshop no written minutes of the results are made. If workshops want to be successful they have to be very formal. This is where the real power of the workshops lies. The three pillars of a good workshop are: A good facilitator Good organisation (including logistics) Appropriate use of the rules. Getting a skilled moderator will solve all of this because he or she will know what to do. The whole idea is to create and maintain an environment in which communication can be super efficient. Difficult but very rewarding if you succeed. One last word about the facilitator. Not everyone is fit to be a facilitator in a workshop but it is a skill that can be learnt given the person has some initial (mainly communication) skills in this area. One of the hardest parts of being a facilitator is not being allowed to participate in the discussions. The facilitator really has to keep two things in mind. On the one hand, to make sure a result is reached within the given time limit and on the other hand to make sure this result is reached in a civilised way paying attention to how it is reached. The facilitator has to focus on the content as well as on the format of the workshop. This may seem easy but it is not. Often a facilitator will under pressure of time drop any consideration for the format of the workshop and start concentrating on reaching a result one way or the other. I have witnessed (more than once) facilitators being very rude to the participants in a workshop in their urge to get a result. One facilitator even ordered one of the participants (literally) to shut up. The other extreme is also possible. Some facilitators are so eager to have a nice and civilised workshop following all the rules that they forget that a result has to be reached. The end result is that they manage to run a model workshop but end up without any result whatsoever. 5. Kernel teams One of the concepts that I used in a previous job was something we called the use of kernel teams. The concept is simple. Instead of people specialise in the execution of one phase of the V-model you have some people participating in more than one phase. I remember a project in which we had a small team (the kernel team hence the name) executing all the phases. Per phase additional people were then brought in just for that phase. This meant that analysts wrote lines of code whereas programmers defined entity relation diagrams. The biggest advantage is an improved communication. Instead of just information (documents), people will pass from one phase to the other. This will improve communication a lot. 11

49 An example on the left hand side of the V-model could be a user who has been involved in setting up the user requirements becoming a full team member in the development team participating in the functional analysis. This will seriously reduce the possibility of the requirements being misinterpreted. A few months ago it occurred to me that the same concept could be of value on the right hand side of the V- model. The motivation would be the same: to improve communication. An example could be a user who has been involved in defining the user acceptance test joining the system test team. This will help to make sure that the right priorities are set while defining these system tests and to make sure systems test are done based on the reality of the system taking into account what it will actually be used for. One can even imagine an analyst who has been involved in writing the functional specs joining the system test team. This example clearly shows that one has to be careful in defining kernel teams in testing. Taking part in more than one phase on the right hand side of the V-model might not always be a good idea. The problem is that in testing a certain level of independence between the person that wrote the specs and the one that tests them is assumed. From this point of view a lot of the interesting kernel team scenarios are not possible because the independence will be lost. Does this mean that these scenarios are to be avoided I personally do not think so. The issue here is the balance between the need for independence and the need for optimal communication. Both advantages have to be balanced the one against each other. The choice really depends on the actual situation. If communication is the most essential issue, independence might be sacrificed in order to gain it. 12

50 6. Convincing IT and Users Supposing you want to try JAT you will have to convince both the users and IT to participate in this. In practice, I have often seen that both parties have their reasons to resist in case the organisation wants to start using JAT. I will give a list of the reasons that I have come across and the answers I think are appropriate in dealing with this. Since the concept of JAT is based on the co-operation between IT and users being able to get their commitment is crucial. Point of view of the users: Q: System testing is not my job. I do not have a lot of time to do all these tests. I already have enough work doing my acceptance tests. A: It is the job of IT to do the system tests but good system testing will reduce the need for acceptance testing. Professionals will do system testing. Q: I do not trust IT to do the systems tests. IT does not know the business. A: You will still do an acceptance test. You will fully participate in the system tests providing business knowledge. Your acceptance test can be used as input for the system test. Q: We already have tests. We will just do the work all over again. A: The existing tests will be reused where possible and additional tests will only be added if needed. Your tests are still needed to execute the acceptance test. System tests and acceptance tests are different type of tests and have different goals. Your acceptance test might not be suitable to be the basis for the system test. Point of view of IT: 13

51 Q: We already have a test set. What do we need the users for How can they help A: The existing tests will be reused where possible and additional tests will only be added if needed. The users know the business and will provide additional resources. Q: Is it not the job of the users to test the functionality What do I have to gain from this co-operation A: Do you expect your car to turn to the right the correct amount when you turn the wheel You get additional resources. The users will gain additional confidence in the software. The relationship with the users will improve. Q: Users get involved. So IT is no longer responsible for testing the functionality then A: Yes you are but now you can do a better job. 14

52 7. Portfolio Analysis I wanted to include some lines on Portfolio analysis because it is about being practical. Portfolio analysis is based on the fact that a standard way of doing things (like JAT) may be nice but it will never exactly fit every project in your organisation. Projects can be very different in nature needing a different approach. Now on the other hand we can not afford to come up with a new approach for every project. You do not have to. What ps_testware always suggests is to make a portfolio analysis of your projects to help you decide how a general concept (like JAT) can be modified to fit certain type of projects. The first step is to group similar projects together. The next step is to define the approach per group but still starting from the general concept of JAT. I will explain this by using an example. Suppose we grouped the projects according to two parameters: the need for consensus building and the ability to do workshops (based on availability of resources and commitment from users and IT). There could be 4 groups: 1. The need for consensus is high, and so is the ability to do workshops. 2. The need for consensus is high but the ability to do workshops is low. 3. The need for consensus is low, and so is the ability to do workshops. 4. The need for consensus is low but the ability to do workshops is High. The approach for each group could be: 1. Use workshops. 2. The trick here is to reduce the work that has to be done in the workshops. Use interviews to prepare the workshops. You can also use a straw man model to further reduce the work to be done in the workshops Use interviews. 4. Use interviews to build the result. You can use a workshop to confirm this in-group. Portfolio analysis will help you to avoid trying to apply a theoretical concept in a situation where it will not work. 2 When using a straw man model the facilitator has prepared what he or she thinks to be the solution. The facilitator will then try steering the group in that direction. Very dangerous. I personally never use it. 15

53 Wim Blommaert Wim Blommaert has eleven years of IT experience mainly in a banking environment. During six years he worked at a large data centre as a capacity planner. Subsequently, he spent two years leading a development methodology group supporting +/- 800 developers. Wim Blommaert was made responsible for setting up a large test organisation for a big European bank. As a Y2K test manager, he finally managed their Y2K test program. Wim Blommaert joined ps_testware in June 1998 as a management consultant. In this capacity, he was involved in setting up and managing Y2K programs and test organisations for different companies (including large banking organisations and production environments). In 1999, Wim became Service Delivery Coordinator for ps_testware managing a large number of projects in the Benelux Countries.