4. COLLECTING PRIMARY DATA

Transcription

1 Section V How-To s 4. Collecting Primary Data 4. COLLECTING PRIMARY DATA A. Overview Needs assessments typically require not only use existing (secondary source) information such as epidemiologic data but also new (primary source) information. Frequently used primary data collection methods for needs assessment include surveys, focus groups, key person interviews, and community forums. Surveys typically provide quantitative data, while the other methods are most often used to obtain qualitative information. The data collection methods used in needs assessment depend on many factors including resources. Many Title III and Title IV grantees as well as rural Title II States have limited funds for needs assessment. This chapter suggests some primary data collection methods that are affordable and practical for such programs. In addition, collaboration with other titles or regions can help to expand or make more effective use of available resources. B. Common Data Collection Methods.. Among the most commonly used methods of primary source data collection for needs assessment are the following: Su rvey s : a method in which a sample of individuals is selected from a target population to respond to a structured set questions. Questions are usually short answer or closed-ended (i.e., provide a limited set of responses that a respondent can select from). Surveys may be conducted in person (an interv i ewer sits down with a respondent), over the telephone, or self-administered (the respondent completes the survey alone). The information (mostly quantitative data) is used to generate summary statistics about respondents and, where appropriate, sampling is used to generalize findings to the target population. Focus groups: a method of collecting qualitative data involving a carefully planned small-group discussion of specific questions or topics led by an experienced moderator. Key person or key informant interviews: a qualitative method involving in-depth interviews with a small number of individuals carefully selected because of their personal experiences and/or knowledge related to the topic of interest. A discussion guide is used to ensure that major topics and issues are addressed. Ryan White CARE Act Needs Assessment Guide V-47

2 . Section V H.. o. w.-. To.. s.. 4. Collecting Primary Data Community forums: a method of collecting qualitative information from community members using a small group, similar to a focus group but usually larger and less formal and with little or no pre - s c reening of participants. A trained moderator facilitates discussion on issues and topics identified in a topic guide, and observers are present to take notes and/or tape-record the session. Information is provided below on how to design and use each of the these primary data collection methods in HIV/AIDS needs assessment. DATA RELIABILITY AND VALIDITY When developing primary data collection methods, consider reliability and validity: Reliability refers to the consistency of a measure or question in obtaining very similar or identical results. If you asked a set of questions to the same people repeatedly, you should be able to get the same results. Validity refers to the extent to which a question or instrument actually measures what it is supposed to measure. For example, if you want to find out what PLWH consider to be their own most important service needs, but your question asks them to identify the most important service needs of PLWH in your community, the question may not generate valid information. C. Surveys Surveys in HIV/AIDS needs assessment are most often used to obtain information from PLWH and from providers. (For more information about provider surveys, see Chapter 6, Conducting an Assessment of Provider Capacity and Capability. ) PLWH surveys can gather information about s e rvice needs and gaps from the perspective of intended beneficiaries, as well as satisfaction with services they are currently receiving. Su rveys also typically ask for sociodemographic information and stage of disease; this makes it possible to analyze service needs for specific populations (e.g., African Americans, women, homeless people, injection drug users). Programs that are hospital- or clinic-based, such as many Title III and Title IV programs, may find it practical to use surveys not only with PLWH enrolled in their programs but also with other HIV-positive individuals who obtain services within the health care facility. In this respect, the focus will be on the broadest array of PLWH, not just those receiving services through CARE Act funds. V-48 Ryan White CARE Act Needs Assessment Guide

3 Section V How-To s 4. Collecting Primary Data PLWH SURVEYS: CLIENT SATISFACTION VERSUS NEEDS ASSESSMENT A client satisfaction survey obtains information about PLWH satisfaction with services they are currently receiving. While a PLWH needs assessment survey may also collect such data, its primary purpose is to determine met and unmet service needs. It seeks information from PLWH who are not currently receiving services as well as those who are in care. Determining Survey Purpose and Scope In planning a PLWH survey, begin by considering the following questions and issues: What resources and time period are available for conducting the survey? Costs are affected by survey scope, length, number and mix of people to be surveyed, and data collection methods. Consider what re s o u rces are available funds for consultants, grantee staff or planning body members who can assist with data collection. Also consider the time available for conducting the survey gathering data and analyzing and reporting results. This will help you decide on the scope of your survey effort. (For more information on low-cost sources of human resources see Section V, Chapter 1, Obtaining Human Resources for Needs Assessment. ) Who will conduct the survey and what will be the role of the CARE Act grantee or planning body? Consultants or university researchers are often hired to do the actual data collection and analysis. (For more information about how to arrange for needed human resources, see Section V, Chapter 1, Obtaining Human Resources for Needs Assessment. ) Sometimes the main roles of the CARE Act grantee or planning body are oversight and review. Other roles they can play include planning and design, identifying key questions to be answered, and ensuring that the data collection methods target and effectively reach all necessary PLWH groups. What information is to be obtained through the survey? Look at your overall needs assessment plan and determine the types of populations of interest, the information to be obtained, and the specific questions to be answe red through the surve y. This helps you to determine the scope, length, and specific content of the surve y. For example, what differe n t PLWH populations do you want to learn about from the survey (e.g., race/ethnicity, gender, co-morbidities, risk factors, place of residence)? Ryan White CARE Act Needs Assessment Guide V-49

4 . Section V H.. o. w.-. To.. s.. 4. Collecting Primary Data Do you want to be able to generalize survey findings to the entire population of the group you are surveying? With a PLWH survey, for example, do you want to be able to assume that the findings from the survey reflect the same findings you would get if you surveyed the entire population of PLWH? If so, you will need to use some form of probability or random sample, most likely a stratified random sample. You may also need a fairly large sample. Even if you decide that random sampling is not feasible, you will still want to ensure that the survey respondents (those who complete the sur vey) reflect the diversity of the HIV/AIDS populations in your service area, so that you can obtain information about the perspectives of various populations. Sampling Sampling is an important consideration in survey design. A sample is a group selected from a total population with the expectation that studying the group will provide important information about the total population. If possible, you want to be able to generalize from your sample to the total population in your service area to assume that the conclusions you reach about the sample are also true of the total population. In order to do this, you will need to use a pro b a b i l i t y or random sample, of which there are several kinds: To be able to generalize only to the entire population, you can use a simple random sample. This kind of sample is also called an equal probability sample, because every person in the population has an equal chance of being included in the sample. To be able to generalize findings to specific populations such as Latinos, women, or re s i d e n t s of a particular city or county, you need either a large simple random sample or a s t r a t i f i e d random sample. You divide the population into subgroups (strata) and then draw random samples from each of the subgroups. You may want to oversample particular populations to be sure that you include in the survey enough individuals from those populations. When random sampling does not seem feasible, but you want to ensure that the survey includes people from certain populations or with specific characteristics, you may want to use a purposive sample that calls for interviewing specified numbers of people with these characteristics. V-50 Ryan White CARE Act Needs Assessment Guide

5 Section V How-To s 4. Collecting Primary Data STEPS IN DOING SURVEYS 1. Determine the survey purpose and scope, including populations to be surveyed, sample size and type, method to be used for administering the survey, and tabulation and analysis methods. Agree on the type of instrument to be developed length, topics to be covered, and mix of closed-ended and open-ended questions. Determine if sampling will be used and how data will be gathered (in person, telephone, or mail-in/ drop-off). 2. Develop the survey instrument; arrange for translation if necessary. 3. Identify and train those who will administer the survey. If the survey will be self-administered, arrange procedures for dissemination and collection. If telephone or personal interviews will be conducted, select and train interviewers. Look for interviewers who are knowledgeable about the community and HIV/AIDS, and reflect the diversity of the population to be interviewed. 4. Pre-test the survey, using the data collection methods planned for administering the full-scale data collection effort. 5. Revise the survey and/or the methods of administration as needed. 6. Collect the data, with careful monitoring and quality control. 7. Review survey forms for completeness, and do any required coding of qualitative data. 8. Tabulate and analyze results. 9. Prepare a report and presentations summarizing survey results and comparing them to other needs assessment findings. Survey Design An appropriate survey instrument or questionnaire requires a written set of questions that are used consistently. The survey can include two different types of questions: Closed-ended, f i xed choice questions. These questions can be pre-coded, making the questionnaire quick to complete and tabulate. (Pre-coding usually involves sequentially numbering each response category e. g., ye s = 1, n o = 2, no response = 0 so that the data can quickly be entered into a computer for analysis.) Closed-ended questions can be of many types, e.g., yes/no, responses based on a numerical scale such as a 4-point scale f rom 0 = not at all to 3 = always, or multiple responses where one or more categories may be checked. Each question has a set of pre-determined choices, and the person re s p o n ding Ryan White CARE Act Needs Assessment Guide V-51

6 . Section V H.. o. w.-. To.. s.. 4. Collecting Primary Data simply picks the choice that is closest to his/her desired answer. Such questions are sometimes called forced-choice questions. Surveys that use mostly closed-ended questions tend to obtain higher response rates because they take less time to complete. However, respondents who do not like any of the fixed choices may not answer the question, or may be forced to choose a response which does not accurately reflect their actual situation or belief. For this reason, such questions sometimes include an other (please specify) response category, so respondents can write in their own answers. Open-ended questions. These questions have no stated response options, so re s p o n d e n t s provide answers in their own words. These responses can be analyzed individually and used to provide a narrative description. However, particularly if the sample is large, they must be coded (grouped into categories based on content, with each category given a number) before they can be analyzed by computer. This can be time-consuming and difficult. Typically, the needs assessment team reads through the responses to a particular question and develops a set of coded categories that cover the range of responses received. Then they read each response and code it according to the agreed-upon categories. Care f u l coding is needed to accurately re p o rt what the respondent wanted to communicate, ensure sensitivity to small but important response differences, and recognize cultural differences. Surveys involving large samples usually include mostly closed-ended questions because they are easier to administer and to tabulate. However, open-ended questions can provide valuable insights. If you are not sure what response options to provide for certain closed-ended questions, you may want to include these questions as open-ended questions in your survey pre-test. In this way you can identify response options to use for making these questions closed-ended in the final survey. To help ensure successful questionnaire design: Be sure that survey questionnaires are constructed by individuals knowledgeable about issues of reliability and validity, so that the questions consistently measure what they were intended to measure (see box earlier in this chapter). Involve individuals from various populations important to the needs assessment in questionnaire preparation to ensure that questions are relevant and appropriate for each of these populations. Pre-test your survey methods and instrument on a small group of individuals similar to those who will be in the sample, including people from the various populations, to be sure that the survey is generating consistent and valid data from all these populations. Use the same method of collecting the data that you plan to use for the full-scale surve y p e rsonal V-52 Ryan White CARE Act Needs Assessment Guide

7 Section V How-To s 4. Collecting Primary Data interviews, telephone interviews, or self-administered surveys. Pre-testing is especially important for self-administered questionnaires. Aim for testing the survey with three people from each major target population. After the pre-test, hold a follow-up discussion with selected members of this pilot group to find out what questions worked for them, what questions or topics were confusing, and whether the reading level was appropriate. Be sure to allow time in your schedule for revising the survey instrument following the pre-test. Collecting Survey Data There are several different ways to conduct a survey. The most appropriate method will depend upon available resources, the content and complexity of the survey, and the types of populations being surveyed. Advantages and disadvantages of these different data collection methods are summarized in the chart below. Data Collection Method Advantages Disadvantages In-Person Interview: the preferred method of conducting PLWH needs assessment surveys. A trained interviewer sits down with the respondent and asks the survey questions at his/her home, at a service provider s facility, or in some other community location. Can be used for complex questions and questions unlikely to be misunderstood Surveys likely to be complete Can be used for low-literacy or l i m i t e d - E n g l i s h - p r o f i c i e n t P LW H Personal contact makes it possible to establish rapport Some information can be collected through observation Can obtain a high response rate with appropriate effort Some PLWH may be concerned about confidentiality and anonymity More costly and time-consuming than other methods Extra steps may be needed to ensure that interview locations are confidential and safe Ryan White CARE Act Needs Assessment Guide V-53

8 . Section V H.. o. w.-. To.. s.. 4. Collecting Primary Data Data Collection Method Advantages Disadvantages Telephone Interview: provides direct contact between the interviewer and respondent at slightly lower cost than in-person interviews. Usually, the respondent is contacted and interviewed in his/her home. Self-Administered Mail Questionnaire Mailed to the Homes of Respondents: respondents are asked to complete and mail back the questionnaire, usually with a stamped, self-addressed envelope provided. Can be used for complex questions and questions unlikely to be misunderstood Surveys likely to be complete Can be used for low-literacy PLWH Can be used for limited- English-proficient PLWH provided survey is translated and interviewers are bilingual Personal contact makes it possible to establish rapport Can obtain a high response rate with appropriate effort Mail-back protects anonymity Relatively low cost Some PLWH don t have telephones, especially those in severe need categories Phone numbers may be hard to find for PLWH not in care May create concerns about confidentiality and anonymity Relatively costly Response rate often low, and severe need groups may be especially unlikely to respond Surveys may be incomplete and questions may be misunderstood Not appropriate for low-literacy PLW H ; survey with very simple language can be used, but response rates are likely to be low Can be used for limited-english-proficient PLWH only if such individuals are identified ahead and sent surveys in the appropriate language May cause concerns about confidentiality since addresses are provided to those conducting the survey V-54 Ryan White CARE Act Needs Assessment Guide

9 Section V How-To s 4. Collecting Primary Data Data Collection Method Advantages Disadvantages Self-Administered Survey at Service Provider or Other Community Facility: respondents are asked to complete the survey and either leave it in a sealed envelope or mail it to a central location. Relatively low cost If surveys are mailed back to a central facility, protects anonymity Using community facilities and non-care Act funded providers can provide a means of includi n g PLWH who are not in care May create concerns about confidentiality Responses may not be frank and honest when surveyed at provider facility, especially if questions are asked about satisfaction with current services Surveys may be incomplete and questions may be misunderstood Response rate often low Not appropriate for low-literacy PLW H ; survey with very simple language can be used, but response rates are likely to be low Appropriate for limited Englishproficient PLWH only if surveys in the appropriate language are readily available To help ensure successful data collection: Be sure that the individuals involved in interviews or other data collection activities a re carefully selected, trained, and monitore d to ensure appropriate diversity and skills, including procedures for protecting the confidentiality of respondents. Make appropriate arrangements to obtain information from individuals who may be limited English-proficient. This usually means both a written translation of the survey and the use of bilingual interviewers. To ensure an accurate and understandable written translation of your survey, it is best that the translator be genuinely bilingual, bicultural, and biliterate able to write as well as read and speak both languages fluently and have worked in the HIV/AIDS field. An individual without training in translation tends to translate literally, focusing on words instead of ideas and concepts. If you have no access to a trained translator, look for a genuinely bilingual person with HIV/AIDS experience. Test your survey translation by asking a bilingual PLWH to complete the survey in both languages and tell you if any of the questions seem to have a different meaning in the two languages. Be sure that interviewers are bilingual and also familiar with the survey in both languages. Ryan White CARE Act Needs Assessment Guide V-55

10 . Section V H.. o. w.-. To.. s.. 4. Collecting Primary Data Be sure that the coding (i.e., grouping of results) of qualitative information from open-ended questions is done and overseen by individuals sensitive to cultural and other population differences, such as members of such populations and people with extensive experience in multicultural service settings. Including PLWH In and Out of Care Both surveys and focus groups of PLWH should include individuals currently receiving HIV/AIDS primary medical care and support services through CARE Act-funded or other community service providers (clients), as well as individuals not receiving such services ( not in care ). Programs that have sought out those not in care have found that such individuals tend to fit into two categories: individuals with relatively high educational and income levels who indicate that they are able to do their own case management, obtain medical care through a private physician, and pay for other needed services; and individuals who are more likely to be low-income, under- or uninsured (particularly people of color and women), and individuals with co-morbidities (e.g., homeless persons, injection drug users, and individuals with seve re mental illness); these are also populations among whom HIV and AIDS rates are increasing. The service needs of those in care and those not in care may be quite different. For example, one community found that the top priority unmet needs for clients were alternative therapy and housing, while the key needs for those not in care were primary medical care and case management. A Title IV program discovered that pregnant women and women with children who were receiving care but not family-centered services identified a variety of service barriers and needs for enabling services such as transportation and child care. Individuals not in care are often more difficult to reach than current clients, who can typically be located through primary care facilities, case managers, or other service providers. PLWH not in care need to be sought out at a variety of locations, using a mix of street, service provider, and media outreach techniques. Some of the same methods used to do outreach designed to get individuals into care can be used to survey such PLWH. For example, Title III and Title IV p rograms can do surveys as well as outreach at counseling and testing sites, and Title IV pro g r a m s can seek out pregnant women who come to a community or hospital clinic for pre-natal care. V-56 Ryan White CARE Act Needs Assessment Guide

11 Section V How-To s 4. Collecting Primary Data Some planning bodies, grantees, and providers have been very successful in locating PLWH not in care by working with a wide range of service providers that may not be funded through the CARE Act but are likely to be providing services to PLWH. They use resource inventories compiled through past needs assessments to identify service providers in their communities and then contact them for help in identifying HIV-positive individuals to interview. Often, provider i n ventories identify key target populations, so it is possible to contact providers that are part i c ularly likely to serve individuals with particular characteristics, such as racial/ethnic minorities, youth, women, pregnant women, the homeless, injection drug users, etc. Service providers may be asked to help in several ways, such as: Identify HIV-positive individuals among their clients and ask them whether they are willing to be interviewed, or provide them surveys to be self-administered and mailed to a central location. Post flyers that offer HIV- p o s i t i ve individuals a food voucher in return for survey or focus group participation, promise confidentiality, and ask them to call a telephone number established for the needs assessment. Those calling can be screened and interviewed in person or by telephone, sent a survey to be completed and mailed back, or asked to participate in a focus group. The following types of service providers may be particularly appropriate contacts for locating PLWH not in care: public and private clinics, such as health department clinics, neighborhood health centers, free clinics, maternal and child health programs, and migrant clinics; substance abuse programs; mental health programs; runaway shelters and other programs serving older youth such as gang intervention pro g r a m s ; homeless shelters; w o m e n s programs such as battered women s shelters and rape crisis centers, and pro g r a m s serving sex workers; STD clinics; Ryan White CARE Act Needs Assessment Guide V-57

12 . Section V H.. o. w.-. To.. s.. 4. Collecting Primary Data partner notification services; Planned Parenthood and other family planning programs; HIV counseling and testing centers; and ex-prisoner housing facilities and service providers. In doing street outreach to locate PLWH for surveys, coordination and cooperation betwe e n CARE Act and HIV prevention programs may be especially helpful. Prevention and early intervention program staff are especially likely to be aware of neighborhoods and community locations where HIV-positive individuals can be found, and surveys can include information useful to all titles and to both prevention and care. Any effort to identify PLWH not in care requires one of the following: Preliminary screening to identify PLWH with specific characteristics either for purposive survey sampling (e.g., interviewing specific numbers of individuals with particular characteristics) or for focus groups. This typically requires asking a small number of questions of identified PLWH to determine whether they are receiving regular primary care, case management, and/or supportive services, and if so, from what sources (e.g., are they receiving care through the CARE Act continuum of care), and to obtain basic demographic information as needed to determine whether they belong to specific targeted populations (e.g., based on race/ethnicity, co-morbidity). Questions on the needs assessment survey that identify providers by name so it is possible to determine whether the respondent is receiving CARE Act services, other regular HIV/AIDS care, or no care. Media can provide valuable publicity to help identify PLWH not in care, including public s e rvice announcements (PSAs) targeting PLWH and giving them a voic number to call, with PSAs in several languages and special telephone numbers for Spanish or other limited-english speakers as needed. Community newspapers, newsletters, and/or radio stations can help in reaching specific target populations. Once PLWH not in care have been located, their participation in surveys and focus groups can be encouraged by providing incentives. Care is needed to protect confidentiality, and data collection methods that protect anonymity are also necessary. V-58 Ryan White CARE Act Needs Assessment Guide

13 Section V How-To s 4. Collecting Primary Data D. Focus Groups... A focus group is a carefully planned discussion among a small group of people (typically 8-12) with certain similar characteristics who interact in a group setting facilitated by a trained moderator. The moderator uses a pre-determined outline to guide the discussion and assure that desired topics and questions are addressed. S/he should be knowledgeable but neutral and not directly involved with the service or activity being assessed. Focus group participants should consist of individuals from a clearly defined population who have certain common characteristics (e.g., women with HIV disease, Latino gay and bisexual men, or male PLWH with a history of substance abuse). The more your focus group can relate to the topic or problem being targeted, the more valuable the information gained. A technique developed for marketing research to test new products, packaging, advertising slogans and approaches focus groups provide results that are less precise than a large-scale surve y but are also far less expensive and complicated to conduct. Results often provide valuable insights, although they cannot be assumed to represent the views of a broader population. Focus groups are a commonly used source of qualitative information about the met and unmet needs of PLWH, client satisfaction with current services, and service barriers and access for particular population groups. Many grantees and planning bodies use a series of focus gro u p s annually to update needs assessments and obtain in-depth input from several specific PLWH populations. Focus groups can be used with underserved populations. Working with service p roviders or advocates that assist or re p resent a particular population, you can identify individuals, schedule a focus group in a convenient setting where they feel comfortable, and obtain valuable insights from individuals who may be hard to reach through other methods. Ryan White CARE Act Needs Assessment Guide V-59

14 . Section V H.. o. w.-. To.. s.. 4. Collecting Primary Data STEPS FOR ORGANIZING A FOCUS GROUP 1. Agree on parameters, including the purpose of the focus group, how many focus groups will be conducted, and what will be the screening characteristics for each focus group (e.g., race/ethnicity, gender, risk behavior, co-morbidity). 2. Develop a strategy for recruiting and screening individuals who are representative of the desired target populations for inclusion in a focus group; decide what incentives to provide. 3. Recruit a trained moderator who is familiar with focus groups, knowledgeable about HIV/AIDS, and appropriate for specific focus group participants, and arrange for at least one observer who will take notes on the discussion and reactions during the session; tape record the session if feasible. 4. Develop a script for the focus group topics and questions that you want answered by the focus group and information you will provide to the group. If you are planning to conduct multiple focus groups for different populations, include a common set of questions to be asked of all groups so you can compare the responses across groups, but include additional questions if information is needed about a specific focus group population. 5. If desired, develop a brief survey to be used individually with focus group members just before the focus group, to obtain demographic information and other data as desired; arrange to have interviewers or to make the survey self-administered. 6. Handle logistics: agree on a time and location for each focus group that is safe and convenient for its specific participants; arrange for transportation assistance if needed, and arrange for refreshments. 7. Recruit and screen focus group participants; be sure screening includes becoming aware of limited literacy among participants. 8. Conduct the focus group, and provide agreed-upon incentives. 9. Debrief and determine any needed changes before the next focus group. 10. Prepare a focus group report. V-60 Ryan White CARE Act Needs Assessment Guide

15 Section V How-To s 4. Collecting Primary Data A successful focus group usually has the following characteristics: The scope of the focus group is clearly defined and manageable it covers a limited number of topics so there is time for discussion involving all the participants. The moderator is experienced at facilitating group discussions and is independent of CARE Act services so that s/he is knowledgeable but does not bias the discussion with his/her own views. If a series of focus groups involves diverse populations, it is helpful for moderators to reflect key participant characteristics. For example, a Latino focus group might be facilitated by a bilingual Latino. While a m a t c h between moderator and group is not essential, it can help to establish a safe setting in which participants feel comfortable sharing their views and experiences. A carefully developed s c r i p t or discussion outline provides a blueprint for the focus group, clearly stating the planned flow of the focus group and the major questions or issues to be covered. The outline should be used flexibly but all the key points covered during the focus group. Planning body staff or members not moderating may observe, but do not part i c i p a t e in the discussion except at the initiative of the moderator. The number of observers is kept to a minimum. If the moderator is not an HIV expert, it is very helpful to have someone knowledgeable about HIV/AIDS services and the CARE Act observing. Several breaks may be built into the session so that the moderator and observers can confer and observers can clarify issues as needed. The focus group environment is we l l - s e l e c t e d the site is accessible, comfortable, quiet, and allows for good-quality audio taping (more sophisticated methods may also be used such as one-way mirrors, but usually are not necessary). Audio-visual equipment works properly. If audio-visual equipment is required or the session is being tape-recorded, all equipment and procedures should be tested before the session to be sure they work properly. The group composition is appropriate: the group is representative of the desired population; all members have been appropriately screened to be sure they possess the desired characteristics; and the group is as homogeneous as possible in terms of key variables (e.g., age, language preference, risk group). Where several different populations have been identified as targets for the focus groups, separate focus groups are used to determine the met and unmet needs of each major group. Ryan White CARE Act Needs Assessment Guide V-61

16 . Section V H.. o. w.-. To.. s.. 4. Collecting Primary Data The group is of manageable size large enough to provide for a variety of views and adequate interaction, but not so large that some members do not participate in the discussion. Usually focus groups should include no fewer than eight or more than 12 people. This means recruiting at least people, to allow for no-shows. The session is carefully scheduled and managed. Usual stages include an introduction to clarify the purpose of the session; a warm-up to introduce group members (usually first names only) and set up a positive group atmosphere; general topic presentation and discussion to obtain broad opinions about HIV/AIDS service needs and gaps from the p a rticipants; presentation of specific questions and specific discussion, with both individual opinions and group discussion; a closing that assures that all needed aspects of the topic have been discussed; and a thank you. If brief individual surveys are included, time is set aside to administer them and the moderator and other staff offer to interview people who would rather not complete the form themselves. This makes it possible for individuals with limited literacy to provide survey data without any embarrassment. The session is of an appropriate length, depending upon the topic and its complexity. Participants are likely to get tired after one and a half to two hours, though the time period can be extended slightly if refreshments are served and a break is provided. Focus group participants receive some incentive or compensation for their participation. Re f reshments or a light meal may be served, and participants are often paid a small a m o u n tin the form of cash or a food voucher ($15 to $20); an extra $5 might be added for an evening session. Transportation and child care are provided as needed. Discussion and debriefing among the moderator, note taker, and any observers follow each focus group, to refine the outline and procedures. Su m m a ry or compre h e n s i ve focus group re p o rts are pre p a re d, in a format appro p r i a t e for the overall needs assessment. Comparisons are made among focus groups composed of different populations. The reports are prepared as soon as possible after focus groups are completed and are revie wed by the moderator and observers. E. Key Informant Interviews. Key informant (or key person) interviews can be used to obtaining qualitative information from knowledgeable individuals about such matters as perceived HIV/AIDS service needs, gaps, and barriers for particular populations or in geographic areas. They may represent or be extremely k n owledgeable about specific types of services or particular PLWH populations. Sometimes, when it is difficult to directly interview individuals from particular populations, such as children with AIDS or PLWH with mental illness, key informants can provide useful information about these V-62 Ryan White CARE Act Needs Assessment Guide

17 Section V How-To s 4. Collecting Primary Data groups. A small number of knowledgeable individuals are selected for in-depth interviews using an interview guide or checklist that specifies major topics and issues to be covered. Key person interviews are useful only when the individuals interviewed are in fact both knowledgeable and willing to provide detailed information. Although the interviews do not provide information that can be generalized to a larger population, they should provide a wide range of perspectives on the services or the issues being studied. Therefore, the individuals to be i n t e rv i ewed should be carefully selected to assure diverse points of view, experiences, and concerns. This may mean people knowledgeable about va rying populations and different services within the continuum of care, staff of providers as well as PLWH group leaders, individuals from different counties or cities within the service area, individuals who are themselves members of different populations, etc. The appropriate categories will depend on the focus of the needs assessment and the populations of particular interest. Because the interview is typically a guided discussion without closed-ended questions, it requires an interviewer who is knowledgeable about HIV/AIDS services and trained to probe for information without presenting his/her views in a way which might bias the responses. The i n t e rv i ewer or an observer should take detailed notes of the responses to the questions and should also consider taping the session for back-up if feasible. After the session, the notes should be compiled into an interview report. Like other qualitative data, the information about a service or population that is generated from key person interviews will need to be supplemented with other information gained from surveys, focus groups, or other methods. If key person interviews are conducted early in the needs assessment process, the results of the interviews can help the planning body or grantee identify and select issues to be covered in focus groups, assessments, and surveys, and frame the questions appropriately. F. Community Forums Another small-group method of collecting information is the community forum or community interview. A community forum is less formal than a focus group; a meeting is advertised but t h e re is limited pre - s c reening. It often includes a larger group of participants than a focus gro u p, but it is much more interactive than a formal event such as a public hearing. Where a focus group involves the actual target group, the community interview often involves a mixed group of individuals all ages, genders, sexual orientations, and racial/ethnic backgrounds. It can also focus on a particular gro u p. For instance, a community forum may invo l ve clients of a part i c u l a r HIV/AIDS service program, residents of a particular housing project, or members of a part i c u l a r PLWH or gay men of color support group. It often includes both PLWH and other interested individuals; often, participants HIV status is not known. Ryan White CARE Act Needs Assessment Guide V-63

18 . Section V H.. o. w.-. To.. s.. 4. Collecting Primary Data Id e a l l y, a community forum invo l ves between 15 and 30 individuals. Open discussion is ve ry difficult to manage if the group is larger than 30. Formal procedures are needed for recognizing individuals so they can speak. If the group is large, time limits for individual comments may be set, and discussion may be time-limited as well. Similar to the focus gro u p, a community forum should be a we l l - s t ru c t u red discussion session. At least two facilitators should facilitate the discussion using a carefully developed interv i ew guide. The session should be planned and conducted with care, and its proceedings should be both taped and summarized in writing. A community forum s results should be viewed as providing valuable insight but not information that can be projected to a broader population. Community forums that are open meetings or occur in public places have important limitations. PLWH may not be willing to self-identify or provide frank opinions. CARE Act experience suggests that the service needs of people with HIV disease, especially women, minorities, and other severe need populations, are usually best obtained through methods that protect the confidentiality of information and the anonymity of respondents. Some grantees and planning councils are addressing this concern by conducting community forums within specific community-based organizations or community facilities and limiting participants to clients or members of these organizations. For example, a community forum might be conducted within a substance abuse treatment center, a homeless shelter, or a women s clinic. A Title III grantee might conduct a forum at a clinic site. Individuals are not asked to disclose their status, but are asked to discuss their perceptions of PLWH service needs, gaps, and barriers. Sometimes, participants are given a number to call if they would like to participate in a PLWH survey so that PLWH can choose to provide more detailed, personal information in a confidential or anonymous setting. Community forums can be used for assessing needs and priorities of PLWH and other community members, identifying service barriers, and obtaining feedback concerning the adequacy of the current continuum of care or of specific service components. By arranging community forums in specific locations, doing targeted outreach, and scheduling one or more forums in a language other than English, a planning body or grantee can obtain useful input from hard-toreach populations such as the homeless, substance users, and limited-english-proficient PLWH. V-64 Ryan White CARE Act Needs Assessment Guide

19 Section V How-To s 4. Collecting Primary Data G. CHOOSING APPROPRIATE DATA COLLECTION METHODS Method When to Use Benefits Limitations Survey To obtain quantitative information about the met and unmet care needs and barriers to care for PLWH both in care and not in care Provides quantitative data Can provide data of high reliability and validity Can provide consistent information from a wide range of individuals, and results can be compared across groups Large samples usually needed Often costly Difficult to design and implement without help from trained researchers If appropriate sampling is used, results can be generalized to the entire PLWH population or specific populations Focus Group To obtain in-depth, qualitative information including perceptions regarding met and unmet service needs, barriers, and access to care for specific PLWH populations Can provide in-depth understanding of issues related to service needs and access for specific PLWH populations Information from different focus groups can be compared to clarify differences in needs and access by population Relatively low-cost and quick to implement Requires careful plann i n g,outreach and screening, and effective f a c i l i t a t i o n Findings cannot be generalized Preparing reports is time-consuming and requires strong analytical skills Can identify factors and issues to consider in developing a survey questionnaire Works best for relatively homogeneous groups; means many focus groups may be required Key Informant Interview To obtain qualitative information about perceived met and unmet needs, barriers, and access for particular populations from individuals with special knowledge and experience regarding those populations A way of obtaining information about populations that may not easily be contacted directly (e.g., infants and children with HIV/AIDS, PLWH with severe mental health problems) Can identify factors and issues to consider in developing a survey questionnaire Low-cost A way of checking information from other sources with knowledgeable individuals Indirect source of data not obtained directly from PLWH Findings cannot be generalized Data quality depends on choice of key persons Ryan White CARE Act Needs Assessment Guide V-65

20 . Section V H.. o. w.-. To.. s.. 4. Collecting Primary Data CHOOSING APPROPRIATE DATA COLLECTION METHODS Method When to Use Benefits Limitations Community Forum To obtain qualitative, often preliminary information about perceived met and unmet service needs and barriers and access to care in specific geographic locations or for specific PLWH populations Can provide perceptions of service needs among specific PLWH populations Low-cost Can help in identifying PLWH to be surveyed Problems of confidentiality and anonymity mean PLWH may be afraid to self-disclose or provide frank and honest information Quality of information extremely varied Diverse or large group can be difficult to facilitate Findings cannot be generalized H. Analyzing Primary Data.. Once you have collected information from all sources, the data need to be ord e red and analyze d. The methods of analysis used will depend partially on the questions to be answered and on the amount and types of data available. Two basic types of analysis statistical and qualitative will be necessary, and are discussed below. Quantitative Analysis Quantitative data, primarily from surveys with PLWH and providers that use multiple-choice and other short-response or closed-ended questions, need to be aggregated and analyzed using statistical methods. Often, this work is done by the health department or by a consultant. At a minimum, statistical analysis requires: Reviewing the completed instruments to be sure they are complete. Coding open-ended or other any responses that were not pre-coded so they can be tabulated. Entering the data into a computer database or tabulating data by hand. Obtaining f requency distributions (the simplest form of statistical tabulations) that pre s e n t the number and percent of respondents giving each possible response to a question, both overall and by population categories (e. g., different target populations, consumers ve r s u s p roviders); be sure you always specify the number of respondents in each population category. V-66 Ryan White CARE Act Needs Assessment Guide

21 Section V How-To s 4. Collecting Primary Data Deciding whether to report numbers or percentages in your analyses. Where your sample for a particular population group is small and is a small proportion of the population being represented a rule of thumb is where the number of respondents is less than 20 or the number of responses in an individual cell (response category) is less than five report numbers rather than percentages. This avoids misleading those who use the data, who may assume that the percentages represent a large number of respondents. STATISTICAL SIGNIFICANCE Significance means that a research finding is statistically meaningful. A research finding is considered to be statistically significant if there is only a small probability that the observed result could have occurred by chance alone, rather than measuring something that is real such as a real difference between the behaviors of two populations, the services being received by PLWH living in two different counties, or the effects of an HIV/AIDS protease inhibitor on two different populations tested. The probability value or p value is a measure of how often a result could occur due to chance alone, rather than a real difference or relationship. Typically, statistical results are considered to be significant if there is less than a 5 percent probability 5 times out of 100 that the observed difference or relationship was due to chance alone. In such situations, the p value is said to be less than.05 (<.05). A more stringent p value is <.01, which means that there is less than a one percent chance one time out of 100 that the observed difference or relationship occurred by chance alone. The p value is calculated to determine whether results are statistically significant, and should be presented in statistical reports along with the findings, so that anyone using the data will know how much confidence to place in the reported relationships and analysis results. It is important to differentiate statistical significance and practical importance. A large random sample can help you obtain needs assessment findings that are statistically significant. However, a research finding such as a difference between two target populations may have no practical importance unless the effect is both statistically significant and of relatively large magnitude. Some researchers suggest that you first decide whether differences have practical importance, and if they do, then carry out the appropriate statistical tests to determine whether the findings are statistically significant. The researchers who carry out your needs assessment should be experienced in determining which statistical tests of significance are appropriate, depending upon whether your findings are based on large numbers, small numbers, or small total populations. Be aware of the need for these tests and ask whether they have been conducted. Ryan White CARE Act Needs Assessment Guide V-67