Data Management and Use of Excel. Joe Hagan School of Public Health

Transcription

1 Data Management and Use of Excel Joe Hagan School of Public Health

2 Data Management Guidelines Research projects often involve the collection of a large volume of data The data have then to be processed and analyzed results and summaries will be published or presented in some form Requires a well-defined system of data management

3 Stages of Data Management in a Research Project The raw data have to be entered into the computer, and checked for accuracy The data have then to be organized into an appropriate form for analysis often in different ways, depending on the analysis The data have to be archived remain available throughout subsequent phases of a project, and afterwards

4 Software for handling data Database (DBMS) packages Access, EpiInfo Statistics packages SAS, SPSS, Stata Spreadsheet packages Excel, Lotus-123 Word Processors Word, WordPerfect or even text editors like Notepad

5 Free Software EpiInfo (available for free download): Allows users to: rapidly develop a questionnaire or form customize the data entry process check data (including double entry) analyze data

6 Layout of Data These various types of software all handle "rectangles" of data Each row refers to a unique observation or case (e.g. a patient/subject) Each column refers to variable (e.g. gender, type of insurance, procedure)

7 Simple Example of Data in Excel

8 Spreadsheets (e.g. Excel) Simplest to use Often automatically chosen Familiar Widespread Flexible Flexibility can result in poor data entry and management

9 Other Software More consideration should be given to alternative software for data entry Access forms can be developed that facilitate easy and standardized data entry talk to statistician prior to data collection SPSS has special modules for data entry Access, SAS and SPSS have tools for data checking Access relational databases easily created

10 Database Structure Flat: all the data exist at a single level and can be held in one database table previous simple example Relational Databases: uses multiple, linked tables to hold all the data one table contains the key variable that is used to link information to other tables e.g. Patient ID number serves to link demographics to clinical data

11 Table with a Key Variable

12 Key Variable is Link

13 Queries pull data together

14 Relational Databases Save memory space and data entry time by reducing the amount of redundant information Queries used to pull together information from multiple tables linked by key variable(s) Easily created in Access as widely available as Excel

15 Using Excel Experience has shown that most researchers use Excel for data entry When using Excel, there are some strategies that should be used to: improve data quality make easier to analyze facilitate accurate data entry

16 Excel Bad Example

17 Problems in Excel Example Two of the names under the "species" heading have been typed slightly differently for the same species The variable "rcd has observations that will cause problems when the data are transferred to a statistics package for analysis row 2 has two measurements entered in one cell in row 10, the cell reports that the plant is dead instead of having a numerical value

18 Avoiding Problems Many errors can be avoided by thinking about the layout of the data in the spreadsheet before starting data collection Good to consider the analyis when organizing the data in the spreadsheet

19 Worse Example

20 Good Example

21 Data Entry Recommendations Unique identifier Freezing panes Drop-down lists Data validation Adding comments to cells Formatting cells Forms

22 Unique Identifier One of the variables entered should give a unique record number serves as an ID (only appears once) the good example above has an extra column, named plot (has been calculated as: plot=100*block+plotwb)

23 Freezing Panes in Excel When entering data, it is useful to be able to keep the headings of columns always visible as you scroll down the screen 1. Highlight one row below the row you want to freeze often a column heading 2. Click Window > Freeze Panes undo by clicking Window -> Unfreeze Panes

24 Drop-Down Lists When the same text string is entered many times, typing/spelling errors inevitably occur Drop-down lists can be used to avoid such errors Ensures standardized and consistent data entry

25 Creating Drop-Down Lists in Excel First, type all choices to be included in the drop-down list in a single column e.g., for the previous example, the five species names for block 1 are entered into cells D2:D6 The list of choices must be maintained, so the list should not be in the actual column where data in entered You can hide the list later

26 Creating Drop-Down After creating the list field: 1. Select the cells to have drop-down lists applied Can apply drop-down list to entire column an later unapply to specific part 2. Click Data > Validation > Allow: > List 3. For the Source of the list, highlight the choices already typed in the list, then click OK

27 Drop-Down Lists (continued)

28 Drop-Down Lists (continued) Once the drop-down list has been created, selecting a cell in that column will bring up an indicator triangle on the right side of the cell Clicking on this will display the drop-down list so that an appropriate selection can be made from the list

29 Drop-Down Lists (continued)

30 Hiding List Fields for Drop-Down Lists To hide the lists so only the actual data fields are displayed: Format > Row > Hide To unhide lists: Format > Row > Unhide Save both versions can t unhide lists after making changes to the spreadsheet

31 Hiding List Fields (continued)

32 Unhiding List Fields

33 Data Validation Validation checks can be set on ranges of cells within the spreadsheet could be an entire column/row, several columns/rows, or just a single cell The validation rules apply when new data are entered

34 Data Validation (continued) Range checks for numerical data can be set up in Excel For the previous example, suppose the measurements recorded for the variable rcd are expected to fall between 10 and 26

35 Setting Up a Range Check Highlight the cells to which the range check is to be applied cells E2 to E21 in the previous example only the data cells are highlighted, not the variable name at the top If want to apply to entire column you can remove validation rules from the column heading

36 Example of Setting Up a Range Check 1. Click Data > Validation 2. Select the Settings tab (if not already chosen) 3. for Allow: choose Decimal (or whatever appropriate) 4. For Data choose Between (or whatever appropriate) 5. Set the Minimum and Maximum Minimum = 10 and Maximum = 26 in the previous example 6. Click OK

37 Setting Up a Range Check (continued)

38 Setting Up a Range Check (continued) While the Data Validation dialog box is being used to set up the range check, you can also set up: 1. An Input Message 2. An Error Alert

39 Setting Up a Range Check with an Input Messages: Input Message Are displayed when a cell to which the message has been applied is selected E.g. Remind the data-entry person of the range of values allowed (or expected) Any type of message could be used

40 Setting Up a Range Check with an Input Message (continued) To create an Input Message: 1. Click Data > Validation 2. Select the Input Message tab 3. Type in the desired Title and Input Message 4. Click OK

41 Setting Up a Range Check with an Input Message

42 Setting Up a Range Check with an Input Message (continued)

43 Setting Up a Range Check with an Error Alert Message Error Alert Messages are displayed when a value outside the range is typed To create an Error Alert Message: 1. Click Data > Validation 2. Select the Error Alert tab 3. Select a Style from the drop-down list 4. Type in the desired Title and Error Message 5. Click OK

44 Setting Up a Range Check with an Error Alert Message (continued)

45 Setting Up a Range Check with an Error Alert Message (continued)

46 Adding Comments to Cells Excel has a facility for adding comments to a cell The comments differ from values within the cell Useful for any unusual observations or questions concerning a particular data value

47 Adding Comments to Cells (continued) Recall the example with the data for plot 101 for rcd where two values (12.7, 13.3) were entered on the data recording sheet Suppose the researchers decided to calculate the mean of the two values and add a comment to the cell If several plots had two values recorded, two columns of rcd data could have been entered with a third column used to calculate the mean

48 Adding Comments to Cells To add a comment to a cell: 1. Highlight the cell to which the comment is to be added 2. Click Insert > Comment 3. Type in the desired comment

49 Adding Comments to Cells (continued)

50 Adding Comments to Cells (continued)

51 Adding Comments to Cells (continued) After the comment is added: The cell will now show a red tab in the upper right corner The comment will be displayed when the cell is selected

52 Removing Comments from Cells To remove a comment from a cell: e.g., when a query has been resolved and the correct value has been entered 1. Highlight the cell 2. Right-click Delete Comment

53 Formatting Cells Excel offers many formats that can be applied to cells Only one example of one kind of number format was be shown Note that many more formats are available

54 Formatting Cells (continued) Excel suppresses trailing zeros by default e.g is displayed as 13 We can change this so that 1 (or more if desired) decimal places are displayed

55 Formatting Cells (continued) 1. Highlight the cells to have decimals displayed 2. Click Format > Cells 3. For Category select Number 4. For Decimal places choose 1 5. Click OK

56 Formatting Columns (continued)

57 Formatting Cells (continued) Excel offers many other options for formatting cells Only one example of one kind of number format has been shown Explore the other options by clicking Format > Cells and looking at all of the tabs i.e. Alignment, Font, Border,Patterns and Protection

58 Forms Data entry forms facilitate data entry Easier to enter data in a list Excel has a built-in Data Form 32 fields maximum

59 Forms for Larger Datasets If a data entry form is desired but you have more than 32 fields: 1.Use Access 2.Use multiple Excel spreadsheets Can merge together later Be sure to include unique identifier in each spreadsheet so can link spreadsheets together 3.Download free enhanced data form:

60 Forms in Excel 1. Select the fields (including column headings) for which you want to use a data entry form 2. Data > Form 3. Click New to add data to the next observation

61 Forms in Excel (continued)

62 Forms in Excel (continued) Note: 1. No entry field for calculated columns E.g. Paid by Insurance = Cost Out of Pocket 2. Ctrl + ; is a shortcut key to enter the current Date 3. Drop-down lists created do not appear in the form, but you will not be allowed to enter something not in the drop-down list

63 Forms in Excel (continued) Much more sophisticated UserForms (e.g. with drop-down lists) can be created using the Visual Basic Editor Similar to Access 02.html

64 Data Auditing To check data that: 1. Has already been entered 2. Has had validation rules (discussed above) applied or changed after data entry

65 Data Auditing (continued) To audit data that has been entered and then had validation rules applied: 1. Click Tools > Formula Auditing > Show Formula Auditing Toolbar 2. On the Formula Auditing toolbar, click the Circle Invalid Data icon

66 Data Auditing (continued)

67 Data Auditing (continued) To remove the red circles from the invalid data click the Clear Validation Circles icon on the Formula Auditing toolbar

68 Metadata Essential if the dataset is to be integrated with datasets from other studies, or is to be passed to someone else for analysis Where the data came from When the data was collected What the data represents Units of measurements used

69 Metadata (continued) Adding rows and columns to the spreadsheet before the body of data can be helpful The extra rows will store documentation that provides background information about the data i.e., the metadata

70 Input Areas of an Excel Spreadsheet

71 Data Collection Form

72 Using Multiple Sheets An alternative to what is described above is to put the Page Information on a separate sheet in the Excel Workbook Convenient when there is a lot of information at the dataset level May still have a small "Page" section in each data sheet describing the type of measurements entered in that sheet

73 Excel s Limitations No easy facilities for skipping fields conditional on the entry of initial codes Limited graphical capabilities Excel graphics intended for presentation No boxplots Lacking other exploratory techniques that could assist in data scrutiny Can t handle too many columns in one sheet

74 Data Entry and Checking The ultimate aim should be a fullydocumented archive of checked, correct, reliable data that can be subjected to scientific scrutiny without raising any doubts in the minds of subsequent researchers

75 Make Data Entry as Simple as Possible In a replicated experiment it should never be necessary to type variety names or long treatment codes for each patient a single letter or number is usually sufficient then, either the data entry system can insert the full code, or the full names may be available in a separate, "look-up" file Simplifying the keying process will speed the task, make it less tedious and hence also less error-prone

76 Enter Data ASAP The data should be entered as soon as possible after data collection not so large and daunting as doing all at the end helps checking, some checks can indicate unusually large changes from the previous value to allow immediate verification feedback of any problems that are noticed to field data collectors can help maintain the data quality

77 Double Entry The ideal way to ensure accurate data entry Two different people enter all of the data separately two different databases Software is used to identify discrepancies between the two data sets inconsistencies resolved to create the final database

78 Only One Master Copy" (if Double Entry not used) Problems can arise if multiple copies are kept of the same data in different formats Master copy will increase in size as data accrues changes through the course of data entry Process should be documented Consistent "version-numbering" system should used by all people making modifications to the data

79 Backing up Data Essential to develop a system for regular "back-ups" (copies) of the data not backing up may result in losing data Back up copies of data should be made on separate media from the original master copy e.g., another computer, on CDs, on a network, etc. The back up copy should be dated date of last revision

80 Data Checking Checking is done both at the time of keying and afterwards The logical checking phase should be done by trained staff who understand the nature of the data

81 Logical Checking Checks to rule out illogical data e.g. pregnant males, or minimum greater than maximum temperature, clinic visits recorded as dates in the future, range checks, etc. Usually involves preliminary analyses, plotting, etc. Reasoned decisions can be made about what to do with unusual observations

82 Audit trail Complete record of changes to the data and decisions made about the data and the analysis like a log book Requirement of the scientific method must ensure the data management work is repeatable Facilitates subsequent writing of reports on the data and answering data queries

83 Audit trail (continued) Important to record everything you do at the time that you do it recollections are always poor at a later stage When errors are found and changes are made to the master copy of the data, a note should be made old and new values recorded

84 Audit trail (continued) Keep notes on the analyses done including the preliminary analyses done for checking purposes Writing down the names of all files created Including back ups Every entry in the log-book should be dated and initialed

85 Archiving Data All data and programs from a research project must be archived in such a way that they are safe and can be accessed by a subsequent user Use a consistent directory structure and naming convention for computer files

86 Archiving Data (continued) The archive should give access to all the information about the study during the project, information is located in many places (e.g. the computer, on paper and other media and in the minds of the research team) The archive need not all be computerized, but it should include all the relevant information The source/location of information not archived electronically should be recorded in the electronic archive

87 Archiving Data (continued) If a proper archiving scheme is not used, when researchers leave: They might take the only copy of their part of the data (the data is lost) Knowledge of the study protocol is lost resulting in great difficulty when new investigators join the project

88 Confidential Data Good idea to password protect confidential data files warn analyst that file is protected Patient names, SSN s etc. should always be removed

89 Prevent Modification of Data To prevent others from modifying the data but they can save changes under a different file name Tools > Options > Security > Password to Open > [enter password]

90 Prevent Viewing of Data To prevent others from viewing the data: Tools > Options > Security > Password to Open > [enter password]

91 References University of Reading: Statistical Services Center: Microsoft Office Applications: ml