Empirical Project 8 Working in Excel

Part 8.1 Cleaning and summarizing the data

Learning objectives for this part

  • detect and correct entries in a dataset
  • recode variables to make them easier to analyse
  • calculate percentiles for subsets of the data.

So far we have been working with data that was formatted correctly. However, sometimes the datasets will be messier and we will need to ‘clean’ the data before starting any analysis. Data cleaning involves checking that all variables are formatted correctly, all observations are entered correctly (e.g. no typos), and missing values are coded in a way that the software you are using can understand. Otherwise, the software will either not be able to analyse your data, or only analyse the observations it recognizes, which could lead to incorrect results and conclusions.

In the data we will use, the European Values Study (EVS) data has been converted to Excel from another program, so there are some entries that were not converted correctly and some variables that need to be recoded (for example, replacing words with numbers, or replacing one number with another).

Download the EVS data and documentation:

  1. The data spreadsheet contains an incomplete Data dictionary and four tabs with the data collected from each wave of the survey. The variable names currently do not tell us what the variable is, so we need to relabel them to avoid confusion.

Throughout this project we will refer to the variables using their original names, so the Data dictionary tab will come in handy. In general, data dictionaries and variable correspondences are useful because they contain important information about what each variable represents and how it is measured, which usually cannot be summarized in a short variable name.

  1. Now we will take a closer look at how some of the variables were measured.

Before doing any data analysis, it is important to check that all variables are coded in a way that the software can recognize. This process involves checking how missing values are coded (usually these need to be coded in a particular way for each software), and that numerical variables (numbers) only contain numbers and not text (in order to calculate summary statistics).

  1. We will now check and clean the dataset so it is ready to use. Make the following changes to all relevant data tabs:

Excel walk-through 8.1 Cleaning data and splitting variables

Follow the walk-through in the Core Economics website video, or in Figure 8.1, to find out how to clean data and split variables in Excel.

How to clean data using find and replace, and split variables using Excel’s LEFT and RIGHT functions.

Figure 8.1 How to clean data using find and replace, and split variables using Excel’s LEFT and RIGHT functions.

The data

This is what the data looks like. Currently some cells have entries ‘.a’ where the data is missing. We will find and replace the variables A009 to X047D (not pictured here).

Figure 8.1a This is what the data looks like. Currently some cells have entries ‘.a’ where the data is missing. We will find and replace the variables A009 to X047D (not pictured here).

Find and replace specific cell entries

‘Find and Replace’ in Excel is similar to this feature in Microsoft Word, except that it searches for text in individual cells rather than in lines of text. You can adapt steps 1–3 to change specific values of cells in a column, for example replacing a particular word with a particular number.

Figure 8.1b ‘Find and Replace’ in Excel is similar to this feature in Microsoft Word, except that it searches for text in individual cells rather than in lines of text. You can adapt steps 1–3 to change specific values of cells in a column, for example replacing a particular word with a particular number.

Extract a specific part of a string of text

Excel’s LEFT and RIGHT functions can help you extract parts of a long string of text, which is useful if you want to shorten variables or only keep particular information in a cell. Here, we are going to save variable X025A as two variables: ‘Education_1’ will contain the numbers before the colon, and ‘Education_2’ will contain the words after the colon.

Figure 8.1c Excel’s LEFT and RIGHT functions can help you extract parts of a long string of text, which is useful if you want to shorten variables or only keep particular information in a cell. Here, we are going to save variable X025A as two variables: ‘Education_1’ will contain the numbers before the colon, and ‘Education_2’ will contain the words after the colon.

Extract a specific part of a string of text

The RIGHT function counts the specified number of characters starting from the right end of the string of text. Since words have different lengths, we use LEN() to calculate the length of the string, then subtract the colon’s position in the string (using the FIND function). For example, if the string is 10 characters long, and the colon is the third character, we want Excel to extract characters 4 to 10 (the first 6 characters from the right).

Figure 8.1d The RIGHT function counts the specified number of characters starting from the right end of the string of text. Since words have different lengths, we use LEN() to calculate the length of the string, then subtract the colon’s position in the string (using the FIND function). For example, if the string is 10 characters long, and the colon is the third character, we want Excel to extract characters 4 to 10 (the first 6 characters from the right).

Although the paper we are following only considered individuals aged 25–80 who were not students, we will retain the other observations in our analysis. However, we need to remove any observations that have missing values for A170, X028, or any of the other variables except X047D. Removing missing data ensures that any summary statistics or analysis we do is done on the same set of data (without having to always account for the fact that some values are missing), and is fine as long as the data are missing at random (i.e. there is no particular reason why certain observations are missing and not others).

  1. In your spreadsheet, remove all rows in all waves that have missing data for A170. Do the same for:

    • X003, X028, X007 and X001 in all waves
    • A009 in Waves 1, 2, and 4 only
    • C036, C037, C038, C039, C041 and X047D in Waves 3 and 4 only
    • X011_01 and X025A, in Wave 4.

    Excel walk-through 8.2 gives guidance on how to do this.
    (Note: Excel may take some time to process the commands.)

Excel walk-through 8.2 Dropping observations that satisfy particular conditions

How to drop observations that satisfy particular conditions using ‘filter’, ‘select’ and ‘delete’.

Figure 8.2 How to drop observations that satisfy particular conditions using ‘filter’, ‘select’ and ‘delete’.

The data.

This is what the data looks like. In this example, we will remove any rows with the variable A170 recorded as missing (blank). First, we will filter the data so that only rows with blank cells in this column are showing.

Figure 8.2a This is what the data looks like. In this example, we will remove any rows with the variable A170 recorded as missing (blank). First, we will filter the data so that only rows with blank cells in this column are showing.

Filter the rows of interest

After step 3, you will see all the data that needs to be deleted.

Figure 8.2b After step 3, you will see all the data that needs to be deleted.

Delete these rows from the dataset

Excel will remove all the selected rows from the dataset. If there are many rows to delete, this command may take some time to process.

Figure 8.2c Excel will remove all the selected rows from the dataset. If there are many rows to delete, this command may take some time to process.

Clear the filter to see the rest of the data

After removing the specified rows, clear the filter to see the rest of your data. You will see that the number of rows has decreased.

Figure 8.2d After removing the specified rows, clear the filter to see the rest of your data. You will see that the number of rows has decreased.

The modified data

This is what the data looks like after removing rows with missing entries in A170. You can adapt steps 1–7 to remove rows that satisfy other conditions, such as containing a particular value or range of values.

Figure 8.2e This is what the data looks like after removing rows with missing entries in A170. You can adapt steps 1–7 to remove rows that satisfy other conditions, such as containing a particular value or range of values.

  1. We will now create two variables, work ethic and relative income, which we will use in our comparison of life satisfaction.

Excel walk-through 8.3 Calculating percentiles from actual values

How to calculate percentiles from actual values.

Figure 8.3 How to calculate percentiles from actual values.

The data

This is what the data looks like. In this example, we will calculate the percentile of X047D (log income), according to the country (S003). You will need to use two new columns (R and S in this case).

Figure 8.3a This is what the data looks like. In this example, we will calculate the percentile of X047D (log income), according to the country (S003). You will need to use two new columns (R and S in this case).

Calculate the rank of each observation

The intuition behind our rank calculation is that an observation’s rank is the number of observations larger than it, plus 1. (For example, if there were only two numbers, the smaller number should be ranked 2 and the larger number ranked 1). By default, Excel will rank observations from largest to smallest (higher numbers will have lower ranks). We need this setting to calculate percentiles. In this example, we need to rank observations only within the country it belongs to, not the entire dataset, so we use this as a condition in COUNTIFS.

Figure 8.3b The intuition behind our rank calculation is that an observation’s rank is the number of observations larger than it, plus 1. (For example, if there were only two numbers, the smaller number should be ranked 2 and the larger number ranked 1). By default, Excel will rank observations from largest to smallest (higher numbers will have lower ranks). We need this setting to calculate percentiles. In this example, we need to rank observations only within the country it belongs to, not the entire dataset, so we use this as a condition in COUNTIFS.

Use the calculated rank to calculate the percentile.

The intuition behind the percentile calculation is that an observation’s percentile is the percentage of observations smaller than it (the +1 in the formula is needed to exclude the observation itself). The 1– at the start of the formula is because we ranked observations from largest to smallest (so we need to ‘reverse’ all the numbers). Here we have multiplied by 100 so the numbers correspond to actual percentiles (e.g. 92 percentile). You can see that higher incomes have lower ranks and higher percentiles, indicating that we have done this correctly. Note: This example was based on observations with non-missing income only (not the fully-cleaned dataset), so the numbers you get will be slightly different.

Figure 8.3c The intuition behind the percentile calculation is that an observation’s percentile is the percentage of observations smaller than it (the +1 in the formula is needed to exclude the observation itself). The 1– at the start of the formula is because we ranked observations from largest to smallest (so we need to ‘reverse’ all the numbers). Here we have multiplied by 100 so the numbers correspond to actual percentiles (e.g. 92 percentile). You can see that higher incomes have lower ranks and higher percentiles, indicating that we have done this correctly. Note: This example was based on observations with non-missing income only (not the fully-cleaned dataset), so the numbers you get will be slightly different.

  1. Now we have all the variables we need in the format we need. We will make some tables to summarize the data. Using the data for Wave 4:

Hint: For help on creating this table, see Excel walk-through 3.1. For help on changing the values in the table to percentages of the row total, follow steps 18–19 in Excel walk-through 3.2, but instead of selecting ‘Average’ (as in step 19), select ‘Show Values As’ (or ‘Show data as’) then ‘% of Row Total’.

Male Female
Mean Standard deviation Mean Standard deviation
Life satisfaction
Self-reported health
Work ethic
Age
Education
Number of children

Summary statistics by gender, European Values Study.

Figure 8.4 Summary statistics by gender, European Values Study.

Part 8.2 Visualizing the data

Note

You will need to have done Questions 1–5 in Part 8.1 before doing this part.

Learning objectives for this part

  • use column charts, line charts, and scatterplots to visualize data
  • calculate and interpret correlation coefficients.

We will now create some summary charts of the self-reported measures (work ethic and life satisfaction), starting with column charts to show the distributions of values. Along with employment status, these are the main variables of interest, so it is important to look at them carefully before doing further data analysis.

The distribution of work ethic and life satisfaction may vary across countries but may also change over time within a country, especially since the surveys are conducted around once a decade. To compare distributions for a particular country over time, we have to use the same horizontal axis, so we will first need to make a separate frequency table for each distribution of interest (survey wave). Also, since the number of people surveyed in each wave may differ, we will use percentages instead of frequencies as the vertical axis variable.

  1. Using the data from Wave 3 and Wave 4 only, for three countries of your choice:
Range of work ethic score Frequency Percentage of individuals (%)
1.00    
1.20    
   
4.80    
5.00    

Frequency table for self-reported work ethic.

Figure 8.5 Frequency table for self-reported work ethic.

  1. We will use line charts to make a similar comparison for life satisfaction (A170) over time. Using data for countries that are present in Waves 1 to 4:
correlation coefficient
A numerical measure, ranging between 1 and −1, of how closely associated two variables are—whether they tend to rise and fall together, or move in opposite directions. A positive coefficient indicates that when one variable takes a high (low) value, the other tends to be high (low) too, and a negative coefficient indicates that when one variable is high the other is likely to be low. A value of 1 or −1 indicates that knowing the value of one of the variables would allow you to perfectly predict the value of the other. A value of 0 indicates that knowing one of the variables provides no information about the value of the other.

After describing patterns in our main variables over time, we will use correlation coefficients and scatterplots to look at the relationship between these variables and the other variables in our dataset.

  1. Using the Wave 4 data:
Variable Life satisfaction Work ethic
Age
Education
Full-time employment
Gender
Self-reported health
Income
Number of children
Relative income
Life satisfaction 1
Work ethic 1

Correlation between life satisfaction, work ethic and other variables, Wave 4.

Figure 8.6 Correlation between life satisfaction, work ethic and other variables, Wave 4.

Next, we will look at the relationship between employment status and life satisfaction and investigate the paper’s hypothesis that this relationship varies with the average work ethic in a country.

  1. Using the data from Wave 4, carry out the following:

So far we have described the data using tables and charts, but have not made any statements about whether what we observe is consistent with an assumption that work ethic has no bearing on differences in life satisfaction between different subgroups. In the next part, we will assess the relationship between employment status and life satisfaction and assess whether the observed data lead us to reject the above assumption.

Part 8.3 Confidence intervals for difference in the mean

Note

You will need to have done Questions 1–5 in Part 8.1 before doing this part. Part 8.2 is not necessary but is helpful to get an idea of what the data looks like.

Learning objectives for this part

  • calculate and interpret confidence intervals for the difference in means between two groups.

The aim of this project was to look at the empirical relationship between employment and life satisfaction.

When we calculate differences between groups, we collect evidence which may or may not support a hypothesis that life satisfaction is identical between different subgroups. Economists often call this testing for statistical significance. In Part 6.2 of Empirical Project 6, we constructed 95% confidence intervals for means, and used a rule of thumb to assess our hypothesis that the two groups considered were identical (at the population level) in the variable of interest. Now we will learn how to construct confidence intervals for the difference in two means, which allows us to make such a judgment on the basis of a single confidence interval.

Remember that the width of a confidence interval depends on the standard deviation and number of observations. (Read Part 6.2 of Empirical Project 6 to understand why.) When making a confidence interval for a sample mean (such as the mean life satisfaction of the unemployed), we use the standard deviation and number of observations in that sample (unemployed people) to obtain the standard error of the sample mean.

When we look at the difference in means (such as life satisfaction of employed minus unemployed), we are using data from two groups (the unemployed and the employed) to make one confidence interval, so the number of observations is the sum of observations across both groups. To calculate the standard deviation for the difference in means, we use the standard deviations (SD) of each group:

This formula requires the two groups of data to be independent, meaning that the data from one group is not related, paired, or matched with data from the other group. This assumption is reasonable for the life satisfaction data we are using. However, if the two groups of data are not independent, for example if the same people generated both groups of data (as in the public goods experiments of Empirical Project 2), then we cannot use this formula.

Once we have the new standard deviation and number of observations, we can calculate the width of the confidence interval (distance from the mean to one end of the interval) as before. The width of the confidence interval tells us how precisely the difference in means was estimated, and this precision tells us how compatible the sample data is with the hypothesis that there is no difference between the population means. If the value 0 falls outside the 95% confidence interval, this implies that the data is unlikely to have been sampled from a population with equal population means. For example, if the estimated difference is positive, and the 95% confidence interval is not wide enough to include 0, we can be reasonably confident that the true difference is positive too. In other words, it tells us that the p-value for the difference we have found is less than 0.05, so we would be very unlikely to find such a big difference if the population means were in fact identical.

In Figure 8.7, for Great Britain, we can be reasonably confident that the true difference in average life satisfaction between the full-time employed and the unemployed is positive. However, for Spain, we do not have strong evidence of a real difference in mean life satisfaction.

95% confidence intervals for the average difference in life satisfaction (full-time employed minus unemployed) in Great Britain and Spain.

Figure 8.7 95% confidence intervals for the average difference in life satisfaction (full-time employed minus unemployed) in Great Britain and Spain.

  1. We will apply this method to make confidence intervals for differences in life satisfaction. Choose three countries: one with an average work ethic in the top third of scores, one in the middle third, and one in the lower third. (See Excel walk-through 6.4 for help on calculating confidence intervals and adding them to a chart.)
  1. The method we used to compare life satisfaction relied on making comparisons between people with different employment statuses, but a person’s employment status is not entirely random. We cannot therefore make causal statements such as ‘being unemployed causes life satisfaction to decrease’. Describe how we could use the following methods to assess better the effect of being unemployed on life satisfaction, and make some statements about causality: