17.1 Sampling Techniques and Study Types

17.1.1 Sampling Errors and Bias

Statistics would be theoretical mathematics if we did not have to operate in the real world. However, because statistics evolved out of a desire to make conclusions about the real world and the real world relies on humans, there are a variety of errors we introduce into the theoretical models that we have to account for when applying statistics to the questions we have posed.

• Selection bias. One of the biggest causes of error when applying statistics is a bias caused by the method of selecting the sample for the analysis. There are many types of sampling techniques that researchers use to simplify the data collection process, with some of the most common listed below.

  • Judgement (purposive) sampling or purposive samples

    The researcher uses their own judgement when choosing members of the population for the sample.

  • Snowball sampling

    The researcher uses participants to recruit more participants.

  • Quota sampling

    The researcher chooses a sample to represent the proportions that exist in the population.

  • Convenience (accidental, opportunity) sampling

    The researcher samples from a subset of the population that is easier to obtain. For example, students in a college class to represent the population of adults.

  • Voluntary sampling

    The sample from the population is based on volunteers participating, rather than those randomly selected.

  • Consecutive Samples

    The researcher continues accepting responses until the desired number of samples is achieved.

• Random sampling error. Another large cause of error in statistics arises as part of the random sampling process. Even if everything is done perfectly, any time one chooses a sample to represent a larger population, there is error introduced. This type of error can never be removed, but can be controlled and understood by paying attention to the sample size and the sample’s representation of the overall population represented.

• Coverage error. If a researcher wishes to study the opinions and preferences of registered voters, they may choose to use a data base of phone numbers. Since not all voters have a listed phone number, and particular demographics are more likely to block unknown numbers on their phones, an under-coverage error may occur. If the researcher instead uses a list of email addresses it is very likely that a person may be contacted multiple times for the same survey if they have multiple email addresses. This would result in a possible over-coverage error.

• Measurement error. Any time that we try to measure something (the width of an object, a person’s opinion, a student’s knowledge) we cannot be entirely precise do to problems with our measurement techniques and instruments. Since we cannot eliminate the errors caused by the issues, we try to understand and control for these errors. For instance, if someone is trying to determine a person’s opinion about something, the way that the question is phrased can have a very significant effect on how the person answers. In order to minimize this effect, pilot groups (a smaller test sample) are used to check how different people interpret the questions and questions are then rephrased.

• Processing error. Sometimes errors arise in the transfer and handling of data (i.e. copying information between a form and a spreadsheet) in the data collection and analysis process. These errors can be reduced by creating detailed protocols and having multiple people checking the work. However, even with the best protocols and backup contingencies, some processing errors make it through.

• Non-response or participation bias. Since most people do not like taking polls or surveys, there is likely to be a non-response bias. It is important in the data analysis process to try to understand this bias and take it into account in the design and interpretation of the analysis.

The good news is that careful planning will help us minimize some of these. The bad news is that because we are humans working with questions that don’t have known answers, we can never be certain whether we’ve addressed all the important sources of error. The job of someone who is setting out to answer questions using statistics is not to eliminate all sources of errors, but rather to acknowledge those sources, explain the procedures they have done to minimize the potential impact they may have on the results, and discuss their findings in light of these potential error sources.

17.1.2 Probability Sampling

As our previous discussion shows, there are a lot of ways that we can bias a sample unintentionally by not doing a good job of picking our sample to reflect our population of interest. Statisticians have a solution to this that you have probably heard of called random sampling. Randomness (or probability sampling) introduces a way to gain a sample that is likely to be reflective of your population, but this is not as simple as it sounds–there are different ways to sample randomly, each which help address different biases you may want to avoid in your sample.

To understand these, imagine you want to understand the citizens opinions about environmental regulation in a large city. The city has \(N\) people (your population) and you plan to sample \(n\) of them (where \(n\leq N\)).

Simple random sampling occurs when each person in the population (here, the city’s citizens) has the same probability of being in your sample and this probability is \(\frac{n}{N}\). With this method, we will have a sample that reflects our population.

Systematic sampling is similar, but instead of selecting your \(n\) people at random from the population, you apply some order to the population and then pick every \(k\)th person, where \(k\) is chosen to ensure the correct sample size at the end of the list. So, for our question, an order may be placed by listing all residential addresses, with a survey team member knocking on every \(100\)th address in order to get a sample of 10 percent of the city’s population.

Both simple random sampling and systematic sampling provide a random sample and so are both good sampling techniques. How a researcher would choose between the two options would be dependent upon the context. However, both of the sampling techniques assume the population is uniform across all the ways that might be important to your question. If there are reasons to expect that is not the case, then, depending on how big \(n\) is relative to \(N\), these techniques may not include people who are members of small groups that are important to you, but not likely to carry a lot of weight in a small sample.

For example, say your city has 100,000 people, and you sample 200 of them. If you are particularly interested in how families with children under 5 years old feel about your question, but only about 5 percent of your city’s population are in this situation, you can expect that about 10 of your 200 people sample meet this criteria. This may not be enough representation of this particular group to have sufficient variance to understand their feelings about the environment.

You could resolve this issue by increasing your sample size, but collecting data tends to be costly, especially if you want a representative sample, so other ways have been developed to preserve randomness while increasing the representation of important categories within the sample.

Stratified random and cluster sampling are sampling techniques that follow this. These techniques divide the overall population into sub-groups and then used random sampling techiques for each of these sub-populations.

So, while “random” is good for sampling, it is not enough–you need to attend to the characteristics of how people are being chosen.

17.1.3 Study Types

Sampling, discussed in the previous section, does not occur in a vacuum: we collect data from a sample because we are conducting a study. The way that we collect data impacts the type of study we end up doing and the types of conclusions we can draw.

The most common type of study people learn about are experiments. Experiments are a type of study where we are able to randomly assign our sample into different conditions and then examine differences between the different conditions. There is a reason these are discussed widely–when we are able to control the environment of our sample, we have a pretty clear causal pathway and can thus make strong claims. Drug trials are often conducted using experiments for this reason. The FDA requires that drug developers show evidence of clear, positive effects when deciding whether a new drug is safe and effective or not. What characterizes an experiment is not randomness–most studies have an element of randomness built into them–but where the randomization occurs. In experiments, the randomness occurs when an individual is assigned into a condition. We may also pick participants randomly, but what is important is that once we have them, random assignment is used to determine which treatment they receive.

Related Content Standards

• (HSS.IC.3) Recognize the purposes of and differences among sample surveys, experiments, and observational studies; explain how randomization relates to each.

Studies without random assignment can still teach us a lot, but the causal pathway may be less clear. Many studies involve randomness, but randomness serves a different function within the study. For example, say we are interested in the relationship between the health of a population and the level of air pollution in their neighborhood. To investigate this question, we pick a random sample of people from a variety of neighborhoods and measure their health and the level of pollution in their neighborhood. This situation includes the selection of a random sample, but it is not an experiment because we do not have control over their conditions–we cannot, once a person is selected for our study, assign to them the condition of whether they have high or low pollution in their neighborhood–instead, we observe the variables of interest. Studies characterized by observing variables of interested on a sample are called observational studies.

A last common type of study is one in which a person uses a sample of convenience. For example, say an engineer is investigating the quality of widgets coming off the line of one of their machines. They use the 20 that came off the line first. This is a sample survey, but there is not randomization going on. These survey samples are common and characterized by the lack of randomization, either in terms of how the sample is collected or in how conditions are assigned. Such studies types are still valuable, but more in providing evidence that a pattern may exist. A more careful study that includes intentional sampling to better account for possible sources of variance and/or random sampling or assignment can improve and strengthen claims.

Note that the above discussion is about the role of randomness in terms of a study type. The sampling type also contributes to the characterization of a study and together, these elements build the study design.

17.1.4 Exercises

1. Write a summary of the various non-probabilistic sampling techniques listed at the beginning of this section.

2. Suppose that a theme park wants to get information to discover ways to improve the park experience and to increase the number of attendees.

   1. What are different statistical questions that would be appropriate for such a study?
   2. What types of sampling methods would be appropriate for answering these questions?