Are Citizen Science Results Valid?
We have gotten some really good questions about the lead sampling approach used for the 300 test kits that are being mailed back to us. These questions were raised through concerns over whether our sampling approach is random and if having citizens conduct sampling will diminish how valid the results are (or how valid they are perceived to be). If you’ve been wondering about this, read on – you’re not alone!
If you wish to submit questions, please use our Facebook page or send us an email on [email protected]
Is it common to have citizens collect lead samples?
Almost all lead in water monitoring done in the United States relies on citizens to draw the samples, including all of the prior monitoring for lead in water done in Flint to determine compliance with the EPA Lead and Copper Rule. So, the sampling that the citizens are conducting is every bit as credible as prior EPA compliance sampling done in Flint (or anywhere else in the U.S.).
Won’t citizens that want to see bad results [to drive change] be motivated to tamper with the sampling process to make the problem seem worse than it really is?
We have noticed that Flint residents have taken the sampling VERY seriously, and have even developed and implemented procedures on their own to minimize the likelihood of someone tampering with them, or to counter accusations that they tampered with samples. We plan to use their innovations in future campaigns. For example, they developed a way of having each homeowner seal and sign the kit, so that no one but the homeowner could have opened the bottle before we check the seal at Virginia Tech. As one women said, “We really just want to know if our water is safe or not, and we do not feel like we can trust the sampling that was done by the city.”
Follow-up: What controls are in place to prevent or detect tampering?
We acknowledge that the sampling approach is not perfect – no sampling approach is. However, the way the EPA test results are calculated allows for some bad data. Out of every 100 samples, the “worst” 9 (i.e., those that have the highest concentrations) are thrown out. You could have 9 samples with over 1,000 ppb lead, and it does not affect the reported 90th percentile result at all. So a few “bad apples” will have no impact on the results in Flint (or anywhere else in the U.S.).
Some people have raised concern over people being allowed to take multiple tests kits away from the pick-up sites. From a practical standpoint, by allowing people to pick-up sampling kits for neighbors/friends/etc., we can increase the amount of people being reached. In order to take multiple kits, a resident had to fill out a card with a unique home address that each kit they took would be delivered to. That card had to have a unique address, phone number, or e-mail that was confirmed by the resident of the house being sampled before being shipped back to us.
We also have ways to “fingerprint” the lead in the water chemically, and can oftentimes determine if that fingerprint is consistent with lead-bearing plumbing as the source. For example, in the home with the highest lead sampled to date in Flint, the “fingerprint” of the lead mineral found in the consumer’s water samples was later found to perfectly match the “fingerprint” of the lead mineral found on the inside of pipes going to their house after they were dug up by the city and retrieved by an EPA scientist who was on site. We are looking at this “fingerprint,” and if anything is out of the ordinary, we will be following up with the person that collected that sample.
Are these samples really considered to be random?
We do not know who has lead pipes in Flint, and who does not. We did not recruit people who only have lead pipe. In that regard, the sampling approach is random relative to the key criteria of having (or not having) lead in the plumbing material. The actual volunteer citizens taking the samples are a self-selected group of people, but that self-selection has nothing to do with the likelihood of having a lead in water risk in their house, and is more likely related to citizens being concerned for their well-being and the well-being of their families.
Why are you posting results before you get all the samples in if you can eventually ignore the worst 9% of results? The number on the website is not really the 90th percentile if you haven’t analyzed all the samples.
We are updating the 90th percentile value as we analyze the samples. So far, the results are worrying. We believe that people can use this information to protect themselves, their children, and developing fetuses from unnecessary and potentially permanent damage known to result from lead exposure.
We believe that withholding this information from Flint residents would be unethical.
Credits: Dr. Marc Edwards, William Rhoads, Siddhartha Roy