As a followup to our introductory post, Gretchen and I wanted to explore the many different types...
Public Lab is an open community which collaboratively develops accessible, open source, Do-It-Yourself technologies for investigating local environmental health and justice issues.
Public Lab Research note
As an open source community, we believe in open licensing of content so that other members of the community can leverage your work legally -- with attribution, of course. By joining the Public Lab site, you agree to release the content you post here under a Creative Commons Attribution Sharealike license, and the hardware designs you post under the CERN Open Hardware License 1.1 (full text). This has the added benefit that others must share their improvements in turn with you.
As a followup to our introductory post, Gretchen and I wanted to explore the many different types of evidence, and talk about their differences, even before getting into the legal definitions and ramifications. Whether you're taking a photo, reading data from a sensor, or cataloging health symptoms, we'd also like to ask what types of information you're interested in using as evidence, and we'll do our best to follow up on each type over the course of the series.
This is a huge topic, and the more we dig in, the more diverse understandings of the term we encounter. In discussions with some of our interviewees for this series, we've encountered distinctions between screening and analytical samples, and samples used in enforcement processes or to gauge remedial progress. We'll look into these to learn more, and get back to you. Can you suggest other types?
Note that the item above, about spreadsheet data, is especially interesting to us -- that is, what about lab analyses themselves? Does a report about a sample or a measurement constitute evidence? Some of the above examples (like this one) are drawn directly from projects we're engaged in, and we'd love to hear from others about specific real-world cases involving different types of evidence.
For one, we're interested in how microscopic photographs of air pollutants like silica (above image by @mathew), and the associated datasheets from their analysis, can be properly collected, stored, and presented as evidence. How does maintaining a chain of custody work with digital data? Do equations in a spreadsheet need to be preserved and documented along with the data themselves? How is that best done -- do we need to, for example, keep a log of timestamps and authors for each edit?
One thing we came across in our exploration has to do with the use of spreadsheets in court, and was hard to believe! Cogent Legal runs a pretty interesting blog on preparing evidence (and derivative graphics) for court, and discussed the logistical and procedural challenges to using or even adding to a spreadsheet used in evidence:
One of Cogent Legal's clients recently went to trial with a spreadsheet similar to the one used for this post. The spreadsheet printed out to be 28 pages long, and the original column had no total on it. For a printed exhibit at trial, counsel added a total to the spreadsheet, but opposing counsel objected to its admission, arguing that the spreadsheet had been altered and lacked foundation. Thus, the attorney offering the spreadsheet as evidence displayed the electronic spreadsheet for the jury on a courtroom screen, and then led the witness through the process of summing the column's total in Excel. Once this total was established, the opposing counsel stipulated to admission of the original document (and also stipulated to the admission of many similar documents). -- http://cogentlegal.com/blog/2014/06/excel-for-litigators/
We'll be looking to compile some best practices to -- hopefully -- avoid that kind of painful walkthrough.
Another usage we're exploring is #timelapse photography (above timelapse by @tonyc) -- useful in monitoring all kinds of activity over long periods of time. There are EPA recognized techniques (like Method 82) which employ timelapse photography of smoke plumes. Folks at LEAFFEST 2016 will be trying out some timelapse methods, and we'll want to know how this sort of evidence ought to be handled as well.
In any case, we'll surely be updating this post as we discover more, and we appreciate any additions you can offer! Thanks, and stay tuned.
@ david-days, Hello, this is dhaffnersr, thank you for such a detailed and informative post. I am really interested to know what are some of your specific thoughts on some of the analytical work that I have done, all my procedures are derived from national laboratory standard practices, and I try to use as many lab grade reagents and compounds in my work whenever possible.
I would be happy to help in any way I can. I will try to post something on your recent work this weekend if I can.
In the meantime, I can say that the kind of work you have been doing is exactly what I believe is necessary to support the community when it comes (inevitably, in my opinion) to convincing someone that there is an important finding. Doing the basic scientific and quality-control checks directly contributes to the overall power of Public Lab in general, and prepares for the eventualities that @warren is talking about in this note.
Reply to this comment...
@david-days, Thanks for your reply, I appreciate it, and I look forward to you next response!
Reply to this comment...
@warren, @david-days is right; and in a number of different ways. Yes, the "integrity" of the data is an accurate reference; both legal and scientific. This includes not only keeping the original data but also not creating error by reporting an extracted result value of eg. 1.254678 when the measurement has an uncertainty of -0.2 +0.6 such that the only result with meaning is a value of 1.2 +/-Uncertainty or, in some cases, 2 (not 2.0, just 2). There is often a tendency to become complacent about collecting / reporting data and not stopping to ask "do these measurements even make any sense?".
This sense of practicality needs to be extended to the question of "could any of [this] data be taken seriously [as compared with professional / certified services]?". I suspect that an unprocessed, time-stamped photo is probably "valued" at least 10x more than a few measurement results (where the potential for points of error can be far greater.). It is not difficult to argue that improper measurement and data collection produces useless information -- an all too common event. However it is difficult to argue that if contributors learn how to do proper measurements and proper data collection, then the information should be accepted evidence. The difference is in the expectations; accurate, meaningful measurements / data should be both a goal and a necessity but expectations of its acceptance, value and use should be tempered; first by peer review and then by the harsh realities of non-scientific perspectives. Unfortunately, scientific 'proof', even if overwhelming, is not a guarantee of legal 'proof' of action or even intent. What that means is that the "front end" of that process is essentially the only part over which there is some measure of control. That means it is possible to deliberately choose to do the up-front work accurately, thoroughly and with peer-review as without doing so, the 'back-end' of the process will either find the information useless or the process will fail to 'prove' its case.
"Typically, scientific notation is used for this purpose. If 200 has two significant figures, then 2.0 x 102 is used. If it has three, then 2.00 x 102 is used. If it had four, then 200.0 is sufficient. See rule #2 above."
@warren, et. al. Seems my point was missed; this is not a trivial point and it isn't just a matter of scientific notation where measurements are concerned. If 2.0 is given w/o the uncertainty, but the uncertainty were +/- 0.8, then 2.0 would imply greater accuracy than is warranted; so '2' is more appropriate when no uncertainty is specified. 2.0 (w/o the uncertainty) implies <0.05 uncertainty because 2.0 is specified and not 2.1 or 1.9. Claiming a calculation result of 2.034685 when using measurements from a 12-inch ruler is misleading because measurement uncertainty is, at best, about +- 0.02. So, '2.03 +/- 0.05' would be a more accurate representation and, depending on the calculation, even that could misrepresent the result's accuracy.
Reply to this comment...
@warren, @stoft, no, your point was not missed, a ruler measurement of notation such as 2.034685 would be irrelevant, since 4685 would be quite difficult to measure on a practical scale of any sort. "my" point was, only that there is a time to split a hair and then there is just a time to let it go.
Reply to this comment...
I will no longer be posting my research here at Plab, now or in the future, my reasons are simple, I work very hard at what I do and what I am trying to accomplish, I work every single day at this, because I am retired and I enjoy it, what I do NOT enjoy is 1) censorship of conversation and thought. 2) There is a lack of real leadership here and it manifests itself by a real lack of focus and attention to the individuals who do put in a lot of hard work in contributions here, there is a "clicky" type atmosphere going on here that I do not appreciate, i.e., seems like the only "real" science going on here is tying a string to a balloon and sending up a camera to take some topographical pictures?
There is no reason in my opinion, to even bother with spectral workbench 2, when it's potential is begin wasted on lamp light. I have pointed out before in my research, that one cannot do absorption spectroscopy using spectral workbench because the "leadership" is not really concerned doing the real science of this craft, but only the topical aspect of it, perhaps only for the reason of a selling point for a more marketable cellphone toy (hence; Desktop spectrometer 3.0.) The "leadership" should be more forthright with newcomers in communicating the true intentions of Plab, which again, is just my opinion and observations, that this is more, just an environmental advocacy and activism organization than a scientific one.
Hi folks, I totally agree with @stoft that it is essential to understand the limitations of any data before trying to present them as evidence (my paraphrase there), and to some degree may be more important than fully understanding the implications of any data too. Even with data that is photographic, I think it is important to assess what the photograph doesn't demonstrate or prove too. It's a bit tricky since each situation is unique, but I'd really like to start a blog/research note series, or a wiki page, with basic guides for data literacy with various kinds of data and various kinds of end goals for data utilization. Would that be useful? Would it be possible to make it actually useful while still being quite general/broadly applicable?
I'd agree with @gretchengehrke that building awareness of what information data contains, and what information it does not contain, is of primary importance. While there are specific examples, learning how to ask the right questions may be more important because "data" can take so many forms and formats. So my $.02 is to focus on what questions to ask and then use specific examples showing how those question perform in the real world. Learning this is a very useful tool as it promotes thinking "in advance" -- i.e. IF I could get the following data, would it be useful? and How detailed / accurate would my data need to be to be meaningful?
I'm going to jump again in as a contrarian, but with a purpose (hopefully it will help). My intent is to step lightly, but I may step on everyone's toes a bit. Please don't take it personally...
As in all things, I am just a layman (IANAL), so take this as just commentary.
The context of this article, and the focus of my approach to it, is the battlefield that is the court proceeding. The scenario wherein a Public Lab member will find themselves in such an environment is, I imagine, something like this: Public Lab member @envwatcher takes an interest in the environmental goings-on in a particular geographic area. Maybe @envwatcher is called in because someone else is looking at environmental damage; maybe something happens in an area for which @envwatcher is already gathering data; or, possibly, @envwatcher is looking through data from others and comes up with something interesting or probative.
For whatever reason, @envwatcher finds [his/her]self talking to an attorney who is putting together a case. (Again, who initiates that contact can go either way). IMHO opinion, a "good" attorney will look at the data and @envwatcher with a critical eye, trying to figure out how the data, methods, and conclusions fit into the narrative that the attorney is building. The narrative is important, because it is how the case will be handled in many ways: The filing, the presentation of the evidence (all types), the fight for summary judgement (US law--we're talking a civil suit of some kind. Criminal law is something else, and I'm not going to even pretend to explain how that might work.), and the relief (injunctions and/or damages) that will be the result of a successful case.
The narrative is also important because the opposing attorney(s) will be trying to tear it down. This is where the critical eye I mentioned comes up: A "good" attorney will be looking at not only the data and conclusions, but also the person who will probably end up explaining it to the court. Does this person sound knowledgeable? Is there an obvious ax that they are grinding that the opposing counsel will use to beat up the witness? Do they know their stuff? Lots more questions that I couldn't even imagine.
Here's where the discussion comes back to this comment thread: Our intrepid potential witness, @envwatcher, is a member of Public Lab. What is Public Lab? Is it a group of amateur scientists? Professional scientists? Is it an environmental advocacy group? A bit of each? Do member fall into separate categories (and across categories) based on their personal interests?
This comment thread (and some others) seems to expose, to me, an uncertainty in the community regarding what Public Lab, as a whole, is or could be. For myself, it's more of a "power to the people" thing, technologically speaking. If you create enough low-cost tools that allow people to do interesting and amazing things, then the world, as a whole is better. A few years ago I was playing with the idea of designing my own arduino-based ultrasound. My wife had had a serious ectopic pregnancy that almost killed her, and I was amazed at both the expense and complexity of the systems. Turns out they are becoming serious items, and I call that a good thing.
This is not necessarily a "pure" approach to Public Lab, however. Equipment, systems, and people cost money, and it would be very easy to morph from a "power to the people" environment to a "and we can be popular (and/or make money)!" mindset that would be corrosive over time.
Alternatively, there is definitely a reason for pushing the scientific aspect of Public Lab. As I mentioned in my earlier comment, knowing the tolerances and principles of the equipment and methods is very important. Wild-ass pronouncements and unfounded declarations hurt any good that Public Lab can do, and this part of the community is absolutely necessary.
However, cloning the approach of the established scientific community has its drawbacks. While users are arguing over how many angstroms can dance on the head of a pin, the real world is happening, and lots of good that can be done will be back-burnered and, possibly, too late. Also, as @liz's article from earlier pointed out, peer review can lead to a pretty stifled culture that moves slowly in a quickly shifting world.
Then there are environmental advocates in Public Lab. I would present a caricature of this member as someone who has a deep felt belief in a position ("Waterway spill X caused serious environmental damage"), and is looking for proof of their belief. This approach would often be the reason that @envwatcher is getting involved in the first place. They have an interest and care, and are willing to put in the time to monitor an area that no one else has the time, money, or interest in looking. It's actually the basis for growing Public Lab--people who care about their environment.
Unchecked, though, can be dangerous in that a frothing-at-the-mouth environmentalist won't even get to present evidence. It's too obvious that the belief outweighs the approach, and I would imagine that attorneys would get the idea that Public Lab members are to be avoided.
So while the debate rages on over which aspect of Public Lab is "most" important, I think that we should understand that Public Lab itself interests people because it does not fall into any category as a whole. We already have the academic scientific community, Silicon Valley "cause" startups, and long-standing environmental watch dogs. If someone is coming to Public Lab, it may be because they find down-sides to those other groups, and they see somewhere that they can fit in without having to swear allegiances that they are unable to support or have reservations about.
The "battlefield" I referenced earlier is about people and evidence. It's also where real change comes about. It doesn't matter that people "knew" that lead-based paint is dangerous--it mattered that the cost of the product became too high because of court cases and EPA fines. This is the larger good that I would like to promote. But it can't happen without a balanced community that cares enough to look, is careful enough to build a solid base, and enjoys the technical challenges enough to invent new ways to go about things.
At the risk of missing his points, it sounds like @david-days is suggesting PLab might be described by (no particular order or emphasis) 1) environmental activism, 2) promote the collection of environmental data (broad definition), 3) involvement in low-cost, "accessible scientific measurement" devices and 4) promote "citizen science". Yes, I'm deliberately using a very broad brush. The reason is because I'd like to suggest that science is the common thread and the scientific method holds the fundamental concepts having the potential to keep all the components well grounded.
The legal, public, social, political issues of evidence are complex but all are limited by the science behind the data. Low-cost, "accessible", measurement devices are useful only to the degree of the science applied to their design, construction and use. "Citizen science" is only as valuable to the degree it remains true to scientific principles. Environmental activism is only as persuasive (long term) as permitted by the validity and openness of the science supporting it.
PLab provides sandboxes where interested citizens can explore but that doesn't guarantee everything in the sandbox will stand up under scientific inquiry -- but that's ok, because that's the definition of a sandbox. PLab also promotes science-based "stuff" (deliberately diffuse so as to be generally inclusive) and that part does require scrutiny and validation in order to pass scientific inquiry.
So, this is not an argument of the very real issues in @david-days reflections. Instead, it leverages those thoughts as a perspective that learning to follow the scientific method is the best reference point for keeping all niches of PLab well grounded.
I found the following quotes in an article in the Guardian science section, and thought they held some basic insight.
"...scientists keep on testing their own assumptions, and questioning their own observations. Yes, if they expect to find something, there is a likelihood that they will see what they are looking for, but they don’t like to be the only people to see it. The lesson is that good
science is replicable."
"...good scientists worry about tiny details: it’s not enough to be broadly right. If you can’t be dead right then something must be wrong."
"...Science is unique among human ways of knowing because it is self-correcting. Every claim is provisional.”
Reply to this comment...
@david-days@stoft,@warren, I agree that sometimes, "peer" review can be a bit like, trying to sprint through a pool of molasses, although this is critically necessary in the context of proving an assumption or scientific experiment. There is also the unique problem of "ego" review, what I mean by this is, "audit of data, by self-perceived intellectual superiority." Example., scientist "A" reviews scientist "B's" research paper, scientist "A" determines that scientist "B's" work is inferior, this process continues a number of times with the same result; inferior.
Scientist "B" starts to correct deficiencies from previous "peer" reviewed work. Scientist "B" again presents a higher level of experimental research, scientist "A" again determines that scientist "B's" work is once again, inferior. This process repeats itself a number of times, until scientist "B" realizes that scientist "A" has not crossed all their "T's" and dotted all their "I's."
Upon further scrutiny, scientist "B" reviews scientist "A's" work, and constructs a systematic profile based on all of scientist "A's" available research, and the results are surprising, "audit of data, by self-perceived intellectual superiority."
This diagnosis can be reduced into its respective component parts;
1) "I speak from a position of authority."
explanation; one cannot "speak" from a position of authority, this implies that you have acquired all knowledge in a particular subject, this is engaging in "absolutes."
2) "I can exercise this authority in the cloak of anonymity."
explanation; The internet provides this type of anonymity, scientist "A" is "faceless," a ghost with a generic avatar, yet is recognized by their peers as an authority figure. There is absolutely no way to validate scientist "A's" credentials, one is only left with the option, of taking scientist "A's" opinion on "faith."
3) Intellectual corruption; "follow the data where it leads, no matter "where" it leads!" Plab, another component in my discussion here, the quote above is a universal truth, but few ever follow this path. "activism" can often be synonymous with moral corruption.
explanation; good intentions, are never the best intentions, yes, this sounds cold and heartless, it is the truth though, example; "but my heart was in the right place" "well, at least we tried something." Many, and I mean, many times there is never a good plan if there was a plan at all in these hypothetical endeavors.
Many instances, and it can be proven, that when it comes to environmental science, research data can often times be manipulated to "fit" an agenda or a particular narrative.
I see that behavior here at Plab, cloaking one's self in the "code of conduct" every time there is the slightest perceived provocation, that is an insidious pathogen, and it is not the engine that drives the creative processes or inspires the imagination, it is the antipathy of it.
4) Plab's "progress toward community pollution analysis"; -profile view: audit of data by self-perceived intellectual superiority.
explanation; Scientist "B," "your research, does not adhere to our standards of conformity nor data quality." Counter argument by scientist "B," "you have no real research data of your own though, that conforms to any standard that I can perceive?"
Plab's counter argument; "of course we have research and data, look at all the oil bottles and eye droppers we have, and balloons that we have sent up in the sky, and what about all of our laser pointer work, shinning those pretty purple beams through all those oil bottles?"
Scientist "B" points out that, not one research paper has ever been posted or presented in any professional or scientific format.
Scientist "B" points out their hypocrisy for the pot calling the kettle black.
4) Plab's "progress toward community pollution analysis"; -profile view: audit of data by self-perceived intellectual superiority. explanation; Plab: "your research, does not adhere to our standards of conformity nor data quality." Counter argument by scientist "B," "you have no real research data of your own though, that conforms to any standard that I can perceive?"