Public Health 250A - Lecture 24

Uploaded by UCBerkeley on 19.10.2012

Art Reingold: Okay. So, first of all in case
people have lost track, again a reminder next Wednesday is the
midterm and you'll need a calculator. This morning we're going
to have a presentation that often we have a little later in the
semester. It's a critically important presentation. Professor
Steinmaus has been here before talking about cross sectional
studies and historical cohort studies. One of the things that
I think we'll be addressing in a little more detail later in
the semester is once we have a body of evidence concerning a
particular exposure outcome relationship, smoking and lung
cancer, whatever it is we need to think about how to gather
that evidence together in some sort of systematic way and
figure out what we think it all means taken collectively. And
generally there are multiple studies about a particular subject
and how do we go about bringing that information together.
One approach in the old days was what might be called
systematic reviews. Another approach that has a lot of
traction these days is what's called meta-analysis and
Professor Steinmaus is certainly one of our local experts on
meta-analysis. In fact he and Professor Smith give a course
each fall on meta-analysis and systematic reviews and if you
are interested in this topic next fall you may want to
investigate taking that course. It's really quite valuable for
those of you who are going to be working in public health. In
any event Craig is just going to give you a taste this morning
about what meta-analysis is all about when we have multiple
studies of the same exposure outcome relationship.
Craig Steinmaus: Okay. Can we start? Can you
hear me in the back? All right. So yeah we'll talk about
meta-analysis today. Unless anybody wants to talk about
somebody else? No? Meta-analysis. Okay. If you're asking
for it we'll talk about it. All right.
So yeah, what Art said. Sometimes on a particular
topic that you're interested you have a whole bunch of studies
and you want to somehow summarize those studies so
meta-analysis is one way of summarizing those studies. Taking
a whole bunch of studies and coming up with an overall summary.
Why do you need to know about meta-analysis? Number one
there's a lot of them out there. On any particular topic
you'll do your Pub Med search and you'll come across
meta-analysis. When I started doing this when I think I went
on Pub Med this is ten years ago there weren't that many. Over
time it seems to grow and grow and grow. This is 2011. 51000
when you type meta-analysis. I'm sure today it's close to 60
or 70000. Meta-analysis is a whole bunch of different topics.
Occupational medicine, environmental medicine, infectious
diseases, clinical trials, genetics. Cohort studies, case
control studies and you combine studies that present data as
relative risk or difference between means or regression
coefficients. It incorporates a whole bunch of different types
of studies where you can summarize those literature or that
Okay. So first question is what are some of the
general ways we have summarized literature in the past? I'm
going to give you an example. The guy at the office next to me
is a pediatrician. His name is Mark miller. His boss came up
to him and said Mark, does environmental tobacco smoke or
secondhand tobacco smoke does that increase the risk of breast
cancer. My office at Cal EPA wanted to know that. We wanted
to know if we should push regulations to limit secondhand
tobacco smoke. Mark had to take a large body of literature on
this topic and summarize it to come up with, yes it does seem
to cause it, no it does not seem to cause it. I don't know is
not a great answer. Right?
Because I don't know means well, you don't do anything
about it. Right?
In epidemiology it's an imperfect science. There's a
lot of times where we can't really prove this causes this. We
don't have absolute 100 percent proof. So what we try to do is
come up with well, what's the general consensus? Is there
enough that we should probably be safe and go ahead and
regulate something like environmental tobacco smoke.
So Mark was asked that question. He went through the
usual steps of summarizing the literature research. Find the
research question. Do your literature search to look for all
the literature you can find on this particular topic. Evaluate
the studies and that's what you are learning in this class to
evaluate the studies. Look at bias, confounding, consistency,
causal inference, look at all that different stuff.
And then take all that stuff that you just did and sort
of somehow summarize it. So that's what we're going to talk
about today.
Now one thing I did want to point out that keeps coming
up in the meta-analysis class is the literature search. I
always get asked what's the best way to get studies where
should we get studies? If I was going to give you a one word
answer, Pub Med is the greatest. I think we've published 12,
13 meta-analysis and I hate to admit this but every single
study that we've ever found that we've ever put into a
meta-analysis are all on Pub Med. Saying that you should
always double check any work you do in epidemiology. Always
double check it. So we always look at another. Try to look at
some other database. So what I did was I actually went on
these major environmental journals and looked up meta-analysis
and looked how they found their articles they ended up
selecting for their meta-analysis. Every single article did
use Pub Med. You can see some other ones. Web of science.
I'm not an expert in those. That's where people generally seem
to do their literature searches in published meta-analysis.
Again, we've defined our research question. We've
gotten, we've done our literature search. We've gotten all the
studies on our particular topic and then again the next step,
you want to evaluate each study. Is there major bias is there
confounding? What's the probabilities due to chance? We're on
semesters right? You are, that's a big giant job right there.
Evaluating a study. That's not what I'm talking about today.
That's all your other classes. Evaluating each study and
deciding what are the good studies and what are the studies
that aren't good? What are the studies going to give you
information on your question and what aren't? What have fatal
flaws. Significant bias, significant confounding. We can push
those to the side.
And then when we're doing this whole process we do want
to set up inclusion and exclusion criteria on what types of
studies we're going to include. That's based on knowledge of
your topic and what the important biases are that could affect
the studies on your topic and the important confounding
variables that could affect the studies on your topic. You
want to set up really strict and formal inclusion exclusion
criteria. I'm going to include these types of studies. Now I
can't give you a whole lot of information on this because it
varies from topic to topic to topic. But the bottom line is
you want to have a good reason for everything that you do if
you are going to exclude a study you want to have good reason
for it and you want to avoid vague criteria. You don't want to
say I'm only going to include good studies. You want to be
specific. This is my first meta-analysis where we looked at
the effect of beta carotene and whether it decreased bladder
cancer. You can see our inclusion criteria. We weren't going
to include cross sectional studies or ecological studies but we
had a reason for each one of those. In the meta-analysis
process you'll do the same thing. And again it's all dependent
on what your topic is. Maybe some topics, cross sectional
studies are okay, some topic they aren't. Strict formal
exclusion inclusion criteria.
Again, that's based on your topic and evaluation of
bias, looking at important confounders. Eventually you'll
reach the point where you'll get what you have decided are the
good studies on your topic. The studies that give you
information you need or want and there's no fatal flaw in terms
of significant or substantial bias. Mark came up with these
particular studies. He did environmental tobacco smoke and
breast cancer in post and premenopausal Wyoming it's thought
that with breast cancer there are two different diseases. The
things that cause breast cancer before you have menopause are
probably different than after menopause. That's why he split
it up.
Anyway he found all these particular epidemiology
studies on this particular topic. Here's the relative risk and
confidence intervals. Some cohort studies, some case control
studies, CC. We'll talk about this in a few minutes.
So those, and he went through that whole process,
evaluating was there major confounding, was there major bias in
these particular studies and he had evaluated each one of those
and decided there wasn't. But these would give him good
information on that particular topic. Okay. Once he's got
these studies then how does he come up with an overall yes or
While I'm talking, while I'm blabbing on go ahead and
take a look at these numbers here and these numbers here and
take my word for it that these did not most likely have major
bias, major confounding, but they were pretty good studies and
in your mind kind of come up with an overall conclusion if you
had this data here knowing they were good studies would you say
yes there's an association or no there's not an association.
Okay. And that's one way of summarizing literature. It's just
kind of looking at the data, mixing everything up in your
brain, right? And then out comes an answer. A yes or no. Yes
it does seem like there's an association or no there doesn't.
But that's not a very quantitative process. Right?
That's a subjective review. Let me emphasize that word
subjective. Different people's brains my evaluate things
differently. You would like to know how somebody's brain did
evaluate something. If they took it all in and spit something
out, a yes or no that's not really science.
We want to quantify things a little bit more than that
than these subjective reviews. That's the goal of
meta-analysis. There's also expert panels. You don't rely on
one guy and his brain to spit out an answer. Maybe you get a
whole bunch of experts including people from UC Berkeley. Get
a whole bunch of experts to sit together in a room and they go
over the data and all of them eventually at the end of the day
they bolt. This was on formaldehyde. Is it a carcinogen?
They looked at all the literature and they eventually at the
end of the day raise their hand for yes or no. That's a little
subjective too. It depends who you get in the committee and
what their tendencies seem to be. Again, that's somewhat
subjective as well. Meta-analysis we want to quantify things a
bit more. Another method is boat tallying. Count the number
of number of positive and negative studies. If there's more
positive studies then yes there's an association.
So you could do something like that. This is a lot of
times in the past was done based on statistical significance.
For each study that is statistically significant in red, found
a statistically significant effect, statistically significant
means it excludes one, that would be a positive study. Each
study that did not find a statistically significant effect that
would be in the black. That would be a negative study. If you
went through these studies you would find what is it? 7 and 7.
Seven positive studies and seven negative studies. It's a
wash. No association. That's vote counting. Counting each
study based on its statistical significance. I'm sure most of
you can see a whole bunch of problems with that. One is you'll
have studies like this, the Delfino study they did find almost
a threefold increased risk of breast cancer in people exposed
to secondhand smoke. That was negative because the confidence
interval included one. But the question is, is that really a
negative study? They found almost a threefold association.
But just because it was small, probably, the confidence
interval included one.
So, is it really negative or was it just -- or was it
positive and just too small? It didn't have enough statistical
power or subjects. Maybe if it was a little bigger that
confidence interval would have been a little narrower. I think
that's a general principle you'll learn in stats. As the
studies get bigger the confidence intervals get narrower.
Again, is that really a negative study?
That's one of the problems with vote tallying. That's
one of the problem that meta-analysis tries to deal with.
Another thing you could do is take an average result and see if
that average result is greater than one. You could take an
average of all these. Again, there's some problem with this.
Not all studies are created equal. Some studies, for example,
are bigger than others.
So wouldn't it make sense if a study, we have one study
with 10000 people. We have a is very similar study and it only
has three people. Wouldn't you want to give greater weight to
that study with 10000 people? Probably. I would think.
Everything else being equal.
So just taking an average you don't really do that.
And maybe it would be nice to do that.
All right. Another thing is that well, all studies are
not created equal. And we could maybe give studies with
greater quality more weight. For example, we could say and
never say it in my class because I do case control studies. We
could say that cohort studies are better than case control
studies. We could say that.
Right? And we could say, okay, well every case control
study we're only going to give half the weight we would give a
cohort study. We could take each result, weigh it by how good
a quality we think it is and then do a weighted average. A
weighted average. But the problem is that, okay, I just said
that cohort studies are better than case control studies.
Sometimes they are, sometimes they're not. Right? It
all depends on the study. You could have the absolutely worst
terrible cohort study that was ever done. Tons of bias. Only
one subject. Absolutely terrible and you're going to still
give it two times as much weight as the world's greatest case
control study. Right?
So, the sort of blanket statements about study quality,
it's difficult to support those statements. So it's difficult
to weight things on study quality. Based on factors like this.
Case control versus a cohort. Then maybe you really do believe
case control studies are better than cohort studies? Why not
1.8? Why two? That's why most of us that do meta-analysis we
don't use these quality scores like this based on these
factors. Where do you come up with the number? A lot of times
there's so many exceptions to these sorts of ratings that they
are not usable.
What we do in most meta-analysis is we weight of study
size. And what I mean by study size is precision. I'll talk
about precision in a second here.
And we don't really weight on study quality except for
what we usually do, what most meta-analysis do. We've gone
through each of our criteria to rate whether a study gives us
the information we want or doesn't. We've assessed is there
major bias? Is there major confounding? If there is we do
weighting. That weight is 0. If there isn't a fatal flaw we
give it a weight of one. We exclude our bad studies. We are
weighting but it's 1 and 0.
There's other ways of incorporating information or
looking at issues of study quality and that's subgroup
analysis. I'll talk about those in a few minutes. The bottom
line is this is how most meta-analysis is done. We take the
studies that are good enough, combine them and weight them on
study size or precision.
All right. So, all right. We're going to go through
the mathematical portion of a meta-analysis. This is all your
other classes. Let's go through this right here, this
mathematical portion. We have our good studies. And we've
selected them. We have our relative risk. We have our
confidence intervals. When you do meta-analysis you put
everything on the log scale.
I don't want to get too into this. It's basically
because the log scale is normally distributed. Whereas the
relative risk scale is not normally distributed. If you don't
get that and you are curious, come ask me. If you don't get it
and you are not curious it's not going to affect anything in
your life. We'll take it out of log scale later. Everything
goes on log scale. All the relative risk, all the confidence
intervals we put it on the log scale. Simple. We calculate
this BI. Let's call it a coefficient. Basically this is the
log of the relative risk. Nice and simple. You can do this in
Or on a calculator or anywhere else. All right. And
again take the log of our relative risk and take the logs of
our confidence intervals. That's what that is, log, log, log.
Nice and simple.
All right. And then we want to weight each one of
these, each study based on our study size. But what is study
size? Is it the number of subjects in the study? Is that
study size?
There's a problem with that. Where you could run into
situations like this. You could have two studies that exactly
the same number of people in them. 200000 people. But this
study only has one case. Right? Versus this study has a lot
more cases. Now can you really, people with the disease,
people with the out come. Can you really tell much about
what's going on with the disease if you only have one person
with that disease in your entire study? Probably not. You are
probably going to get a lot more information out of this study.
So you can see there's potential problems with just weighting
the individual studies based on the number of total subjects.
And so, and also did you talk about statistical power
in this class?
Art Reingold: Later in the semester.
Craig Steinmaus: You'll learn about statistical
power, how much power your study has to identify an effect.
That is really driven not all that much by this, but
that's really driven by how many people with your outcome that
you have. This is really the driving factor.
So maybe we want to weight our studies based on the
number of people in our study with the outcome of interest.
But there's potential problem with that. We could run into a
scenario like this. You know, where we have studies with the
same number of people with the disease but we only have one non
case here. Again, can you really get a whole lot of
information about a disease if you only have one person in the
comparison group. Not much.
Not much. I think you can see you'd probably want to
give this study more weight. We can't really weight based on
the number of people with the outcome of interest. Or even the
total sample size. There's problems there. So what do we do?
It turns out we weight by the variance of the relative
risk. When you get the variance of the relative risk that
incorporates all that stuff. It incorporates the study size,
it incorporates the distribution of the cases and the controls.
So the study that gives you the most information that's
most precise information will have the lowest variance. Okay?
Big studies have low variance. All right? And by big I mean
that not only the total study size but that distribution of
cases and controls.
So we actually do end up weighting by the variance. Or
I should say the inverse of the variance. Does that help you?
It doesn't help you that much. What's the variance of the
relative risk? There's a whole bunch of different equations to
calculate the variance. The bottom line is the variance
incorporates the study size, big studies have smaller variance
and it incorporates that distribution. There's a whole bunch
of different equations for variance. This is one. I think
this is an estimate of the variance. Look at this equation for
a case control study to calculate the variance. The A, B, C, D
in your two by two table boxes. What would happen to the
variance in A, B, C and D are really big? Right? They are in
the denominator. If they are really big the variance is going
to drop. You can sort of see how study size is related to the
variance. Bigger studies, lower variance. That still doesn't
help you, does it? How do you get the variance?
It turns out we can get the variance of the relative
risk from the standard error. Great. That still doesn't help
you. How do you get the standard error? It turns out we can
get the standard error from the confidence interval and guess
what? We have the confidence interval. We can take the
confidence interval, calculate the standard error and then use
that, square it to calculate the variance. Look at this
equation here. And remember what I said, bigger studies the
confidence interval will get narrower.
So look what happens when you have a big study. What
happens to this quantity right here? It gets smaller. All
right? So big studies this quantity will get smaller. And as
this gets smaller this gets bigger. As this gets bigger this
gets bigger, this gets bigger. Did I say that right? No I
didn't say that right. Let's start all over again. Ready?
How come nobody raised their hand? You're not listening. As
this gets smaller this gets smaller. Right?
As this gets smaller this gets smaller. As this gets
smaller this gets smaller. For bigger studies. And as the
variance gets smaller the weight gets bigger.
So we will end up giving greater weight to bigger
study. Bigger study, tighter confidence interval. This gets
smaller, this gets smaller, this gets smaller, this gets
bigger. Bigger studies get greater weight by weighting on the
confidence interval. That's what we do in meta-analysis,
that's how we weight studies. So we'll do that for each
individual study we calculate the standard error using that
equation I showed you. We have the confidence interval. We do
that, we calculate the standard error for each individual
And then we use that equation then, calculate the
variance and calculate the weight. Nice and simple. Simple
equations. Nothing is complicated here. You may ask about
that 3.92. It turns out if you look at the equations on how to
calculate a confidence interval for rate ratio, odds ratio, you
had the 99 percent confidence interval. Do you remember the
1.96 from your stats class. That's the Z-score for the P value
of 1.05. That's that double. If you look at the equations you
end up doubling that when you calculate the standard error.
That's where the 3.92 comes from. You guys don't remember
that? Do you remember that Art? 1.96. All right. Okay.
Again, look and see what happened. We had the Reynolds
study which has a pretty tight confidence interval. It's
pretty narrow. It was a big study. I can't remember how many
people were in it. It was a big study. Compare that to the
Sandler study where the confidence interval is wide. That was
a smaller study. We end up giving greater weight to the bigger
study. This is greater weight than the Sandler study.
Okay. That's what we want it to do. And so what we
can do then is take those weights, multiply each weight times
each coefficient and then divide that by the sum of the
weights. Just add up all the weights, divide that by the sum
of the weights and we get an overall summary coefficient. We
summarized all these. We calculated a weighted coefficient.
This isn't 0. It should be past the decimal point. We
summarize -- we calculated basically weighted average, weighted
by the precision of each study where we got the precision based
on the confidence interval.
Okay. Remember what I said we did everything in the
log scale. This number doesn't really mean all that much to me
or probably to most people. Well, okay, this summary
coefficient .32 what does that mean? Let's take it out of the
log scale. Put it back on the relative risk scale by taking
the exponential. It correlates to a relative risk of 1.37. I
think if you go back and look at those data and in your own
mind you kind of average those data, look at the confidence
interval, see which are the bigger studies.
You'll probably come up with a number right about here
in your own mind. So, we, this is our overall weighted
average, again weighted by the precision of each individual
Art Reingold: Craig can you go back to the slide
with relative risk?
Craig Steinmaus: Yeah. In your mind you can tell
which are the bigger studies because they have the smaller
confidence intervals. This is a big study.
Art Reingold: If I were looking at that in my
mind I would come up with something bigger.
Craig Steinmaus: I probably would too. These are
smaller studies. That one did. A lot of these other ones.
This 1, 3.6. That didn't get a whole lot of weight. It got
7.4 percent of the total weight. Oops. I'm sorry. Where my
percent weight? I didn't put it on here. Look at Moriaba, it
didn't get a lot of weight compared to some of the others.
This one got a lot of weight and the relative risk was 1.1.
That's a good point. I think I've been doing this enough I
would have come up with about a 1.37 (laughter). Again, if you
haven't maybe you see all the twos.
Art Reingold: I look at all the twos and threes
and seven. I would come up with a bigger number.
Craig Steinmaus: All the more reason to not mix
everything up in your brain. Go ahead and quantify this.
And then you don't need to know this. This isn't going
to be in the midterm these equations.
Art Reingold: No.
Craig Steinmaus: But you can get a 95 percent
confidence interval. It's basically based on the sum of the
waits. And these equations here. You have all this
information already. It's nothing fancy so you get a
confidence interval. We have a 1.37 statistically significant
All right. Now you can also calculate a heterogeneity
statistic. Have you talked about consistency? You'll have a
lecture I believe on causal inference where you'll talk about
what are the things you look for to decide whether there's an
association. One of the things is do all the studies give you
the same result? Do all the studies give you a relative risk
of around two or is there a lot of spread in the data?
And that's a normal step of meta-analysis is we can do
that quantitatively using this heterogeneity statistic. And
again, you have this information. You have the coefficient for
each individual study. You have the summary coefficient. You
have the weights. So this is easy to calculate. And you can
even once you get a chi-square in whatever computer program you
are using if you are using Excel or STATA you can get a P value
for this. You can get statistically significant heterogeneity
or not. It does measure how much spread in the data that you
have. Are these really far away from this? Are these all over
the place? A lot really high or really low? Or are they
really close? Is this two and all these about two. You can
see how this does give you an indication of how spread out the
data are?
>>>: Quick question. You may have already
mentioned this. Is there a confidence interval associated with
the relative risk that you got the 1.36 for the entire
Craig Steinmaus: Yeah. That's this.
>>>: Okay.
Craig Steinmaus: Yeah. That's the confidence
interval for that. All right. That's a heterogeneity.
Now, quality, not all studies are created equal. We
don't really want to do the quality scores like I told you
because there's some difficulties there. There's other ways of
assessing quality. By quality I mean is there potential bias?
Is there potential confounding? There's ways of doing that.
I'll just give you some examples. One way of doing that is
subgroup analysis. All right? And so this is an example where
these are the studies I just showed you. Some of the studies
had good exposure assessment and some didn't. Some assessed
all the major ways you can get exposed to environmental tobacco
smoke. At work, at home as a child and at home as an adult.
Those are the three major ways. Some of these studies assessed
each one of those, I mean all three of them. And some of the
studies did not. They only assessed one and they ignored all
the other. Clearly you probably want to assess all three. If
you are going to have a good idea of a person's true exposure
to secondhand smoke you probably want to assess all three.
Some did and some didn't. Mark wanted to assess what's the
affect of the quality indicator on my meta-analysis. You can
do a subgroup analysis. That means do one group analysis with
all the studies. And do a separate meta-analysis where you
only include these that did a good job of exposure assessment.
Throw out the bad ones. That's a subgroup analysis. He did
that and here's what we found. Studies that use all three
sources he got a much higher relative risk. Better data, you
get a stronger effect. That's one way of assessing quality,
doing these subgroup analysis.
You can do that for a whole bunch of exposure
classifications. Here's one done by my colleagues. Diesel
exhaust in lung cancer. They had all these studies and came up
with an overall relative risk, statistically significant. Some
of the studies included data on smoking and some didn't. It's
a study of lung cancer. You worry about confounding from
smoking. What they did was they did one meta subgroup analysis
that only included smoking adjusted studies and a separate one
where they didn't give any data on smoking to see did it make a
difference? Were the smoking adjusted studies better or worse?
Was the relative risk higher or lower? Here's the results.
You can see relative risk 1.5 here. Overall 1.33. This is the
confidence interval lower and upper. For smoking adjusted
studies you get a relative risk of about 1.3. And for
unadjusted studies you get about the same thing. It didn't
make a difference.
This is evidence, again, it doesn't prove anything.
It's pretty strong evidence that adjusting for smoking doesn't
make a difference in diesel and lung cancer studies. That
smoking was not a major confounder in these studies.
Does anybody know why smoking wasn't a major
confounder? It causes lung cancer. You would think people
that did these dirty jobs that had a lot of diesel smoke, they
were blue collar workers. Maybe they would smoke more. Any
idea why it didn't cause major confounding? A lot of these
studies were done in the 50s and 60s and back then everybody
smoked. It didn't matter if you were white collar, blue
collar, rich, poor. Everybody smoked. It wasn't associated
with diesel exhaust. People in the office smoked, people in
the diesel truck yards smoked. This is a meta-analysis we just
completed. Keep this confidential. We just completed this.
It's on chromium six and does it cause stomach cancer.
Chromium six is a well known cause of lung cancer. That's
important because we're passing regulations in our state based
on chromium six causes stomach cancer in animals. We want to
know does it cause stomach cancer in humans. We reviewed all
the different studies. You can see we had 71 different
studies. Different job types exposed to chromium six. You can
see we got an overall effect of 1.32. I'm not sure why all my
examples are 1.3 something. Then we did these different
subgroup analysis based on what work you did. Chromium six is
a known cause of lung cancer. We know that. We separated out
those studies that also found elevated lung cancer and those
studies that didn't where they had data on lung cancer but it
wasn't elevated. We did separate meta-analysis of those
studies. We found the studies that had positive lung cancer
findings also has higher relative risk for stomach cancer. We
figured if a study was good enough to find an association that
we know it was probably a good study. And it probably had high
chromium exposures.
So basically what we're doing is we're finding a higher
association we know are good and found a known association
versus those that weren't. It's higher here than here. We
also did this. We figured maybe, I have to be careful with
this. Because I work for industry too. I consult for
We figured that maybe if you are funded by the chromium
industry you might tend not to find something. If you work in
academia, maybe you tend to find something. We did studies
that were funded by industry and whoa, they didn't find much.
Not statistically significant. Versus studies done by
academia. And again, it found something.
So this gives us evidence that maybe, potentially
there's some bias here. Right? Now, again don't get me wrong.
I'm not saying all industry funded studies are wrong because I
work for industry too. But there you go.
All right. Let's skip this one. Just another example.
And I want to show you this one just because this BCG
vaccine decrease leprosy. Art sends me all the best students.
It's great. It's not just chemical exposure. It's also
infectious disease you can do meta-analysis. Does BCG vaccine
decrease leprosy. It's for tuberculosis.
So it's not all good. I gave you all the good stuff
here. You can do meta-analysis on a lot of different stuff.
Summarize literature, look at confounding, look at exposure
misclassification. All that different stuff. It's not all
good. There's a lot of problems. Sometimes studies don't give
you all the data you need. Some studies will give you relative
risk but won't give you a confidence interval. Some studies
will say we didn't find anything and won't give you the
There's also this issue of publication bias.
Publication bias is the tendency of journals to publish results
that are statistically significant.
And not to publish results that are negative.
All right. That's publication bias. And it turns out
it actually is true. It actually is out there. There does
seem to be this tendency for journals to publish results that
are statistically significant. This is perhaps the earliest
study. This is major journals in psychology at the time. It
was one of the classical studies at the time. They took every
single study in each one of these journals over an entire year
and they saw how many studies gave a test of statistical
significance, a P value. How many studies gave a P value and
how many of the P values were less than 0.05. How many results
were statistically significant? You can see in these journals
97 percent. 95 percent of higher of all results were
statistically significant. That's huge. Maybe researchers are
just really good at deciding to study things where they are
going to find an effect. Maybe there's this publication bias
where only positive studies, only journals tend to report
positive studies. There's been subsequent studies like this.
It's not nearly as bad as it used to be. I don't think we've
still sort of solved the problem of publication bias. Yes.
>>>: I'm trying to get a sense of how bad this is
and sort of what implications you could draw from it? I look
at that. You know, those percentages are greater than the
95 percent confidence interval. So isn't that telling you the
studies you get that are showing, you have a likelihood that,
good likelihood the studies are showing 95 confidence are
happening just by chance and they are happening just by chance
with the same likelihood they are happening because there's an
actual association there.
Craig Steinmaus: I think what you are getting at
if the journals only have positive studies and you only use
journal articling for your meta-analysis and you only use
journal articles for your meta-analysis it's always going to be
>>>: Does that you couldn't make the call that a
study that shows statistical significance is due to anything
other than chance alone?
Craig Steinmaus: I think what it says is it says
that you are getting, if you just go to the journals you are
getting a biased selection of studies a lot of time. You are
only getting those positive studies. Those negative studies
are out there and they are not being published. It's not the
editor's fault. It's not the editors saying well I have ten
positive studies and ten negative studies. I'm only going to
publish the positive ones. It turns out it's the researcher's
fault. You have two studies. One positive and one negative.
The negative one will sit on your desk for a while. I'll
publish it later. You never get to it. Researchers tend not
to publish their negative results. That's the cause of this.
Again, if you are only publishing positive results your
meta-analysis is always going to be positive based on that.
There's ways of getting around this or not so much
getting around this. There's ways of assessing how bad this
is. If anybody is going to do a meta-analysis come to me and
we can talk about those ways. There's funnel plots. There's
statistical tests you can look at. There's a whole variety of
ways you can see not necessarily correct for it but to see how
big of a problem it is. Remember, that's epidemiology. You
are not looking at whether there's a bias or whether there's
confounding. There's always bias and there's always
confounding. What you are looking at is how big is it. And
there are statistical tests and other ways you can see if it's
a big problem or a little problem and it's different for
different topics. Some topics it's a huge problem and others
it's not a big problem. There's ways to check that out. I
don't have time to talk about each one.
Two minutes. Let me skip this and I'll get to this.
Which is there's a couple of good programs. I showed
you the mathematical calculations. You don't have to do those
by hand. There's a couple of programs that can do it. Perhaps
the two best are STATA and this one that was named after me,
Craigs program. I have an Excel spreadsheet that you plug in
the relative risk and confidence interval. All the numbers you
want pop up. If you are comfortable with STATA and not so
comfortable with Excel you can do it in STATA. STATA is good
for meta-analysis. You can supposedly do it in SAS but it's
Any questions? About any of that. Yes.
>>>: I came in a few minutes late. Would you
mind defining the BI coefficient?
Craig Steinmaus: It's the log of the relative
risk. But again you don't need to worry too much about that.
Remember, we take it back out of the log later on. Any other
questions? Okay. Free to go (applause).