Uploaded by UCBerkeley on 19.10.2012

Transcript:

Art Reingold: Okay. So, first of all in case

people have lost track, again a reminder next Wednesday is the

midterm and you'll need a calculator. This morning we're going

to have a presentation that often we have a little later in the

semester. It's a critically important presentation. Professor

Steinmaus has been here before talking about cross sectional

studies and historical cohort studies. One of the things that

I think we'll be addressing in a little more detail later in

the semester is once we have a body of evidence concerning a

particular exposure outcome relationship, smoking and lung

cancer, whatever it is we need to think about how to gather

that evidence together in some sort of systematic way and

figure out what we think it all means taken collectively. And

generally there are multiple studies about a particular subject

and how do we go about bringing that information together.

One approach in the old days was what might be called

systematic reviews. Another approach that has a lot of

traction these days is what's called meta-analysis and

Professor Steinmaus is certainly one of our local experts on

meta-analysis. In fact he and Professor Smith give a course

each fall on meta-analysis and systematic reviews and if you

are interested in this topic next fall you may want to

investigate taking that course. It's really quite valuable for

those of you who are going to be working in public health. In

any event Craig is just going to give you a taste this morning

about what meta-analysis is all about when we have multiple

studies of the same exposure outcome relationship.

Craig Steinmaus: Okay. Can we start? Can you

hear me in the back? All right. So yeah we'll talk about

meta-analysis today. Unless anybody wants to talk about

somebody else? No? Meta-analysis. Okay. If you're asking

for it we'll talk about it. All right.

So yeah, what Art said. Sometimes on a particular

topic that you're interested you have a whole bunch of studies

and you want to somehow summarize those studies so

meta-analysis is one way of summarizing those studies. Taking

a whole bunch of studies and coming up with an overall summary.

Why do you need to know about meta-analysis? Number one

there's a lot of them out there. On any particular topic

you'll do your Pub Med search and you'll come across

meta-analysis. When I started doing this when I think I went

on Pub Med this is ten years ago there weren't that many. Over

time it seems to grow and grow and grow. This is 2011. 51000

when you type meta-analysis. I'm sure today it's close to 60

or 70000. Meta-analysis is a whole bunch of different topics.

Occupational medicine, environmental medicine, infectious

diseases, clinical trials, genetics. Cohort studies, case

control studies and you combine studies that present data as

relative risk or difference between means or regression

coefficients. It incorporates a whole bunch of different types

of studies where you can summarize those literature or that

literature.

Okay. So first question is what are some of the

general ways we have summarized literature in the past? I'm

going to give you an example. The guy at the office next to me

is a pediatrician. His name is Mark miller. His boss came up

to him and said Mark, does environmental tobacco smoke or

secondhand tobacco smoke does that increase the risk of breast

cancer. My office at Cal EPA wanted to know that. We wanted

to know if we should push regulations to limit secondhand

tobacco smoke. Mark had to take a large body of literature on

this topic and summarize it to come up with, yes it does seem

to cause it, no it does not seem to cause it. I don't know is

not a great answer. Right?

Because I don't know means well, you don't do anything

about it. Right?

In epidemiology it's an imperfect science. There's a

lot of times where we can't really prove this causes this. We

don't have absolute 100 percent proof. So what we try to do is

come up with well, what's the general consensus? Is there

enough that we should probably be safe and go ahead and

regulate something like environmental tobacco smoke.

So Mark was asked that question. He went through the

usual steps of summarizing the literature research. Find the

research question. Do your literature search to look for all

the literature you can find on this particular topic. Evaluate

the studies and that's what you are learning in this class to

evaluate the studies. Look at bias, confounding, consistency,

causal inference, look at all that different stuff.

And then take all that stuff that you just did and sort

of somehow summarize it. So that's what we're going to talk

about today.

Now one thing I did want to point out that keeps coming

up in the meta-analysis class is the literature search. I

always get asked what's the best way to get studies where

should we get studies? If I was going to give you a one word

answer, Pub Med is the greatest. I think we've published 12,

13 meta-analysis and I hate to admit this but every single

study that we've ever found that we've ever put into a

meta-analysis are all on Pub Med. Saying that you should

always double check any work you do in epidemiology. Always

double check it. So we always look at another. Try to look at

some other database. So what I did was I actually went on

these major environmental journals and looked up meta-analysis

and looked how they found their articles they ended up

selecting for their meta-analysis. Every single article did

use Pub Med. You can see some other ones. Web of science.

I'm not an expert in those. That's where people generally seem

to do their literature searches in published meta-analysis.

Again, we've defined our research question. We've

gotten, we've done our literature search. We've gotten all the

studies on our particular topic and then again the next step,

you want to evaluate each study. Is there major bias is there

confounding? What's the probabilities due to chance? We're on

semesters right? You are, that's a big giant job right there.

Evaluating a study. That's not what I'm talking about today.

That's all your other classes. Evaluating each study and

deciding what are the good studies and what are the studies

that aren't good? What are the studies going to give you

information on your question and what aren't? What have fatal

flaws. Significant bias, significant confounding. We can push

those to the side.

And then when we're doing this whole process we do want

to set up inclusion and exclusion criteria on what types of

studies we're going to include. That's based on knowledge of

your topic and what the important biases are that could affect

the studies on your topic and the important confounding

variables that could affect the studies on your topic. You

want to set up really strict and formal inclusion exclusion

criteria. I'm going to include these types of studies. Now I

can't give you a whole lot of information on this because it

varies from topic to topic to topic. But the bottom line is

you want to have a good reason for everything that you do if

you are going to exclude a study you want to have good reason

for it and you want to avoid vague criteria. You don't want to

say I'm only going to include good studies. You want to be

specific. This is my first meta-analysis where we looked at

the effect of beta carotene and whether it decreased bladder

cancer. You can see our inclusion criteria. We weren't going

to include cross sectional studies or ecological studies but we

had a reason for each one of those. In the meta-analysis

process you'll do the same thing. And again it's all dependent

on what your topic is. Maybe some topics, cross sectional

studies are okay, some topic they aren't. Strict formal

exclusion inclusion criteria.

Again, that's based on your topic and evaluation of

bias, looking at important confounders. Eventually you'll

reach the point where you'll get what you have decided are the

good studies on your topic. The studies that give you

information you need or want and there's no fatal flaw in terms

of significant or substantial bias. Mark came up with these

particular studies. He did environmental tobacco smoke and

breast cancer in post and premenopausal Wyoming it's thought

that with breast cancer there are two different diseases. The

things that cause breast cancer before you have menopause are

probably different than after menopause. That's why he split

it up.

Anyway he found all these particular epidemiology

studies on this particular topic. Here's the relative risk and

confidence intervals. Some cohort studies, some case control

studies, CC. We'll talk about this in a few minutes.

So those, and he went through that whole process,

evaluating was there major confounding, was there major bias in

these particular studies and he had evaluated each one of those

and decided there wasn't. But these would give him good

information on that particular topic. Okay. Once he's got

these studies then how does he come up with an overall yes or

no.

While I'm talking, while I'm blabbing on go ahead and

take a look at these numbers here and these numbers here and

take my word for it that these did not most likely have major

bias, major confounding, but they were pretty good studies and

in your mind kind of come up with an overall conclusion if you

had this data here knowing they were good studies would you say

yes there's an association or no there's not an association.

Okay. And that's one way of summarizing literature. It's just

kind of looking at the data, mixing everything up in your

brain, right? And then out comes an answer. A yes or no. Yes

it does seem like there's an association or no there doesn't.

Right?

But that's not a very quantitative process. Right?

That's a subjective review. Let me emphasize that word

subjective. Different people's brains my evaluate things

differently. You would like to know how somebody's brain did

evaluate something. If they took it all in and spit something

out, a yes or no that's not really science.

We want to quantify things a little bit more than that

than these subjective reviews. That's the goal of

meta-analysis. There's also expert panels. You don't rely on

one guy and his brain to spit out an answer. Maybe you get a

whole bunch of experts including people from UC Berkeley. Get

a whole bunch of experts to sit together in a room and they go

over the data and all of them eventually at the end of the day

they bolt. This was on formaldehyde. Is it a carcinogen?

They looked at all the literature and they eventually at the

end of the day raise their hand for yes or no. That's a little

subjective too. It depends who you get in the committee and

what their tendencies seem to be. Again, that's somewhat

subjective as well. Meta-analysis we want to quantify things a

bit more. Another method is boat tallying. Count the number

of number of positive and negative studies. If there's more

positive studies then yes there's an association.

So you could do something like that. This is a lot of

times in the past was done based on statistical significance.

For each study that is statistically significant in red, found

a statistically significant effect, statistically significant

means it excludes one, that would be a positive study. Each

study that did not find a statistically significant effect that

would be in the black. That would be a negative study. If you

went through these studies you would find what is it? 7 and 7.

Seven positive studies and seven negative studies. It's a

wash. No association. That's vote counting. Counting each

study based on its statistical significance. I'm sure most of

you can see a whole bunch of problems with that. One is you'll

have studies like this, the Delfino study they did find almost

a threefold increased risk of breast cancer in people exposed

to secondhand smoke. That was negative because the confidence

interval included one. But the question is, is that really a

negative study? They found almost a threefold association.

But just because it was small, probably, the confidence

interval included one.

So, is it really negative or was it just -- or was it

positive and just too small? It didn't have enough statistical

power or subjects. Maybe if it was a little bigger that

confidence interval would have been a little narrower. I think

that's a general principle you'll learn in stats. As the

studies get bigger the confidence intervals get narrower.

Again, is that really a negative study?

That's one of the problems with vote tallying. That's

one of the problem that meta-analysis tries to deal with.

Another thing you could do is take an average result and see if

that average result is greater than one. You could take an

average of all these. Again, there's some problem with this.

Not all studies are created equal. Some studies, for example,

are bigger than others.

So wouldn't it make sense if a study, we have one study

with 10000 people. We have a is very similar study and it only

has three people. Wouldn't you want to give greater weight to

that study with 10000 people? Probably. I would think.

Everything else being equal.

So just taking an average you don't really do that.

And maybe it would be nice to do that.

All right. Another thing is that well, all studies are

not created equal. And we could maybe give studies with

greater quality more weight. For example, we could say and

never say it in my class because I do case control studies. We

could say that cohort studies are better than case control

studies. We could say that.

Right? And we could say, okay, well every case control

study we're only going to give half the weight we would give a

cohort study. We could take each result, weigh it by how good

a quality we think it is and then do a weighted average. A

weighted average. But the problem is that, okay, I just said

that cohort studies are better than case control studies.

Sometimes they are, sometimes they're not. Right? It

all depends on the study. You could have the absolutely worst

terrible cohort study that was ever done. Tons of bias. Only

one subject. Absolutely terrible and you're going to still

give it two times as much weight as the world's greatest case

control study. Right?

So, the sort of blanket statements about study quality,

it's difficult to support those statements. So it's difficult

to weight things on study quality. Based on factors like this.

Case control versus a cohort. Then maybe you really do believe

case control studies are better than cohort studies? Why not

1.8? Why two? That's why most of us that do meta-analysis we

don't use these quality scores like this based on these

factors. Where do you come up with the number? A lot of times

there's so many exceptions to these sorts of ratings that they

are not usable.

What we do in most meta-analysis is we weight of study

size. And what I mean by study size is precision. I'll talk

about precision in a second here.

And we don't really weight on study quality except for

what we usually do, what most meta-analysis do. We've gone

through each of our criteria to rate whether a study gives us

the information we want or doesn't. We've assessed is there

major bias? Is there major confounding? If there is we do

weighting. That weight is 0. If there isn't a fatal flaw we

give it a weight of one. We exclude our bad studies. We are

weighting but it's 1 and 0.

There's other ways of incorporating information or

looking at issues of study quality and that's subgroup

analysis. I'll talk about those in a few minutes. The bottom

line is this is how most meta-analysis is done. We take the

studies that are good enough, combine them and weight them on

study size or precision.

All right. So, all right. We're going to go through

the mathematical portion of a meta-analysis. This is all your

other classes. Let's go through this right here, this

mathematical portion. We have our good studies. And we've

selected them. We have our relative risk. We have our

confidence intervals. When you do meta-analysis you put

everything on the log scale.

I don't want to get too into this. It's basically

because the log scale is normally distributed. Whereas the

relative risk scale is not normally distributed. If you don't

get that and you are curious, come ask me. If you don't get it

and you are not curious it's not going to affect anything in

your life. We'll take it out of log scale later. Everything

goes on log scale. All the relative risk, all the confidence

intervals we put it on the log scale. Simple. We calculate

this BI. Let's call it a coefficient. Basically this is the

log of the relative risk. Nice and simple. You can do this in

Excel.

Or on a calculator or anywhere else. All right. And

again take the log of our relative risk and take the logs of

our confidence intervals. That's what that is, log, log, log.

Nice and simple.

All right. And then we want to weight each one of

these, each study based on our study size. But what is study

size? Is it the number of subjects in the study? Is that

study size?

There's a problem with that. Where you could run into

situations like this. You could have two studies that exactly

the same number of people in them. 200000 people. But this

study only has one case. Right? Versus this study has a lot

more cases. Now can you really, people with the disease,

people with the out come. Can you really tell much about

what's going on with the disease if you only have one person

with that disease in your entire study? Probably not. You are

probably going to get a lot more information out of this study.

So you can see there's potential problems with just weighting

the individual studies based on the number of total subjects.

And so, and also did you talk about statistical power

in this class?

Art Reingold: Later in the semester.

Craig Steinmaus: You'll learn about statistical

power, how much power your study has to identify an effect.

That is really driven not all that much by this, but

that's really driven by how many people with your outcome that

you have. This is really the driving factor.

So maybe we want to weight our studies based on the

number of people in our study with the outcome of interest.

But there's potential problem with that. We could run into a

scenario like this. You know, where we have studies with the

same number of people with the disease but we only have one non

case here. Again, can you really get a whole lot of

information about a disease if you only have one person in the

comparison group. Not much.

Not much. I think you can see you'd probably want to

give this study more weight. We can't really weight based on

the number of people with the outcome of interest. Or even the

total sample size. There's problems there. So what do we do?

It turns out we weight by the variance of the relative

risk. When you get the variance of the relative risk that

incorporates all that stuff. It incorporates the study size,

it incorporates the distribution of the cases and the controls.

So the study that gives you the most information that's

most precise information will have the lowest variance. Okay?

Big studies have low variance. All right? And by big I mean

that not only the total study size but that distribution of

cases and controls.

So we actually do end up weighting by the variance. Or

I should say the inverse of the variance. Does that help you?

It doesn't help you that much. What's the variance of the

relative risk? There's a whole bunch of different equations to

calculate the variance. The bottom line is the variance

incorporates the study size, big studies have smaller variance

and it incorporates that distribution. There's a whole bunch

of different equations for variance. This is one. I think

this is an estimate of the variance. Look at this equation for

a case control study to calculate the variance. The A, B, C, D

in your two by two table boxes. What would happen to the

variance in A, B, C and D are really big? Right? They are in

the denominator. If they are really big the variance is going

to drop. You can sort of see how study size is related to the

variance. Bigger studies, lower variance. That still doesn't

help you, does it? How do you get the variance?

It turns out we can get the variance of the relative

risk from the standard error. Great. That still doesn't help

you. How do you get the standard error? It turns out we can

get the standard error from the confidence interval and guess

what? We have the confidence interval. We can take the

confidence interval, calculate the standard error and then use

that, square it to calculate the variance. Look at this

equation here. And remember what I said, bigger studies the

confidence interval will get narrower.

So look what happens when you have a big study. What

happens to this quantity right here? It gets smaller. All

right? So big studies this quantity will get smaller. And as

this gets smaller this gets bigger. As this gets bigger this

gets bigger, this gets bigger. Did I say that right? No I

didn't say that right. Let's start all over again. Ready?

How come nobody raised their hand? You're not listening. As

this gets smaller this gets smaller. Right?

As this gets smaller this gets smaller. As this gets

smaller this gets smaller. For bigger studies. And as the

variance gets smaller the weight gets bigger.

So we will end up giving greater weight to bigger

study. Bigger study, tighter confidence interval. This gets

smaller, this gets smaller, this gets smaller, this gets

bigger. Bigger studies get greater weight by weighting on the

confidence interval. That's what we do in meta-analysis,

that's how we weight studies. So we'll do that for each

individual study we calculate the standard error using that

equation I showed you. We have the confidence interval. We do

that, we calculate the standard error for each individual

study.

And then we use that equation then, calculate the

variance and calculate the weight. Nice and simple. Simple

equations. Nothing is complicated here. You may ask about

that 3.92. It turns out if you look at the equations on how to

calculate a confidence interval for rate ratio, odds ratio, you

had the 99 percent confidence interval. Do you remember the

1.96 from your stats class. That's the Z-score for the P value

of 1.05. That's that double. If you look at the equations you

end up doubling that when you calculate the standard error.

That's where the 3.92 comes from. You guys don't remember

that? Do you remember that Art? 1.96. All right. Okay.

Again, look and see what happened. We had the Reynolds

study which has a pretty tight confidence interval. It's

pretty narrow. It was a big study. I can't remember how many

people were in it. It was a big study. Compare that to the

Sandler study where the confidence interval is wide. That was

a smaller study. We end up giving greater weight to the bigger

study. This is greater weight than the Sandler study.

Okay. That's what we want it to do. And so what we

can do then is take those weights, multiply each weight times

each coefficient and then divide that by the sum of the

weights. Just add up all the weights, divide that by the sum

of the weights and we get an overall summary coefficient. We

summarized all these. We calculated a weighted coefficient.

This isn't 0. It should be past the decimal point. We

summarize -- we calculated basically weighted average, weighted

by the precision of each study where we got the precision based

on the confidence interval.

Okay. Remember what I said we did everything in the

log scale. This number doesn't really mean all that much to me

or probably to most people. Well, okay, this summary

coefficient .32 what does that mean? Let's take it out of the

log scale. Put it back on the relative risk scale by taking

the exponential. It correlates to a relative risk of 1.37. I

think if you go back and look at those data and in your own

mind you kind of average those data, look at the confidence

interval, see which are the bigger studies.

You'll probably come up with a number right about here

in your own mind. So, we, this is our overall weighted

average, again weighted by the precision of each individual

study.

Art Reingold: Craig can you go back to the slide

with relative risk?

Craig Steinmaus: Yeah. In your mind you can tell

which are the bigger studies because they have the smaller

confidence intervals. This is a big study.

Art Reingold: If I were looking at that in my

mind I would come up with something bigger.

Craig Steinmaus: I probably would too. These are

smaller studies. That one did. A lot of these other ones.

This 1, 3.6. That didn't get a whole lot of weight. It got

7.4 percent of the total weight. Oops. I'm sorry. Where my

percent weight? I didn't put it on here. Look at Moriaba, it

didn't get a lot of weight compared to some of the others.

This one got a lot of weight and the relative risk was 1.1.

That's a good point. I think I've been doing this enough I

would have come up with about a 1.37 (laughter). Again, if you

haven't maybe you see all the twos.

Art Reingold: I look at all the twos and threes

and seven. I would come up with a bigger number.

Craig Steinmaus: All the more reason to not mix

everything up in your brain. Go ahead and quantify this.

And then you don't need to know this. This isn't going

to be in the midterm these equations.

Art Reingold: No.

Craig Steinmaus: But you can get a 95 percent

confidence interval. It's basically based on the sum of the

waits. And these equations here. You have all this

information already. It's nothing fancy so you get a

confidence interval. We have a 1.37 statistically significant

effect.

All right. Now you can also calculate a heterogeneity

statistic. Have you talked about consistency? You'll have a

lecture I believe on causal inference where you'll talk about

what are the things you look for to decide whether there's an

association. One of the things is do all the studies give you

the same result? Do all the studies give you a relative risk

of around two or is there a lot of spread in the data?

And that's a normal step of meta-analysis is we can do

that quantitatively using this heterogeneity statistic. And

again, you have this information. You have the coefficient for

each individual study. You have the summary coefficient. You

have the weights. So this is easy to calculate. And you can

even once you get a chi-square in whatever computer program you

are using if you are using Excel or STATA you can get a P value

for this. You can get statistically significant heterogeneity

or not. It does measure how much spread in the data that you

have. Are these really far away from this? Are these all over

the place? A lot really high or really low? Or are they

really close? Is this two and all these about two. You can

see how this does give you an indication of how spread out the

data are?

>>>: Quick question. You may have already

mentioned this. Is there a confidence interval associated with

the relative risk that you got the 1.36 for the entire

meta-analysis.

Craig Steinmaus: Yeah. That's this.

>>>: Okay.

Craig Steinmaus: Yeah. That's the confidence

interval for that. All right. That's a heterogeneity.

Now, quality, not all studies are created equal. We

don't really want to do the quality scores like I told you

because there's some difficulties there. There's other ways of

assessing quality. By quality I mean is there potential bias?

Is there potential confounding? There's ways of doing that.

I'll just give you some examples. One way of doing that is

subgroup analysis. All right? And so this is an example where

these are the studies I just showed you. Some of the studies

had good exposure assessment and some didn't. Some assessed

all the major ways you can get exposed to environmental tobacco

smoke. At work, at home as a child and at home as an adult.

Those are the three major ways. Some of these studies assessed

each one of those, I mean all three of them. And some of the

studies did not. They only assessed one and they ignored all

the other. Clearly you probably want to assess all three. If

you are going to have a good idea of a person's true exposure

to secondhand smoke you probably want to assess all three.

Some did and some didn't. Mark wanted to assess what's the

affect of the quality indicator on my meta-analysis. You can

do a subgroup analysis. That means do one group analysis with

all the studies. And do a separate meta-analysis where you

only include these that did a good job of exposure assessment.

Throw out the bad ones. That's a subgroup analysis. He did

that and here's what we found. Studies that use all three

sources he got a much higher relative risk. Better data, you

get a stronger effect. That's one way of assessing quality,

doing these subgroup analysis.

You can do that for a whole bunch of exposure

classifications. Here's one done by my colleagues. Diesel

exhaust in lung cancer. They had all these studies and came up

with an overall relative risk, statistically significant. Some

of the studies included data on smoking and some didn't. It's

a study of lung cancer. You worry about confounding from

smoking. What they did was they did one meta subgroup analysis

that only included smoking adjusted studies and a separate one

where they didn't give any data on smoking to see did it make a

difference? Were the smoking adjusted studies better or worse?

Was the relative risk higher or lower? Here's the results.

You can see relative risk 1.5 here. Overall 1.33. This is the

confidence interval lower and upper. For smoking adjusted

studies you get a relative risk of about 1.3. And for

unadjusted studies you get about the same thing. It didn't

make a difference.

This is evidence, again, it doesn't prove anything.

It's pretty strong evidence that adjusting for smoking doesn't

make a difference in diesel and lung cancer studies. That

smoking was not a major confounder in these studies.

Does anybody know why smoking wasn't a major

confounder? It causes lung cancer. You would think people

that did these dirty jobs that had a lot of diesel smoke, they

were blue collar workers. Maybe they would smoke more. Any

idea why it didn't cause major confounding? A lot of these

studies were done in the 50s and 60s and back then everybody

smoked. It didn't matter if you were white collar, blue

collar, rich, poor. Everybody smoked. It wasn't associated

with diesel exhaust. People in the office smoked, people in

the diesel truck yards smoked. This is a meta-analysis we just

completed. Keep this confidential. We just completed this.

It's on chromium six and does it cause stomach cancer.

Chromium six is a well known cause of lung cancer. That's

important because we're passing regulations in our state based

on chromium six causes stomach cancer in animals. We want to

know does it cause stomach cancer in humans. We reviewed all

the different studies. You can see we had 71 different

studies. Different job types exposed to chromium six. You can

see we got an overall effect of 1.32. I'm not sure why all my

examples are 1.3 something. Then we did these different

subgroup analysis based on what work you did. Chromium six is

a known cause of lung cancer. We know that. We separated out

those studies that also found elevated lung cancer and those

studies that didn't where they had data on lung cancer but it

wasn't elevated. We did separate meta-analysis of those

studies. We found the studies that had positive lung cancer

findings also has higher relative risk for stomach cancer. We

figured if a study was good enough to find an association that

we know it was probably a good study. And it probably had high

chromium exposures.

So basically what we're doing is we're finding a higher

association we know are good and found a known association

versus those that weren't. It's higher here than here. We

also did this. We figured maybe, I have to be careful with

this. Because I work for industry too. I consult for

industry.

We figured that maybe if you are funded by the chromium

industry you might tend not to find something. If you work in

academia, maybe you tend to find something. We did studies

that were funded by industry and whoa, they didn't find much.

Not statistically significant. Versus studies done by

academia. And again, it found something.

So this gives us evidence that maybe, potentially

there's some bias here. Right? Now, again don't get me wrong.

I'm not saying all industry funded studies are wrong because I

work for industry too. But there you go.

All right. Let's skip this one. Just another example.

And I want to show you this one just because this BCG

vaccine decrease leprosy. Art sends me all the best students.

It's great. It's not just chemical exposure. It's also

infectious disease you can do meta-analysis. Does BCG vaccine

decrease leprosy. It's for tuberculosis.

So it's not all good. I gave you all the good stuff

here. You can do meta-analysis on a lot of different stuff.

Summarize literature, look at confounding, look at exposure

misclassification. All that different stuff. It's not all

good. There's a lot of problems. Sometimes studies don't give

you all the data you need. Some studies will give you relative

risk but won't give you a confidence interval. Some studies

will say we didn't find anything and won't give you the

results.

There's also this issue of publication bias.

Publication bias is the tendency of journals to publish results

that are statistically significant.

And not to publish results that are negative.

All right. That's publication bias. And it turns out

it actually is true. It actually is out there. There does

seem to be this tendency for journals to publish results that

are statistically significant. This is perhaps the earliest

study. This is major journals in psychology at the time. It

was one of the classical studies at the time. They took every

single study in each one of these journals over an entire year

and they saw how many studies gave a test of statistical

significance, a P value. How many studies gave a P value and

how many of the P values were less than 0.05. How many results

were statistically significant? You can see in these journals

97 percent. 95 percent of higher of all results were

statistically significant. That's huge. Maybe researchers are

just really good at deciding to study things where they are

going to find an effect. Maybe there's this publication bias

where only positive studies, only journals tend to report

positive studies. There's been subsequent studies like this.

It's not nearly as bad as it used to be. I don't think we've

still sort of solved the problem of publication bias. Yes.

>>>: I'm trying to get a sense of how bad this is

and sort of what implications you could draw from it? I look

at that. You know, those percentages are greater than the

95 percent confidence interval. So isn't that telling you the

studies you get that are showing, you have a likelihood that,

good likelihood the studies are showing 95 confidence are

happening just by chance and they are happening just by chance

with the same likelihood they are happening because there's an

actual association there.

Craig Steinmaus: I think what you are getting at

if the journals only have positive studies and you only use

journal articling for your meta-analysis and you only use

journal articles for your meta-analysis it's always going to be

positive.

>>>: Does that you couldn't make the call that a

study that shows statistical significance is due to anything

other than chance alone?

Craig Steinmaus: I think what it says is it says

that you are getting, if you just go to the journals you are

getting a biased selection of studies a lot of time. You are

only getting those positive studies. Those negative studies

are out there and they are not being published. It's not the

editor's fault. It's not the editors saying well I have ten

positive studies and ten negative studies. I'm only going to

publish the positive ones. It turns out it's the researcher's

fault. You have two studies. One positive and one negative.

The negative one will sit on your desk for a while. I'll

publish it later. You never get to it. Researchers tend not

to publish their negative results. That's the cause of this.

Again, if you are only publishing positive results your

meta-analysis is always going to be positive based on that.

There's ways of getting around this or not so much

getting around this. There's ways of assessing how bad this

is. If anybody is going to do a meta-analysis come to me and

we can talk about those ways. There's funnel plots. There's

statistical tests you can look at. There's a whole variety of

ways you can see not necessarily correct for it but to see how

big of a problem it is. Remember, that's epidemiology. You

are not looking at whether there's a bias or whether there's

confounding. There's always bias and there's always

confounding. What you are looking at is how big is it. And

there are statistical tests and other ways you can see if it's

a big problem or a little problem and it's different for

different topics. Some topics it's a huge problem and others

it's not a big problem. There's ways to check that out. I

don't have time to talk about each one.

Two minutes. Let me skip this and I'll get to this.

Which is there's a couple of good programs. I showed

you the mathematical calculations. You don't have to do those

by hand. There's a couple of programs that can do it. Perhaps

the two best are STATA and this one that was named after me,

Craigs program. I have an Excel spreadsheet that you plug in

the relative risk and confidence interval. All the numbers you

want pop up. If you are comfortable with STATA and not so

comfortable with Excel you can do it in STATA. STATA is good

for meta-analysis. You can supposedly do it in SAS but it's

miserable.

Any questions? About any of that. Yes.

>>>: I came in a few minutes late. Would you

mind defining the BI coefficient?

Craig Steinmaus: It's the log of the relative

risk. But again you don't need to worry too much about that.

Remember, we take it back out of the log later on. Any other

questions? Okay. Free to go (applause).

people have lost track, again a reminder next Wednesday is the

midterm and you'll need a calculator. This morning we're going

to have a presentation that often we have a little later in the

semester. It's a critically important presentation. Professor

Steinmaus has been here before talking about cross sectional

studies and historical cohort studies. One of the things that

I think we'll be addressing in a little more detail later in

the semester is once we have a body of evidence concerning a

particular exposure outcome relationship, smoking and lung

cancer, whatever it is we need to think about how to gather

that evidence together in some sort of systematic way and

figure out what we think it all means taken collectively. And

generally there are multiple studies about a particular subject

and how do we go about bringing that information together.

One approach in the old days was what might be called

systematic reviews. Another approach that has a lot of

traction these days is what's called meta-analysis and

Professor Steinmaus is certainly one of our local experts on

meta-analysis. In fact he and Professor Smith give a course

each fall on meta-analysis and systematic reviews and if you

are interested in this topic next fall you may want to

investigate taking that course. It's really quite valuable for

those of you who are going to be working in public health. In

any event Craig is just going to give you a taste this morning

about what meta-analysis is all about when we have multiple

studies of the same exposure outcome relationship.

Craig Steinmaus: Okay. Can we start? Can you

hear me in the back? All right. So yeah we'll talk about

meta-analysis today. Unless anybody wants to talk about

somebody else? No? Meta-analysis. Okay. If you're asking

for it we'll talk about it. All right.

So yeah, what Art said. Sometimes on a particular

topic that you're interested you have a whole bunch of studies

and you want to somehow summarize those studies so

meta-analysis is one way of summarizing those studies. Taking

a whole bunch of studies and coming up with an overall summary.

Why do you need to know about meta-analysis? Number one

there's a lot of them out there. On any particular topic

you'll do your Pub Med search and you'll come across

meta-analysis. When I started doing this when I think I went

on Pub Med this is ten years ago there weren't that many. Over

time it seems to grow and grow and grow. This is 2011. 51000

when you type meta-analysis. I'm sure today it's close to 60

or 70000. Meta-analysis is a whole bunch of different topics.

Occupational medicine, environmental medicine, infectious

diseases, clinical trials, genetics. Cohort studies, case

control studies and you combine studies that present data as

relative risk or difference between means or regression

coefficients. It incorporates a whole bunch of different types

of studies where you can summarize those literature or that

literature.

Okay. So first question is what are some of the

general ways we have summarized literature in the past? I'm

going to give you an example. The guy at the office next to me

is a pediatrician. His name is Mark miller. His boss came up

to him and said Mark, does environmental tobacco smoke or

secondhand tobacco smoke does that increase the risk of breast

cancer. My office at Cal EPA wanted to know that. We wanted

to know if we should push regulations to limit secondhand

tobacco smoke. Mark had to take a large body of literature on

this topic and summarize it to come up with, yes it does seem

to cause it, no it does not seem to cause it. I don't know is

not a great answer. Right?

Because I don't know means well, you don't do anything

about it. Right?

In epidemiology it's an imperfect science. There's a

lot of times where we can't really prove this causes this. We

don't have absolute 100 percent proof. So what we try to do is

come up with well, what's the general consensus? Is there

enough that we should probably be safe and go ahead and

regulate something like environmental tobacco smoke.

So Mark was asked that question. He went through the

usual steps of summarizing the literature research. Find the

research question. Do your literature search to look for all

the literature you can find on this particular topic. Evaluate

the studies and that's what you are learning in this class to

evaluate the studies. Look at bias, confounding, consistency,

causal inference, look at all that different stuff.

And then take all that stuff that you just did and sort

of somehow summarize it. So that's what we're going to talk

about today.

Now one thing I did want to point out that keeps coming

up in the meta-analysis class is the literature search. I

always get asked what's the best way to get studies where

should we get studies? If I was going to give you a one word

answer, Pub Med is the greatest. I think we've published 12,

13 meta-analysis and I hate to admit this but every single

study that we've ever found that we've ever put into a

meta-analysis are all on Pub Med. Saying that you should

always double check any work you do in epidemiology. Always

double check it. So we always look at another. Try to look at

some other database. So what I did was I actually went on

these major environmental journals and looked up meta-analysis

and looked how they found their articles they ended up

selecting for their meta-analysis. Every single article did

use Pub Med. You can see some other ones. Web of science.

I'm not an expert in those. That's where people generally seem

to do their literature searches in published meta-analysis.

Again, we've defined our research question. We've

gotten, we've done our literature search. We've gotten all the

studies on our particular topic and then again the next step,

you want to evaluate each study. Is there major bias is there

confounding? What's the probabilities due to chance? We're on

semesters right? You are, that's a big giant job right there.

Evaluating a study. That's not what I'm talking about today.

That's all your other classes. Evaluating each study and

deciding what are the good studies and what are the studies

that aren't good? What are the studies going to give you

information on your question and what aren't? What have fatal

flaws. Significant bias, significant confounding. We can push

those to the side.

And then when we're doing this whole process we do want

to set up inclusion and exclusion criteria on what types of

studies we're going to include. That's based on knowledge of

your topic and what the important biases are that could affect

the studies on your topic and the important confounding

variables that could affect the studies on your topic. You

want to set up really strict and formal inclusion exclusion

criteria. I'm going to include these types of studies. Now I

can't give you a whole lot of information on this because it

varies from topic to topic to topic. But the bottom line is

you want to have a good reason for everything that you do if

you are going to exclude a study you want to have good reason

for it and you want to avoid vague criteria. You don't want to

say I'm only going to include good studies. You want to be

specific. This is my first meta-analysis where we looked at

the effect of beta carotene and whether it decreased bladder

cancer. You can see our inclusion criteria. We weren't going

to include cross sectional studies or ecological studies but we

had a reason for each one of those. In the meta-analysis

process you'll do the same thing. And again it's all dependent

on what your topic is. Maybe some topics, cross sectional

studies are okay, some topic they aren't. Strict formal

exclusion inclusion criteria.

Again, that's based on your topic and evaluation of

bias, looking at important confounders. Eventually you'll

reach the point where you'll get what you have decided are the

good studies on your topic. The studies that give you

information you need or want and there's no fatal flaw in terms

of significant or substantial bias. Mark came up with these

particular studies. He did environmental tobacco smoke and

breast cancer in post and premenopausal Wyoming it's thought

that with breast cancer there are two different diseases. The

things that cause breast cancer before you have menopause are

probably different than after menopause. That's why he split

it up.

Anyway he found all these particular epidemiology

studies on this particular topic. Here's the relative risk and

confidence intervals. Some cohort studies, some case control

studies, CC. We'll talk about this in a few minutes.

So those, and he went through that whole process,

evaluating was there major confounding, was there major bias in

these particular studies and he had evaluated each one of those

and decided there wasn't. But these would give him good

information on that particular topic. Okay. Once he's got

these studies then how does he come up with an overall yes or

no.

While I'm talking, while I'm blabbing on go ahead and

take a look at these numbers here and these numbers here and

take my word for it that these did not most likely have major

bias, major confounding, but they were pretty good studies and

in your mind kind of come up with an overall conclusion if you

had this data here knowing they were good studies would you say

yes there's an association or no there's not an association.

Okay. And that's one way of summarizing literature. It's just

kind of looking at the data, mixing everything up in your

brain, right? And then out comes an answer. A yes or no. Yes

it does seem like there's an association or no there doesn't.

Right?

But that's not a very quantitative process. Right?

That's a subjective review. Let me emphasize that word

subjective. Different people's brains my evaluate things

differently. You would like to know how somebody's brain did

evaluate something. If they took it all in and spit something

out, a yes or no that's not really science.

We want to quantify things a little bit more than that

than these subjective reviews. That's the goal of

meta-analysis. There's also expert panels. You don't rely on

one guy and his brain to spit out an answer. Maybe you get a

whole bunch of experts including people from UC Berkeley. Get

a whole bunch of experts to sit together in a room and they go

over the data and all of them eventually at the end of the day

they bolt. This was on formaldehyde. Is it a carcinogen?

They looked at all the literature and they eventually at the

end of the day raise their hand for yes or no. That's a little

subjective too. It depends who you get in the committee and

what their tendencies seem to be. Again, that's somewhat

subjective as well. Meta-analysis we want to quantify things a

bit more. Another method is boat tallying. Count the number

of number of positive and negative studies. If there's more

positive studies then yes there's an association.

So you could do something like that. This is a lot of

times in the past was done based on statistical significance.

For each study that is statistically significant in red, found

a statistically significant effect, statistically significant

means it excludes one, that would be a positive study. Each

study that did not find a statistically significant effect that

would be in the black. That would be a negative study. If you

went through these studies you would find what is it? 7 and 7.

Seven positive studies and seven negative studies. It's a

wash. No association. That's vote counting. Counting each

study based on its statistical significance. I'm sure most of

you can see a whole bunch of problems with that. One is you'll

have studies like this, the Delfino study they did find almost

a threefold increased risk of breast cancer in people exposed

to secondhand smoke. That was negative because the confidence

interval included one. But the question is, is that really a

negative study? They found almost a threefold association.

But just because it was small, probably, the confidence

interval included one.

So, is it really negative or was it just -- or was it

positive and just too small? It didn't have enough statistical

power or subjects. Maybe if it was a little bigger that

confidence interval would have been a little narrower. I think

that's a general principle you'll learn in stats. As the

studies get bigger the confidence intervals get narrower.

Again, is that really a negative study?

That's one of the problems with vote tallying. That's

one of the problem that meta-analysis tries to deal with.

Another thing you could do is take an average result and see if

that average result is greater than one. You could take an

average of all these. Again, there's some problem with this.

Not all studies are created equal. Some studies, for example,

are bigger than others.

So wouldn't it make sense if a study, we have one study

with 10000 people. We have a is very similar study and it only

has three people. Wouldn't you want to give greater weight to

that study with 10000 people? Probably. I would think.

Everything else being equal.

So just taking an average you don't really do that.

And maybe it would be nice to do that.

All right. Another thing is that well, all studies are

not created equal. And we could maybe give studies with

greater quality more weight. For example, we could say and

never say it in my class because I do case control studies. We

could say that cohort studies are better than case control

studies. We could say that.

Right? And we could say, okay, well every case control

study we're only going to give half the weight we would give a

cohort study. We could take each result, weigh it by how good

a quality we think it is and then do a weighted average. A

weighted average. But the problem is that, okay, I just said

that cohort studies are better than case control studies.

Sometimes they are, sometimes they're not. Right? It

all depends on the study. You could have the absolutely worst

terrible cohort study that was ever done. Tons of bias. Only

one subject. Absolutely terrible and you're going to still

give it two times as much weight as the world's greatest case

control study. Right?

So, the sort of blanket statements about study quality,

it's difficult to support those statements. So it's difficult

to weight things on study quality. Based on factors like this.

Case control versus a cohort. Then maybe you really do believe

case control studies are better than cohort studies? Why not

1.8? Why two? That's why most of us that do meta-analysis we

don't use these quality scores like this based on these

factors. Where do you come up with the number? A lot of times

there's so many exceptions to these sorts of ratings that they

are not usable.

What we do in most meta-analysis is we weight of study

size. And what I mean by study size is precision. I'll talk

about precision in a second here.

And we don't really weight on study quality except for

what we usually do, what most meta-analysis do. We've gone

through each of our criteria to rate whether a study gives us

the information we want or doesn't. We've assessed is there

major bias? Is there major confounding? If there is we do

weighting. That weight is 0. If there isn't a fatal flaw we

give it a weight of one. We exclude our bad studies. We are

weighting but it's 1 and 0.

There's other ways of incorporating information or

looking at issues of study quality and that's subgroup

analysis. I'll talk about those in a few minutes. The bottom

line is this is how most meta-analysis is done. We take the

studies that are good enough, combine them and weight them on

study size or precision.

All right. So, all right. We're going to go through

the mathematical portion of a meta-analysis. This is all your

other classes. Let's go through this right here, this

mathematical portion. We have our good studies. And we've

selected them. We have our relative risk. We have our

confidence intervals. When you do meta-analysis you put

everything on the log scale.

I don't want to get too into this. It's basically

because the log scale is normally distributed. Whereas the

relative risk scale is not normally distributed. If you don't

get that and you are curious, come ask me. If you don't get it

and you are not curious it's not going to affect anything in

your life. We'll take it out of log scale later. Everything

goes on log scale. All the relative risk, all the confidence

intervals we put it on the log scale. Simple. We calculate

this BI. Let's call it a coefficient. Basically this is the

log of the relative risk. Nice and simple. You can do this in

Excel.

Or on a calculator or anywhere else. All right. And

again take the log of our relative risk and take the logs of

our confidence intervals. That's what that is, log, log, log.

Nice and simple.

All right. And then we want to weight each one of

these, each study based on our study size. But what is study

size? Is it the number of subjects in the study? Is that

study size?

There's a problem with that. Where you could run into

situations like this. You could have two studies that exactly

the same number of people in them. 200000 people. But this

study only has one case. Right? Versus this study has a lot

more cases. Now can you really, people with the disease,

people with the out come. Can you really tell much about

what's going on with the disease if you only have one person

with that disease in your entire study? Probably not. You are

probably going to get a lot more information out of this study.

So you can see there's potential problems with just weighting

the individual studies based on the number of total subjects.

And so, and also did you talk about statistical power

in this class?

Art Reingold: Later in the semester.

Craig Steinmaus: You'll learn about statistical

power, how much power your study has to identify an effect.

That is really driven not all that much by this, but

that's really driven by how many people with your outcome that

you have. This is really the driving factor.

So maybe we want to weight our studies based on the

number of people in our study with the outcome of interest.

But there's potential problem with that. We could run into a

scenario like this. You know, where we have studies with the

same number of people with the disease but we only have one non

case here. Again, can you really get a whole lot of

information about a disease if you only have one person in the

comparison group. Not much.

Not much. I think you can see you'd probably want to

give this study more weight. We can't really weight based on

the number of people with the outcome of interest. Or even the

total sample size. There's problems there. So what do we do?

It turns out we weight by the variance of the relative

risk. When you get the variance of the relative risk that

incorporates all that stuff. It incorporates the study size,

it incorporates the distribution of the cases and the controls.

So the study that gives you the most information that's

most precise information will have the lowest variance. Okay?

Big studies have low variance. All right? And by big I mean

that not only the total study size but that distribution of

cases and controls.

So we actually do end up weighting by the variance. Or

I should say the inverse of the variance. Does that help you?

It doesn't help you that much. What's the variance of the

relative risk? There's a whole bunch of different equations to

calculate the variance. The bottom line is the variance

incorporates the study size, big studies have smaller variance

and it incorporates that distribution. There's a whole bunch

of different equations for variance. This is one. I think

this is an estimate of the variance. Look at this equation for

a case control study to calculate the variance. The A, B, C, D

in your two by two table boxes. What would happen to the

variance in A, B, C and D are really big? Right? They are in

the denominator. If they are really big the variance is going

to drop. You can sort of see how study size is related to the

variance. Bigger studies, lower variance. That still doesn't

help you, does it? How do you get the variance?

It turns out we can get the variance of the relative

risk from the standard error. Great. That still doesn't help

you. How do you get the standard error? It turns out we can

get the standard error from the confidence interval and guess

what? We have the confidence interval. We can take the

confidence interval, calculate the standard error and then use

that, square it to calculate the variance. Look at this

equation here. And remember what I said, bigger studies the

confidence interval will get narrower.

So look what happens when you have a big study. What

happens to this quantity right here? It gets smaller. All

right? So big studies this quantity will get smaller. And as

this gets smaller this gets bigger. As this gets bigger this

gets bigger, this gets bigger. Did I say that right? No I

didn't say that right. Let's start all over again. Ready?

How come nobody raised their hand? You're not listening. As

this gets smaller this gets smaller. Right?

As this gets smaller this gets smaller. As this gets

smaller this gets smaller. For bigger studies. And as the

variance gets smaller the weight gets bigger.

So we will end up giving greater weight to bigger

study. Bigger study, tighter confidence interval. This gets

smaller, this gets smaller, this gets smaller, this gets

bigger. Bigger studies get greater weight by weighting on the

confidence interval. That's what we do in meta-analysis,

that's how we weight studies. So we'll do that for each

individual study we calculate the standard error using that

equation I showed you. We have the confidence interval. We do

that, we calculate the standard error for each individual

study.

And then we use that equation then, calculate the

variance and calculate the weight. Nice and simple. Simple

equations. Nothing is complicated here. You may ask about

that 3.92. It turns out if you look at the equations on how to

calculate a confidence interval for rate ratio, odds ratio, you

had the 99 percent confidence interval. Do you remember the

1.96 from your stats class. That's the Z-score for the P value

of 1.05. That's that double. If you look at the equations you

end up doubling that when you calculate the standard error.

That's where the 3.92 comes from. You guys don't remember

that? Do you remember that Art? 1.96. All right. Okay.

Again, look and see what happened. We had the Reynolds

study which has a pretty tight confidence interval. It's

pretty narrow. It was a big study. I can't remember how many

people were in it. It was a big study. Compare that to the

Sandler study where the confidence interval is wide. That was

a smaller study. We end up giving greater weight to the bigger

study. This is greater weight than the Sandler study.

Okay. That's what we want it to do. And so what we

can do then is take those weights, multiply each weight times

each coefficient and then divide that by the sum of the

weights. Just add up all the weights, divide that by the sum

of the weights and we get an overall summary coefficient. We

summarized all these. We calculated a weighted coefficient.

This isn't 0. It should be past the decimal point. We

summarize -- we calculated basically weighted average, weighted

by the precision of each study where we got the precision based

on the confidence interval.

Okay. Remember what I said we did everything in the

log scale. This number doesn't really mean all that much to me

or probably to most people. Well, okay, this summary

coefficient .32 what does that mean? Let's take it out of the

log scale. Put it back on the relative risk scale by taking

the exponential. It correlates to a relative risk of 1.37. I

think if you go back and look at those data and in your own

mind you kind of average those data, look at the confidence

interval, see which are the bigger studies.

You'll probably come up with a number right about here

in your own mind. So, we, this is our overall weighted

average, again weighted by the precision of each individual

study.

Art Reingold: Craig can you go back to the slide

with relative risk?

Craig Steinmaus: Yeah. In your mind you can tell

which are the bigger studies because they have the smaller

confidence intervals. This is a big study.

Art Reingold: If I were looking at that in my

mind I would come up with something bigger.

Craig Steinmaus: I probably would too. These are

smaller studies. That one did. A lot of these other ones.

This 1, 3.6. That didn't get a whole lot of weight. It got

7.4 percent of the total weight. Oops. I'm sorry. Where my

percent weight? I didn't put it on here. Look at Moriaba, it

didn't get a lot of weight compared to some of the others.

This one got a lot of weight and the relative risk was 1.1.

That's a good point. I think I've been doing this enough I

would have come up with about a 1.37 (laughter). Again, if you

haven't maybe you see all the twos.

Art Reingold: I look at all the twos and threes

and seven. I would come up with a bigger number.

Craig Steinmaus: All the more reason to not mix

everything up in your brain. Go ahead and quantify this.

And then you don't need to know this. This isn't going

to be in the midterm these equations.

Art Reingold: No.

Craig Steinmaus: But you can get a 95 percent

confidence interval. It's basically based on the sum of the

waits. And these equations here. You have all this

information already. It's nothing fancy so you get a

confidence interval. We have a 1.37 statistically significant

effect.

All right. Now you can also calculate a heterogeneity

statistic. Have you talked about consistency? You'll have a

lecture I believe on causal inference where you'll talk about

what are the things you look for to decide whether there's an

association. One of the things is do all the studies give you

the same result? Do all the studies give you a relative risk

of around two or is there a lot of spread in the data?

And that's a normal step of meta-analysis is we can do

that quantitatively using this heterogeneity statistic. And

again, you have this information. You have the coefficient for

each individual study. You have the summary coefficient. You

have the weights. So this is easy to calculate. And you can

even once you get a chi-square in whatever computer program you

are using if you are using Excel or STATA you can get a P value

for this. You can get statistically significant heterogeneity

or not. It does measure how much spread in the data that you

have. Are these really far away from this? Are these all over

the place? A lot really high or really low? Or are they

really close? Is this two and all these about two. You can

see how this does give you an indication of how spread out the

data are?

>>>: Quick question. You may have already

mentioned this. Is there a confidence interval associated with

the relative risk that you got the 1.36 for the entire

meta-analysis.

Craig Steinmaus: Yeah. That's this.

>>>: Okay.

Craig Steinmaus: Yeah. That's the confidence

interval for that. All right. That's a heterogeneity.

Now, quality, not all studies are created equal. We

don't really want to do the quality scores like I told you

because there's some difficulties there. There's other ways of

assessing quality. By quality I mean is there potential bias?

Is there potential confounding? There's ways of doing that.

I'll just give you some examples. One way of doing that is

subgroup analysis. All right? And so this is an example where

these are the studies I just showed you. Some of the studies

had good exposure assessment and some didn't. Some assessed

all the major ways you can get exposed to environmental tobacco

smoke. At work, at home as a child and at home as an adult.

Those are the three major ways. Some of these studies assessed

each one of those, I mean all three of them. And some of the

studies did not. They only assessed one and they ignored all

the other. Clearly you probably want to assess all three. If

you are going to have a good idea of a person's true exposure

to secondhand smoke you probably want to assess all three.

Some did and some didn't. Mark wanted to assess what's the

affect of the quality indicator on my meta-analysis. You can

do a subgroup analysis. That means do one group analysis with

all the studies. And do a separate meta-analysis where you

only include these that did a good job of exposure assessment.

Throw out the bad ones. That's a subgroup analysis. He did

that and here's what we found. Studies that use all three

sources he got a much higher relative risk. Better data, you

get a stronger effect. That's one way of assessing quality,

doing these subgroup analysis.

You can do that for a whole bunch of exposure

classifications. Here's one done by my colleagues. Diesel

exhaust in lung cancer. They had all these studies and came up

with an overall relative risk, statistically significant. Some

of the studies included data on smoking and some didn't. It's

a study of lung cancer. You worry about confounding from

smoking. What they did was they did one meta subgroup analysis

that only included smoking adjusted studies and a separate one

where they didn't give any data on smoking to see did it make a

difference? Were the smoking adjusted studies better or worse?

Was the relative risk higher or lower? Here's the results.

You can see relative risk 1.5 here. Overall 1.33. This is the

confidence interval lower and upper. For smoking adjusted

studies you get a relative risk of about 1.3. And for

unadjusted studies you get about the same thing. It didn't

make a difference.

This is evidence, again, it doesn't prove anything.

It's pretty strong evidence that adjusting for smoking doesn't

make a difference in diesel and lung cancer studies. That

smoking was not a major confounder in these studies.

Does anybody know why smoking wasn't a major

confounder? It causes lung cancer. You would think people

that did these dirty jobs that had a lot of diesel smoke, they

were blue collar workers. Maybe they would smoke more. Any

idea why it didn't cause major confounding? A lot of these

studies were done in the 50s and 60s and back then everybody

smoked. It didn't matter if you were white collar, blue

collar, rich, poor. Everybody smoked. It wasn't associated

with diesel exhaust. People in the office smoked, people in

the diesel truck yards smoked. This is a meta-analysis we just

completed. Keep this confidential. We just completed this.

It's on chromium six and does it cause stomach cancer.

Chromium six is a well known cause of lung cancer. That's

important because we're passing regulations in our state based

on chromium six causes stomach cancer in animals. We want to

know does it cause stomach cancer in humans. We reviewed all

the different studies. You can see we had 71 different

studies. Different job types exposed to chromium six. You can

see we got an overall effect of 1.32. I'm not sure why all my

examples are 1.3 something. Then we did these different

subgroup analysis based on what work you did. Chromium six is

a known cause of lung cancer. We know that. We separated out

those studies that also found elevated lung cancer and those

studies that didn't where they had data on lung cancer but it

wasn't elevated. We did separate meta-analysis of those

studies. We found the studies that had positive lung cancer

findings also has higher relative risk for stomach cancer. We

figured if a study was good enough to find an association that

we know it was probably a good study. And it probably had high

chromium exposures.

So basically what we're doing is we're finding a higher

association we know are good and found a known association

versus those that weren't. It's higher here than here. We

also did this. We figured maybe, I have to be careful with

this. Because I work for industry too. I consult for

industry.

We figured that maybe if you are funded by the chromium

industry you might tend not to find something. If you work in

academia, maybe you tend to find something. We did studies

that were funded by industry and whoa, they didn't find much.

Not statistically significant. Versus studies done by

academia. And again, it found something.

So this gives us evidence that maybe, potentially

there's some bias here. Right? Now, again don't get me wrong.

I'm not saying all industry funded studies are wrong because I

work for industry too. But there you go.

All right. Let's skip this one. Just another example.

And I want to show you this one just because this BCG

vaccine decrease leprosy. Art sends me all the best students.

It's great. It's not just chemical exposure. It's also

infectious disease you can do meta-analysis. Does BCG vaccine

decrease leprosy. It's for tuberculosis.

So it's not all good. I gave you all the good stuff

here. You can do meta-analysis on a lot of different stuff.

Summarize literature, look at confounding, look at exposure

misclassification. All that different stuff. It's not all

good. There's a lot of problems. Sometimes studies don't give

you all the data you need. Some studies will give you relative

risk but won't give you a confidence interval. Some studies

will say we didn't find anything and won't give you the

results.

There's also this issue of publication bias.

Publication bias is the tendency of journals to publish results

that are statistically significant.

And not to publish results that are negative.

All right. That's publication bias. And it turns out

it actually is true. It actually is out there. There does

seem to be this tendency for journals to publish results that

are statistically significant. This is perhaps the earliest

study. This is major journals in psychology at the time. It

was one of the classical studies at the time. They took every

single study in each one of these journals over an entire year

and they saw how many studies gave a test of statistical

significance, a P value. How many studies gave a P value and

how many of the P values were less than 0.05. How many results

were statistically significant? You can see in these journals

97 percent. 95 percent of higher of all results were

statistically significant. That's huge. Maybe researchers are

just really good at deciding to study things where they are

going to find an effect. Maybe there's this publication bias

where only positive studies, only journals tend to report

positive studies. There's been subsequent studies like this.

It's not nearly as bad as it used to be. I don't think we've

still sort of solved the problem of publication bias. Yes.

>>>: I'm trying to get a sense of how bad this is

and sort of what implications you could draw from it? I look

at that. You know, those percentages are greater than the

95 percent confidence interval. So isn't that telling you the

studies you get that are showing, you have a likelihood that,

good likelihood the studies are showing 95 confidence are

happening just by chance and they are happening just by chance

with the same likelihood they are happening because there's an

actual association there.

Craig Steinmaus: I think what you are getting at

if the journals only have positive studies and you only use

journal articling for your meta-analysis and you only use

journal articles for your meta-analysis it's always going to be

positive.

>>>: Does that you couldn't make the call that a

study that shows statistical significance is due to anything

other than chance alone?

Craig Steinmaus: I think what it says is it says

that you are getting, if you just go to the journals you are

getting a biased selection of studies a lot of time. You are

only getting those positive studies. Those negative studies

are out there and they are not being published. It's not the

editor's fault. It's not the editors saying well I have ten

positive studies and ten negative studies. I'm only going to

publish the positive ones. It turns out it's the researcher's

fault. You have two studies. One positive and one negative.

The negative one will sit on your desk for a while. I'll

publish it later. You never get to it. Researchers tend not

to publish their negative results. That's the cause of this.

Again, if you are only publishing positive results your

meta-analysis is always going to be positive based on that.

There's ways of getting around this or not so much

getting around this. There's ways of assessing how bad this

is. If anybody is going to do a meta-analysis come to me and

we can talk about those ways. There's funnel plots. There's

statistical tests you can look at. There's a whole variety of

ways you can see not necessarily correct for it but to see how

big of a problem it is. Remember, that's epidemiology. You

are not looking at whether there's a bias or whether there's

confounding. There's always bias and there's always

confounding. What you are looking at is how big is it. And

there are statistical tests and other ways you can see if it's

a big problem or a little problem and it's different for

different topics. Some topics it's a huge problem and others

it's not a big problem. There's ways to check that out. I

don't have time to talk about each one.

Two minutes. Let me skip this and I'll get to this.

Which is there's a couple of good programs. I showed

you the mathematical calculations. You don't have to do those

by hand. There's a couple of programs that can do it. Perhaps

the two best are STATA and this one that was named after me,

Craigs program. I have an Excel spreadsheet that you plug in

the relative risk and confidence interval. All the numbers you

want pop up. If you are comfortable with STATA and not so

comfortable with Excel you can do it in STATA. STATA is good

for meta-analysis. You can supposedly do it in SAS but it's

miserable.

Any questions? About any of that. Yes.

>>>: I came in a few minutes late. Would you

mind defining the BI coefficient?

Craig Steinmaus: It's the log of the relative

risk. But again you don't need to worry too much about that.

Remember, we take it back out of the log later on. Any other

questions? Okay. Free to go (applause).