Multilevel Interventions in Health Care Conference: Discussant comments by Brian Mittman, PhD

>>>DR. STEVEN CLAUSER: For our last part of this presentation
we do have Brian Mittman who will be our discussant.
Brian is Director of the Veteran's Affairs Center for
Implementation, Practice and Research Support, and is a
senior social scientist at the Veteran's Administration,
UCLA and Rand Center for the study of health care
provider behavior. His research interests include
implementation science, health care quality improvement and
health care management; and he received his
Ph.D. in organizational behavior from Stanford University's
graduate school of business.
>>>BRIAN MITTMAN: And despite BA's encrypted USB drives,
it often causes problems, I manage to get the slides up.
So a couple of disclosures. First of all, thanks to
Paul Cleary for helping me transfer the slides. There was
a bit of disagreement or a difference of opinion between
us. Some of the folks at NCI asked Marty as to
when his paper will be finished. I chose to listen to those who
told me it would be finished before the conference.
It wasn't until yesterday that I finally realized that I didn't
have it and I should find out. Marty's understanding was
that he was finishing after the conference.
So my comments build on the presentation that
I received in advance, but the full paper. Quick note on how I
went about reviewing these, but also an approach that I think
would be helpful for a final review and perhaps revisions of
some of the other papers as well. I began by quick review
of what the goals are of multi-level interventions and
multi-level intervention research. But I focused then on
the issue of what is distinctive about MLI versus
single level approaches. And perhaps more importantly for
the papers, especially these two that take existing material
and adapt it to the problem of multi-level interventions, what
is it that's distinctive about the world of multi-level
interventions both in terms of intervening as well as
implementing and spreading those interventions, and
conducting research to understand multi-level
intervention. So to me that's the key question. And therefore
the fourth point, which is do these manuscripts help us in
terms of our desire to design better multi-level
interventions, to implement and spread them, and of course to
understand their effectiveness. And as the case with any good
discussant or commentator, the focus is not on the compliments
and pointing out the good things about the papers, but
identifying the gaps, and as I would prefer to think about it,
offering what I hope are helpful recommendations for
strengthening. So beginning with the Charns paper in
alphabetical order, measures and measurement. The first
question, what is distinctive about MLI's relative to
measures and measurement. When we think about the end points
of these interventions, is there anything different at the
end of the causal chain in the multi-level world?
Probably not. So everything that we learn and we can bring in
from the field of measures and measurement in the single level
intervention world probably applies to us in the
multi-level interventions. Are there any differences when we
look at each different level of the process and the impacts?
Possibly some differences. So there may be some need for
adaptation and some new development. What about the
issue of a synergistic emergent patterns and phenomena,
interaction effects and so on? There I think the answer is
very clearly probably there are many things that are likely to
be different, and our approach to measuring and measures would
need to be different. And finally, causal processes and
mechanisms, especially cutting across levels. This is where
things do get interesting, where we are likely to need
significant amounts of new development. So this is just a
way of thinking about the issues, and I hope in the paper
providing some guidance to think about the problem of
taking existing measures and measurement approaches and
adapting them and bringing them into the multi-level world, and
where we need to focus our efforts. And that is on the
emergent kinds of processes and phenomena, and
understanding how to measure those processes. So what are
some of the other implications of these distinctions?
The differences that we see in the multi-level interventions?
Again, I would hope that the authors are able to provide
guidance for researchers and evaluators, as Marty indicated
in one of the final slides, identify and develop a theory
or logic or program model, document very clearly the
causal chain, and identify and adopt measurement approaches
that would document that causal chain and allow us to
understand how things play out. There are many opportunities
in interventional studies, RCT's, for things to go wrong.
And understanding how to measure and monitor so that we
understand why the endpoint outcomes were not as we
expected, and what it is about the different steps of the
causal chain, the different interventions at multiple
levels that are responsible for those disappointments. That
seems to be an important focus of measures and measurement.
Locating and adapting existing measures and measurement
approaches. Measures that would allow us to measure the usual,
context, intermediate impacts, mediators and moderators, the
causal chain. But also perhaps most importantly in the paper
ideally there would be guidance in developing new measures and
new measurement approaches, especially as I said, for the
emergent phenomena. To the extent that we don't know in
advance what is likely to emerge, this is a bit of a
challenge. But I think thinking through the problem
and developing a measurement plan in a study that recognizes
the possibility of unexpected phenomena, that ideally would
be taken into account in the study design. And again, the
paper would ideally provide guidance. And then the last
point that I hope Marty and colleagues can address and
provide guidance in is, how do we allocate limited measurement
resources. The four quadrants of the table, other parts of
the presentation, and I'm sure the tables in the final paper
will have very lengthy lists of measures that ideally would be
used. Unless things change dramatically at the end of the
federal budget or within NCI, we're not likely to be funded
to operationalize even a portion of the idealized
measurement plan. So providing some guidance in deciding how
to allocate the resources I think would be a useful
contribution. Let me move then to the first paper, and again
begin with the question, what is distinctive about
multi-level interventions in health care, vis-à-vis
simulation approaches and computer simulation modeling.
My first comment has to do with the focus on the beneath
the skin or below the skin versus above the skin aspects
or levels. And here to me the key distinction is between
those aspects that are well understood and predictable, in
other words the sorts of phenomena that our clinical
research colleagues have the pleasure of studying and
dealing with because life for them is easy, versus those that
are much more poorly understood, more highly
variable and heterogeneous, emergent, complex.
My understanding and reading of the four models in the paper is
that models one and four do begin to address the higher
levels where things are more interesting and more
complicated, whereas two and three at least to me seem a
little bit less interesting and don't offer that much value for
understanding multi-level interventions in health care
delivery. So my hope would be to see more discussion of
models one and four and more elaboration of how they offer
guidance in beginning to use computer simulation modeling to
understand multi-level phenomena. There's also the
possibility I presume, and it's a good 25 years since my
undergraduate degree which was in Operations Research, so I
had at least some background exposure to computer simulation
modeling. But the question is whether linked models and other
approaches might be used when we are cutting across levels
where we have some more predictable, where some of the
simpler approaches are likely to be useful, and we're
combining those with higher levels where things are more
interesting, more variable, and more unpredictable. And that
might offer some way forward. I would hope also that the
revised version of the paper, or in future work, a follow on
paper perhaps, that the authors would provide guidance in
addressing the standard concerns that we all have with
simulation approaches. And this is addressed in the paper to
some extent. First of all, where do we go for data to
parameterize the models? Where do we go to find knowledge and
insights into the causal processes and mechanisms that
we are trying to model. I think providing more guidance
and more assistance and a leg up, or push forward to those
who are interested in diving into simulation modeling so
that you offer some guidance in answering the questions that
you raise in the paper. And then on the issue of model
validation, how do we convince our fellow researchers and
grant reviewers. But also, as the authors indicate, patients,
clinicians and others, that these models are valid and that
the predictions of the models can be trusted. I don't know
to what extent ideas such as split sample strategies work in
this field, where you would take half the sample and
develop and parameterize the model, and then you test it
against the second half of the sample and test the
predictions. If the natural variation that we see in the
ways that states have approached tobacco control
using settlement money, if that offers some ability to develop
a model on the basis of several states and then use it to
predict what happened in other states. Then use that sort of
retrospective approach for validating. But again, I think
some additional guidance in the paper on the issue of
validation would be helpful to those who hopefully will take
up the call and begin to think about using simulation modeling
in this field. And I would also hope that the authors could
highlight some of the other advantages of simulation
modeling. On the issue of model parameters and knowledge of
causal of processes, one of the nice things about being
forced to develop the simulation model is it's not very easy to
hand waive. You actually need to develop code. And you need to
specify explicitly certain aspects of the process that
you're studying. That causes you to, or allows you to
identify clear knowledge gaps, gaps in the available evidence
and data, as well as gaps in our knowledge of causal
processes. So this I would see as a major benefit and output
of simulation modeling that ideally would be highlighted.
And finally on the issue of interactions and emergent
patterns, as the authors indicate, simulation modeling
does allow us in a much more feasible manner than empirical
study to begin to explore the emergent patterns, the
synergies, the interactions. And the next step in the
process ideally would be to seek empirical evidence of
those predictive phenomena and again show the value of
simulation modeling. And this too is an area where I think
simulation offers a value and where the authors point out
that value, but I think could be highlighted and strengthened
in the paper as a way of further showing the value and
the many positive aspects of this paper. So one final slide
and a couple of very minor suggestions to the authors.
Each of the four models has a very brief paragraph, it's
entitled How Can the Model be Extended. I would hope that the
authors could provide more explicit guidance in who to
extend it. I think it indicates I one model four for example,
they offer some suggestions for using the disparities model to
better understand disparities. There's a large literature on
various hypothesized causes of disparities addressing
differences in patient knowledge and beliefs, values,
cultural values and so on. Resources available to them as
well as hypothesized causes related to differences in
position knowledge and beliefs, unintended or unknowable,
decision biases, differences in community resources. I think
extending the discussion in pointing out ways in which
simulation modeling could be used to take into account all
of these hypothesized causes and simulate the effect of
providing increases physician knowledge or providing some
sort of decision support that would overcome some of the
decision biases. These are the kinds of problems and questions
I think that simulation modeling would be ideal for,
and providing more explicit guidance and illustrations
would increase the value of the paper. And then finally, I'm
personally not very knowledgeable about but very
interested in our committees. And to the extent that it is a
prime example of this kind of modeling and has been used with
some success, I think expanding the description as a way of
allowing those of us who either have 20 year old knowledge of
simulation or have even less knowledge of simulation to
better understand what it's all about and what it can
do for us. I think using your committee's model as a
case study would be helpful. And with that I will stop.
>>>[APPLAUSE]