Day 2 Clarifying and Probing Questions Differentiated Accountability 2

bjbj MR. JUPP: So we are going to begin with a short round of clarifying questions. I am
going to turn it over to my colleagues, the discussants, in just a second, but the clarifying
question I want to ask the two presenters to start off with is, I actually think we're
going to need to ground some of your conversation that you started with in sort of material
examples of practice. I think it was helpful advice, and I think the altitude at which
you flew when you gave us the advice was necessary in part because there are not a lot of practical
examples of State systems that we're trying to talk about now, and I think we have to
begin by admitting that, okay? But I'm going to start with something I think that we didn't
talk a whole lot about, and I am going to ask Ben to weigh in on this first and Rich
to do so second and do this in a way that is not probing. I am going to ask you to list.
We are talking about differentiated incentives and supports. I would like you to make a short
list to five incentives that would be useful for driving school performance in the ways
that you've described and in the systems that you've described. Ben, do you want to take
a whack at it first? MR. LEVIN: Sure. Using examples that we used? MR. JUPP: Sure. MR.
LEVIN: Yeah. So one thing is we gave people an opportunity to get more help. We had a
turnaround school program at one point in Ontario. We merged it into something we called
the "Ontario Focused Intervention Program," but essentially, we identified schools that
we thought weren't performing well enough, and we provided additional support to them.
And I would not underestimate at all the incentive power of saying to teachers, "We are going
to help you do better work." This is very motivating for teachers when they see it happen.
So this was a matter of sending in experts. Usually, there were teams of people. We would
send in reading specialists, experienced principals, coaches, and provide some money to the school
for PD and materials. And that had a big, big effect on moving teacher practice. We
had a system called "Statistical Neighbors," which we used to assess school performance
relative to other similar schools on demographics, and by the way, one thing I would say about
that, that I didn't earlier, is we never assessed a school based on 1 year of data. We always
used an average of 3 years of data, because the reliability of 1 year is too small for
small schools, and that's why you get the bounces that were talked about this morning.
So we had a system to do that, and as was suggested this morning, that was a system
that we built through interaction with the field. So, when people saw their assessment,
they actually knew that that had a reliability and validity to it. We gave people a lot of
recognition. We declared Lighthouse Schools, and we gave Lighthouse Schools money, so other
people could come and visit them, so that they had some release time and so on. So we
did a lot of here's a school that does really well at this, here is the district that does
really well at that, it's not a ranking system, but we want other people to learn from you.
And people loved that. So those are a few. MR. JUPP: Good. Rich? MR. WENNING: I will
throw out four. One, consistent with what you just said, is recognition, but also financial
incentives for high growing schools, schools that are getting it done, as well as incentives
to replicate schools that are doing great as well, another there. Incentives are both
sanctions and rewards, from my perspective, so I feel strongly that having State authority
to close schools is important, that districts and schools know there is an end game at a
point, and that shows that one is serious. Another incentive is how you shine light on
what's working. So, by having really engaging information available for the public, where
you can highlight schools, basically, you can create a system we've done this in Colorado
where anyone can benchmark schools on the Web and understand performance of schools
with different kinds of achievement and demographics to see what the best look like. So it becomes
very easy to find out who the best is. That shines light on them. One thing we ought to
do that we are not yet doing is then not just showing the performance, but really being
able to document the adult practices going on in those buildings and making it available
in a manner that's just not doesn't require you to actually go visit the school, but that
we can document and share digitally the adult practices that are going in those places,
so those are a few I would throw out as incentives and sanctions for innovations. MR. JUPP: Great.
Some clarifying questions for you all, it looks like Delia has got on, so, Delia, jump
in. MS. POMPA: My head is full of probing questions, but I will try and make them clarifying
questions. [Laughter.] MR. JUPP: I appreciate that. That's discipline. MS. POMPA: From everything
you've said and I respect and admire the schools that are doing the short lists that you all
just talked about. However, it seems to me from everything you said, it is an issue of
capacity, and perhaps this is a question for the Department. I'm not sure. We're talking
about waivers that are going to roll out, if you look at the timeline, over 3 years.
So I guess my simple question would be, can you give us an example of where capacity has
been built in that period of time and then turned into a systematic response for an entire
district? MR. LEVIN: Yeah, I can in not just the district, but in fact, I would say across
the whole province. In 2004 in Ontario, we had 20 percent of our elementary schools where
one third or fewer of the kids were meeting our performance standard, right, so 20 percent
of the schools. Now the performing standard is not one third. It is 50 percent, and we
got 6 percent of our schools under 50 percent. So there has been a huge improvement in performance
at the bottom, and we did that by providing supports to those schools, sharing practices,
networking them, networking their principals, networking the teachers, providing those skilled
teams in those schools, and those ideas have been very widely taken up. MS. HAYCOCK: Can
I follow that up? So I am sitting here listening to you, and I am a chief in a State that's
got horrible budget issues right now. What of what you learned what would you do if you
were in that circumstance? As you look back on the infrastructure you've built and what
made the biggest different and what didn't and you were doing this in a horrible budget
time, what would you do first? MR. LEVIN: Well, the reality is that we did spend quite
a bit of money, and I am going to put the money into two buckets. Most of the money
was spent on kind of keeping the system going, so collective agreement issues, right? That
is where most of the money went, and building, we spent a lot of money on fixing buildings.
The cost of driving improvement was about 1 percent. That is what the cost of creating
the infrastructure that really drove the improvement, but I want to say that if teachers are all
pissed off about their salary and working conditions, the 1 percent won't drive the
improvement. MS. HAYCOCK: The 1 percent, you are talking about spent at the province level,
or this is 1 percent at the local and province? MR. LEVIN: Both. Both, because we have a system
that is 100 percent provincially funded, which gives us another big lever, of course, but
that 1 percent, a lot of that money went to districts to do things related to strategy.
Rich's point about strategy, I am completely on side with that. MS. HAYCOCK: Got it. MR.
JUPP: Rich, do you have a response to Delia's question? MR. WENNING: Just Kati's. Two examples.
One, at the State level, budget cuts, you have got capacity challenges. Philanthropically
supported change agents can be very important. Colorado uses them quite a bit, so don't underestimate
your philanthropic community to help you out. At the same time, tap what you have more effectively.
Kati asked me a question before. I gave an example of a rural district in Colorado. They
did an amazing job of improving their performance through their RTI process, the focused inquiry
through the planning process, and then the most important mechanical thing they did was
cross grade span planning, so they could plan the transitions more effectively. And that's
something that we can manage across our feeder patterns right now. It doesn't take more money
necessarily, but it takes a different use of our time and how we interact across schools,
and so there are ways to tap capacity that you're not really utilizing very well right
now through the way you engage professionals in this work. MR. LEVIN: Can I give one more
example on that point about the money? One of the biggest wastes of money in the whole
system now is how much money we spend in high schools on kids repeating years. In Ontario,
we had 30 percent of our kids not completing in this is not in 4 years in 5 years, 40 percent
not completing in 4 years. That's 40 percent, is 50,000 kids a year coming back to do a
fifth year of high school at 10,000 bucks a pop, $500 million for those kids to come
back and do an extra year. One third of them completed in that extra year, so $350 million
in the toilet. If we had taken that money, as we have now, by improving our graduation
rate, we can spend less money and have more kids graduate by providing the support, so
that they don't fail the bloody courses in the first place. Big efficiency, Kati. MR.
JUPP: Eric, do you want to come in after that? DR. SMITH: Oh, yeah, sure. And I see a lot
of droopy eyes out here after lunch, so we got to fire this thing up a little, this conversation,
I think, and get it going. These States, our States have been at accountability in some
form or another for a fair amount of time. You kind of blew through the pathologies at
the very end without a slide. You had to actually write the words down, but it range so true.
Teaching to the test. Well, have we ever heard that before, gang? I used to call in Florida
I'd go to meetings, and the superintendents would be more concerned about me being the
referee and defining the rules of the game rather than how the team is going to play
the game and couldn't get them to talk about instruction. They would talk about the grading
system and the percentages. It drove me crazy. So, if you're in a State, you say, okay, over
the weekend, I am going to dream up these objectives, and I am going to figure out the
formula and present it to my governor on Monday, and then we're going to get rolling on it,
but you are doing that in a culture of school districts that have already learned how to
do accountability. And they probably have learned well the pathologies. So what suggestions
is this a clarifying, or is this a MR. JUPP: It is not a clarifying. [Laughter.] DR. SMITH:
So what suggestions would either of you give to because there is another page to this thing
about how aas commissioner or State chief, how am I going to approach the pathologies
that already exist, and through smart design of a new system, move the conversation to
where it ought to be, about kids and learning. MR. WENNING: I'll take a stab, and this chart
is probably useful for this. As we are going through this design of the feedback loops,
we can anticipate the they don't need to be unintended consequences. We can anticipate
where the gaming will be. Just what we did with AYP, there was a clear disconnect in
terms of what we were trying to accomplish, and kids getting there, the disincentive immediately
became focusing on bubble kids. So, when we construct these multi measure frameworks,
which is just a feedback loop to the public, signal setting, we need to choose to mitigate
unintended consequences and anticipate them. That is one of the reasons I personally feel
very strongly about focusing on growth. It is hard to game growth, and then if you can
get credit along the achievement spectrum with your growth, it becomes harder for folks
to pick certain kids versus others. That's just one example. Another is to bring in other
evidence and to make sure you are balancing this normative, this criterion referenced
evidence. We are focused on college and career readiness. One area that we have gotten some
traction with is on graduation rate, and our framework now in Colorado, we calculate a
4 year, 5 year, and 6 year on time rate. You get credit for whichever is better. We have
too many kids that are showing up at the destination that aren't ready with getting diplomas. We
would rather folks take the time to actually get them there. But if we focus all the consequence
on a 4 year on time, we would be creating an incentive for schools to move kids out.
We don't. We want to keep them in and re attract them. So I think as we go through each measure
and we think about its value for promoting inquiry as well as judgment, we can consider
what might be unintended consequences and then design smartly to balance a set of measures
to mitigate those. You want to eliminate them, but we can do a pretty good job of mitigating
them. The other aspect is to enter qualitative review into the work, which we haven't touched
on much, to supplement the quantitative, and that's another way to get a much better perspective.
Great Britain, of course, and other nations do a lot more time on the qualitative side
and less on the quantitative. MR. JUPP: Ben, do you want to respond to that question? MR.
LEVIN: I agree completely with that. I think what I'd say is that in the two experiences
that I have had doing this, where you are trying to turn a system which has been used
to one set of rules to a different set of rules, you got to recognize that even when
people like the new rules better, they are really used to playing the old game, and they
can't switch games that easily. So you have to have a process of helping people learn
that now we are going to do things in a new way, and you can't expect them all like new
governments do, you know, we're the good guys, everyone is going to love us? No. They are
going to treat you the same way they treated the other guys who you thought were the bad
guys until you teach them to do something differently. So there has to be a process
here, and I think this is implied in what Rich has just been talking about, of helping
people get off that way of thinking and into that way of thinking, and that has be to be
deliberately built into your communications and the way you work with people. MR. WENNING:
I had one, just a quicky on this. Keith mentioned this before, and that's the Request to Reconsideration,
which is a tough name, but we didn't want to call it an "appeal." Districts were really
interested to see what kind of posture we were going to take. This serves two purposes.
One, it is saying we might not have gotten it right, present other evidence to us, and
so if we are creating some unintended consequences, tell us about it, and present another case
to us that we can evaluate. And that can be a cumbersome process, but it also serves another
purpose, and that is inquiry into evidence. Now districts are saying, "Well, what does
our evidence tell us?" and that step, again, helps mitigate some of the hard edges maybe
on a framework, and it allows local evidence to be incorporated. But it's also useful,
because it causes that internal inquiry that we wanted to have happen anyway, so we make
a messier process, but it probably produces a better outcome, because we are engaging
folks in the examination of that evidence. MR. LEVIN: It goes back to that Dee Hock quote
I showed at the beginning. MR. WENNING: Right, exactly. MR. JUPP: So we are clearly in the
rome of probing questions. [Laughter.] MR. JUPP: I am not going to hold you back, Delia.
I think my discussants actually lost the discipline for those questions, so let's let them do
what they want. MS. POMPA: I think there is going to be a natural tendency I hope I'm
wrong for a while to focus on the 5 percent lowest and the 10 percent gap, because of
that capacity that isn't there yet. How do we connect the interventions in that group
of schools with all the other schools to show that there could be a prevention aspect to
this? MR. WENNING: Shall I take that? I think in order to accomplish that, we need a consistent
approach to planning, and how do you make improvement planning actually meaningful as
opposed to a compliance activity? As long as that plan has consequence to it, I think
we can motivate that, but the plan is the same in that respect, so that all schools
and all districts in a State and when we're creating an accountability system for all
schools and districts, not just Title I but for the entire State, a single State system,
the planning cuts across every tier of schools. The variant really is the support you all
are providing at the State level versus the support they might be getting themselves or
through their own networks, but again, a consistent body of evidence, a consistent planning process,
a categorization of all schools, including those bellwether schools and your lowest performing
and essentially, what an accountability system is, it's an RTI model, where we are trying
to figure out where we are going to support and pay attention, and we are going to adjust
that over time. Ultimately, you run into a capacity limitation in terms of how much you
can do from the State, so you are going to focus it on places where you hopefully are
going to get the most bang for the buck. Some folks might say we ought to be focusing more
on the middle because we might have better success with them. I think we have to be somewhat
patient but also a little urgent on those bottom 5 percent in terms of how much we want
to mess around trying to fix those schools versus considering phase in/phase out strategies,
particularly at the high school level. But I think the system has to feel that all schools
are subject to it. We in Colorado made sure all these plans were public, so everyone got
to see them. Whether the State reviewed them or not, they become a body of evidence that
all can inquire into. I'm not sure if that's getting at it, but I think that's got to be
part of their plan. MR. JUPP: Rich, good. Ben, your whole speech began with how important
it was to address all schools. MR. LEVIN: Yep. MR. JUPP: Delia's question is about how
do address all schools, so let's talk about how it happened as a material. MR. LEVIN:
Sure. Well, that's the first thing I'd say. One other thing I want to say about this is
we were very clear in our communication that we're not interested in great teachers and
bad schools. We're not interested in great schools and bad districts, and we're not interested
in great districts and a bad provincial system. We want every school to be at least good,
every teacher to be good, and so everyone was responsible for everyone else. And we
didn't want anyone to feel proud, because even though the rest of the district sucked,
my school did really good, that is not something we wanted anybody to be proud of. At the Summit
in New York, one of the powerful things that was said by a couple of countries, Singapore
and China particularly, was that when a teacher or principal is asked to go from a high performing
school to a really low performing school, this is regarded as a huge compliment. You
only get asked to do that because people think you're really good. That's the kind of view
we want to have of system ness. We worked with our districts, essentially. I mean, to
Rich's point about capacity, no matter how much infrastructure you build at a State level
and we built a lot you can't reach everybody, so you have to have the districts as active
partners, and we relied on districts to do that work of creating a feeling in the district
that it's not just about those six schools. It's like everybody in this district is committed
to improvement for every kid. MR. WENNING: I will throw one other quick one out. As you
think about what you are going to do for all schools, each of you will need to decide what
are those couple things we do great from a universal support perspective. My bias on
that goes to being really great at building performance management capacity and providing
those feedback loops. That is something States are really in the unique position of, because
States have all that data. So, by being much better about using that data and turning it
into really actionable information, feeding it back, that is something you can do for
every school in your State, and it becomes very challenging for those good schools to
ignore that evidence, when their parents are talking about it, teachers are talking about
it. And by the way, that's actually easier support to give from a State level than actually
trying to go in and fix a school. So thinking about what foundational elements ought to
be part of your universal support that you can provide at scale is an important step
to making all schools feel part of this. MR. LEVIN: England has had a lot of success by
twinning schools, high and low performing schools, and creating a mutual responsibility
across those schools. That is another strategy that can work well. DR. SMITH: I think it
is important not to miss a point. The State looks at differentiation of accountability,
because I think every State is really challenged with capacity on two respects. One, they don't
have enough. Second, the people that do have are not trained to do the kind of work, the
waivers asking to be done in most cases, so got double challenges at the State level.
The point of contact, if I heard you correctly, is really informed about the local school
problems, but it is really at the district level. Otherwise, you are going to really
run out of capacity at the State. MR. LEVIN: Yeah. When we started with working with 100
turnaround schools, which was a holdover from the previous government, but we could see
we couldn't get anywhere near the scale. We had to have the districts mobilized, absolutely.
DR. SMITH: Right, which is a totally different strategy for a State to think about how they
are going to do that rather than I am going to send a bunch of people out there to work
in schools and classrooms. MR. LEVIN: Yeah. And also, because the districts control so
much of what happens in the schools, so you do some work with the school, and the next
thing you know, the principal has been bobbed off somewhere else, right? So it has to be
what Michael Fullan calls "tri level." You have got to be aligned. MR. JUPP: Eric, I
want you to actually elaborate on the work that you have just described, because the
way you said it too easily goes without saying. There is a temptation for the State to get
into the hands on work of turning schools around, when in fact that is probably not
going to be effective at scale. You are used to thinking about things on the enormous scale
of Florida, which is actually much, much larger than Ontario. How did you actually set up
systems, and what work did State office people do when they addressed underperforming schools,
and how did districts actually work to get those underperforming schools to turn around?
DR. SMITH: When I first went to Florida, we had I call them a "system of checkers," and
they'd go out and check on Title I money being spent correctly, and they would usually check
the day of graduation to make sure all the things were going correction. And they were
totally worthless. I went to Miami one time, and I saw my checkers in shorts, getting ready
to fly back to Tallahassee after diligent work in the south part of the State, and I
said they are going to be fired on Monday. And we did. We got rid of all of them. We
turned around and hired people that really knew the business well, which is impossible
for State agencies to do because you can't pay good enough, well enough to do that. You
can't compete with district salaries. So, to beat that, we actually hired on loan staff
from districts to come with us. We pay their freight, but it would be a contract with the
districts, so it didn't show upon my books. This is on tape, too, isn't it? Uh [Laughter.]
MR. JUPP: You're not working there anymore, though. What happened in Florida stays in
Florida. [Laughter.] DR. SMITH: Then we could keep a salary structure and a benefit system
for these folks that were coming in at a principal level, assistant superintendent level, to
be regional executive directors and then really strong people in reading and mathematics.
The trick for us at least was that the regional person, I personally told them they were to
know what was going on in the failing schools, the struggling schools, but their point of
contact wasn't with central office and an assistant superintendent or whatever. It was
with the superintendent of schools, and their job was to report to the superintendent of
schools on a weekly basis, the work that they are observing and seeing, the challenges they
are seeing, more breaking down. Because particularly in large urbans, quite often we see that the
problem was not with the school, but with that distance between the school and the superintendent's
office, and the superintendent really wasn't aware of the challenges that were being the
books weren't unpacked for 2 weeks after the school year started. The principal didn't
have a clue. But it isn't the tendency. Even if you make the shift in talented people to
do the work in buildings, the tendency is to roll up your shirt sleeves and start telling
the teacher what to do. The important thing is to tell the superintendent to tell the
supervisor to work with the teacher in the district. It ends up not being just about
the 150 schools you got to worry about. It ends up being with all schools within the
district, and it all becomes one huge culture change at the district level, a huge culture
change at the State level. It's hard work. MR. WENNING: May I ask one thing? This topic,
as Eric just said, it is a cultural shift in roles, and it's a profound opportunity
we have with this waiver package to get the roles kind of back in sync. The metaphor I
like to use is one of portfolio management. If Arne is portfolio manager of State, State
is portfolio manager of districts, the districts are portfolio manager of schools, the school
is portfolio managers of kids and teachers. All those roles are incredibly important,
and as we construct these waiver plans and consider the different role in evidence and
who is making summative judgments, we do have an opportunity to build capacity at each of
those levels to play those roles, as messy as they are in our country, but those are
the roles that are a fact in our country. And I think respecting them and making sure
that we help districts help their schools is absolutely an essential aspect of this.
MS. HAYCOCK: Ben, I want to draw you out, if I can, on sort of the differences between
our two contexts. When you entered, you didn't enter after a regime of a particular kind
of accountability system. MR. LEVIN: Yeah. Actually, we did. MS. HAYCOCK: But not like
ours, right? MR. LEVIN: No. MS. HAYCOCK: So the question here is, it is a challenge, as
Mitch said and others have said today, that 80 percent of schools in some of the States
are labeled with a label that makes them all seem the same. But it will also be a problem
if only 15 percent of our schools feel like they need to improve. So the question is,
given our recent history and given what you think you know, how is it that States display
sufficient seriousness of intent, so that a new regime of accountability is not regarded
by the many people who have fought against educating all kids well? So what do you need
to do to demonstrate a seriousness of intent to make sure that those who have only reluctantly
come to this party stay at it? MR. LEVIN: Right. Well, I think in the Ontario circumstance,
we had had 10 years of not very good times for education, budget cuts, a lot of teacher
bashing, lot of negativity, lot of people moving their kids out of the public system
into the private system. That was the context when the government changed in 2003. I would
say several things happened. One thing is they did some quite high profile reversals
that signaled it's a different direction, and those were things that maybe didn't have
a lot of impact, but they were very, very prominent signals that said to people, "Okay.
We're going to go a different way." But we were very relentless on the messages about
what the central goals were, about improving outcomes and reducing gaps and outcomes. We
had three: improve the outcomes, reduce the gaps, increase public confidence in public
schools. Those were the three goals, and those were talked about endlessly. The political
leadership in Ontario played a huge rule in this, and our premier and then the three ministers
that I worked with over 3 years, they were out there a lot talking about this, and they
were very serious about it. I remember a conversation between my minister and a group of high school
leaders who were talking about the challenges of all these kids and all the problems that
kids brought to school. He looked at them. There were about a hundred of them. He said,
"You're telling me you don't want the kids? Because if you don't want them, I'll find
someone who does." [Laughter.] MR. LEVIN: I can tell you, that was the end of that conversation,
right? So there was a lot of that. That was very clear, but the whole system supported
that, too. So the literacy/numeracy sector there, which was run by this fantastic woman
named Avis Glaze, I mean, Avis was in people's offices, our superintendents and our assistant
superintendents, all the time about, "You know you sent me this plan. I mean, this isn't
a plan to get better. You can't be serious that this is your plan. Let's talk about a
plan that's real in terms of ambition here. Achievable, yes, but ambitious." I would say
we were very friendly to our districts, but I like to say there was nowhere to hide or
for any school. MR. JUPP: Delia. MS. POMPA: There are some groups of students for whom
we haven't done a very good job, even compared to other students. We haven't done a good
job for English language learners, students with disabilities, and the list goes on, immigrant
students. For those groups, we really even haven't built the pieces of the infrastructure
that are so important, like the assessments, how they fit into accountability, those kinds
of things. So I guess my question, if I'm sticking to the topic about incentives, is
how do you create layers of incentive for those kids, for schools to do well with those
kids, rather than saying one size fits all, here are the incentives for all kids, or saying
you have a different set of incentives for those kids? MR. WENNING: I think I am going
to rephrase your question just slightly. On the incentive part, let me see if I am answering
the question. I am not sure where the incentive comes in. MS. POMPA: Well, that's a question.
If it is going to be incentive driven, how does the incentive come in? MR. WENNING: Well,
this is where, on the one hand, how you weight the measurement system to focus on students
that you are describing is a really important part of this, and so how prominently do gaps
exist. In Colorado, we chose really to focus a great deal on that issue of the rate of
progress kids are making, and of course, that is all disaggregated by each group as well,
and that evidence becomes very important. Of course, the waiver guidance is making it
very clear that we want to pay attention. So the issue of identification is one step,
and weighting it in your framework to say we value this issue, so therefore we're weighting
it a great deal. But then the issue is of the quality of the information you are providing,
and this gets into our idea of achievement gaps and growth gaps. I know an important
validator for folks to feel that the framework we built in Colorado was fair was that the
students they were interested in had information about them that could be used by parents and
other educators, and folks were surprised that some groups of kids, like English language
learners, our problem actually as a State wasn't that they weren't making much progress
on our assessment and turned out actually English language learners were outgrowing
native language peers frequently, but it was close to enough. And in our frameworks, we
always make sure we focus on what would be enough as well, so no one is ever going to
rest on their laurels that maybe they are just doing as well as other students that
have disabilities or other students that are further behind. So I think you signal it in
the weighting, and then you back it up with the information you are providing, and then
how you organize your plan to make sure everyone is paying attention to that, forcing that
root cause analysis into the challenges faced by those students, doing that systematically,
doing it that scale creates something of an incentive system because of what we are paying
attention to and what we are valuing. MR. JUPP: So, Ben, you talked about how in Ontario,
there were gap closings between the English language learner population and the native
language population. MR. LEVIN: Yep. MR. JUPP: What systems of incentives did you put in
place? What supports did you put in place? MR. LEVIN: Well, it was really consistently
not so much incentives but supports. And I want to say Ontario is a place with a huge
diversity. Twenty seven percent of the kids in our system are born outside the country,
and in Toronto, it's 50 percent. So we have a huge diversity in our schools, especially
in our urban schools. In ESL, the things that we did were, the first thing is building what
Rich said. You shine a light on it, and you say, "Hello. We actually care about those
people. What is the data telling us, and where are we?" and then you start getting people
to say, "What could we do to get better?" And it turned out that once we started to
examine the challenges, they were we hadn't given our boards, our districts any advice
on how to organize ESL effectively. We give them money and say figure it out. There is
actually a lot we know about that. I happen to have a colleague who is one of the world's
top experts on it, who nobody had asked. We asked him, and then we started to say to districts,
"If you got very few kids, organize it this way. If you got lots of kids, organize it
this way." What do we know about self contained classes, about second language instruction,
about first language instruction? Tell people, and then put in place the structures and the
processes, so this becomes a regular part of what people talk about at the provincial
level and at the district level. And when you do that, you use data. I think this is
exactly what you were saying in some different words. You use data. You get people to pay
attention. You build on the expertise. You provide some policy frameworks. Stuff starts
to happen, because educators want to do a good job. MR. WENNING: Let me add one other
thing to this. MS. POMPA: Good answers so far. [Laughter.] MR. WENNING: And that's you've
got at least three mechanisms to accomplish this. We talked about the multi measure framework,
which hopefully is as parsimonious as possible, so it is not overly complicated. That can
weight and signal. We also have our recognition approach. Do we want to create a statewide
award to highlight? And then we've got our reporting. We have an entire reporting arsenal
that should be consistent with our multi measure framework, but we can report and highlight
a great deal more that's not part of a consequential accountability system, but it's actually part
of our disclosure to the public. And that's that public inquiry side. If we highlight
and allow inquiry by the entire public and educators on schools that are doing the job
for students that we want to succeed and do much better, that's another approach that's
not just part of our reward sanction accountability aspect, so we've got a number of levers to
accomplish that. MR. JUPP: We have time for probably our discussants to do one more round
of questions. I want to put the warning out that we are going to sit here for the entire
30 minutes while you guys come up with questions, so start coming up with questions, okay? We
are not going to let you out early just because you are quiet. The incentive is to be talkative,
not to try and sneak out. MR. LEVIN: How can you tell he was a junior high school teacher?
[Laughter.] MR. JUPP: But while you're preparing and while Delia and Kati are beginning to
reload as well, I want Eric to talk a little bit about his experience working with the
English language acquisition population in Florida. Florida made noteworthy gains, and
I think that I want to hear a little bit about the way the States differentiated accountability
system, the way the States incentives and supports for schools that were struggling
helped build that momentum in Florida. DR. SMITH: It made huge gains in Florida, and
I think a lot of it is attributed to the way the accountability structure was established,
the objectives, and the way the formula was written. For example, at the high school level,
we had points for high schools for increasing their graduation rate. We had an equal number
of points eligible for a high school that moved their bottom quartile in graduation
rates. So the very clear focus of the school wasn't just to move the bubble kids, as you
would say, to get those kids that you think you might be able to hang on to and get them
through graduation, but those that are academically in the bottom quartile to graduate. We find
that throughout other measures, middle school, elementary school, and so forth, in the system.
So that helped a great deal, I think, in moving the whole system. The last 4 years, our graduation
rate increased by 10 percent, and the largest percent increases were with African American
and Hispanic and Latino, so they are the ones that really showed the great progress. I think
part of it has to do with the nuance around how you structure the policy. On top of that,
I think the issue of applying, getting away from people, just focusing on the formula
at that school and district level and focusing on best practices, and I think we have developed
some pretty good strategies on how to really drive a different kind of instruction for
non English speakers and so forth. That helped us a great deal. But it is found throughout
the formula. At the high school level, we expanded beyond issues required by No Child
Left Behind. We went to AP and IB and dual enrollment and entry certification. All those
areas showed just incredible growth in minority populations, non English speaking populations,
low income children, special needs kids, where there is the greatest strength. So a lot of
it deals with the macro, how you make your overall design to make sure that, again, there's
not holes in the boat where you lose huge populations of kids to the process. MR. JUPP:
Kati, do you want to go next? MS. HAYCOCK: Sure. Ben,I want to draw you out a little
bit more on the peer school network think. That, as you probably know, is something that
we've had a little bit of experience but very limited here in the U.S., some in the charter
world, a few in other places, but not much. In fact, many States have organized even their
comparison school data systems in a way that actually don't create real groups. You get
one school. It has a unique peer group, which essentially cuts against this issue to put
people together. So tell us what you have learned about the circumstances under which
peer school networks work best as opposed to one can imagine easily the way they might
just reinforce bad practices. MR. LEVIN: Right. And I completely agree with that. This is
one of the big problems about professional learning communities. MS. HAYCOCK: Yeah. MR.
LEVIN: In my view, they're not a goal; they're only a means. MS. HAYCOCK: Mm hmm. MR. LEVIN:
They're only worthwhile if they get to better outcomes. MS. HAYCOCK: Mm hmm. MR. LEVIN:
Everything is about that. Well, we did a bunch of things around that. In this case, like
in every case, we actually borrowed stuff that people were already doing. We had some
districts who were ahead of us on everything we were trying to do, and we used you know,
shamelessly used what we could learn from them about what was working well. So districts
that worked with their principals in very collaborative ways, districts that work with
teams of schools in which it was understood that the whole team was looking after the
whole team, and everyone was applauding everyone else's success, what Michael Fullan calls
"positive competition," in which your getting better is just a spur for me to get better.
And I'm happy if you improve, and I want to improve even more. So, building those networks
of people, which was not about competition in the sense of the worse you are, the happier
I am, but actually the better you are, the happier I am, but also doing that not just
inside the districts but beyond the districts. Now, we are the size of Illinois in population,
but we got 72 districts, not 900. So the district challenge is very different in Canada than
in the U.S. because of the gigantic number of districts you have in many States, and
I don't underestimate the importance of that. And that's another reason that with small
districts, why you've got to try and connect people across districts and get people outside
those zones. So we did a lot of things. We had a network of principals called "Leading
Student Achievement," which we built with the principal associations. So these were
triads of principals from very different kinds of schools, with very different levels of
performance, that met regularly as groups of three to talk about what they were doing
to improve performance from three different districts, and then we bring groups of the
threes together. We did that with superintendents. Avis Glaze set up this network. She invited
our 10 lowest performing districts, the superintendents, to join a group with 5 of our highest performing
districts. They all agreed to come. It was voluntary. Everyone came, the 10 and the 5.
They met every month for several months around what are you guys doing that works in your
district that we could copy, but it was all about in the context of we've got these goals
about where we are trying to get to with achievement, and if it's not helping us get there, let's
stop doing it. MR. JUPP: Time for one more question. MS. POMPA: My question is actually
a comment on Jim's comments earlier and the qualitative review system and how well that's
working there and has worked in other States actually. It's an expensive process, though,
and for many States in here, I don't know that they are going to have the resources
to do that immediately. What are some qualitative what are some measures that you can get qualitatively
from that kind of review that you can build into an accountability system in a more simple
way? MR. WENNING: The purpose of the qualitative review is one aspect. Is it a diagnostic?
Is it a summative judgment? In the charter community, for example, renewal inspections
are different than what you would do for a qualitative inspection. So the question is,
is the qualitative review part and parcel of the accountability system, is it high stakes
or is it lower stakes. I'd probably lean towards making sure we have lower stakes ones that
are for diagnostic purposes that would be followed up after the quantitative evidence
is returned and schools are now categorized. Before we decide to spend millions of dollars
on a school, we ought to understand what is going on in it. I am reminded of Michael Barber
who told me that. Great Britain has to do three qualitative reviews during a year of
a turnaround school to really understand what is going on. Costs? These can be of different
costs. I know frequently, they might be in the 25 to $30,000 range for several days.
I just talked with Tom Boasberg in Denver who is doing shorter views with one of the
outsourced providers at about $8,000 per school. I don't think we can underestimate the value
of good qualitative inquiry into the practices of a building when we're spending a lot of
Federal money on their repair, and so whether you do something much shorter term and the
folks Jim from New York City can talk about the kind of expedited reviews that Joel Klein
did, which were pretty tight and short, versus ones that might be multi day. And I think
you can decide what you want to focus on if your budget is small, but the key is going
to be aligning that evidence with the evidence that's in the framework, making sure that
people are not confused by conflicting signals and bringing that alignment, so a few ideas
there. gd9Z gd9Z gd9Z gd9Z gd9Z gd9Z gd9Z [Content_Types].xml #!MB ;c=1 _rels/.rels
theme/theme/themeManager.xml sQ}# theme/theme/theme1.xml w toc'v )I`n 3Vq%'#q :\TZaG L+M2 e\O* $*c?
Qg20pp \}DU4 hsF+ ,)''K K4'+ vt]K O@%\w S; Z |s*Y 0X4D) ?*|f -45x /Y|t theme/theme/_rels/themeManager.xml.rels
6?$Q K(M&$R(.1 [Content_Types].xmlPK _rels/.relsPK theme/theme/themeManager.xmlPK theme/theme/theme1.xmlPK
theme/theme/_rels/themeManager.xml.relsPK
accent2="accent2" accent3="accent3" accent4="accent4" accent5="accent5" accent6="accent6" hlink="hlink"
folHlink="folHlink"/> Joshua Caitlin Normal.dotm Joshua Hoover Microsoft Macintosh
Word U.S. Department of Education Title Microsoft Word 97-2004 Document NB6W Word.Document.8