Winning The DARPA Grand Challenge

MALE SPEAKER: All right so I'd like to welcome Sebastian
Thrun from the Stanford AI Lab and he's going talk today
about winning the DARPA Grand Challenge, and what it took to
build a robotic vehicle that would race across
the desert by itself.

SEBASTIAN THRUN: Thanks Chris, for your short introduction.
Very much appreciated.
It's great to be here among friends.
I see lots of familiar faces people I studied with at Bonn
when I was 15 years younger Peachtree students, visitors,
it's great.
I gave this talk about 18 months ago, and I had Mike
Montemerlo, who is like the key person behind all of us,
and I had really nothing technical to say, so I had a
lot of fluff, and I remember the directional of the
projector wasn't working either, which was kind of
commenting on the quality of my talk.
Today I have a little more technical stuff to talk about
because we just did something so I want to
share this with you.
But before you get into it, this would be the work of a
team of people at Stanford University and Volkswagen,
Mohr Davidow, and so on.
So I'm just the spokesperson who's taking all the credit
and some of the money but I didn't do much work.
So let's see, who here has heard about the Grand
Challenge before?
OK, there's one person who hasn't.
It's OK.

There is a video.
I have a couple of them with me if you want some, that NOVA
makes, it's a really nice summary.
So I'm going to tell you the story of the Grand Challenge,
and some of the technology that went into it, and some of
the reasons why we do this the next, what do we have, 45
minutes or so.
So the US Government has had an interest in unmanned ground
vehicles for several decades, the reason being that on
current battlefield, lots of people die because, for
example, roadside bombs and so on.
And they've funded in the order of about half a billion
dollars of research work at universities and companies
over the last couple of decades on this topic.
And somewhat cynical, if you look at what
they got out of it.
Because you know how defense contractors work.
They are the opposite of Google.
They got relatively brittle systems that would require
often multiple interventions per kilometer driven, and they
would often offer train go as fast as ten miles per hour but
not faster.
So they weren't really operational.
So DARPA decided to construct a complete new funding model,
which is double the speed, no human intervention whatsoever,
and no funding.
So, let's see how far we can get without funding since the
funding, of course, didn't quite get us there.
So they created the DARPA Grand Challenge originally in
2004, and March was the first race.
It went originally from Los Angeles to Vegas, but then
they figured out there's too many kids in Los Angeles.
You can't do racing over here.
So they started in Barstow.
It's the place where Kill Bill II when Uma Thurman gets out
of the grave, you know, it's there.
And premised like the first Casino behind the vaudeville
people who can't afford the suit to go to Vegas.
So it's really kind of an interesting countryside.
And they picked a route right north of I-15, which is a
power lines support wall, and here's some pictures that they
posted before the race.
The idea was to build a robot that can go by itself.
You give it GPS breadcrumbs, about 2,007 of those, that
define the route.
But they define it only very inaccurately, so as you go,
you have to basically make lots of steering decisions and
velocity decisions.
There is no person involved.
The machine has to run by itself.
So the moment the race starts, you, as a designer, go back to
a tent, have a beer or whatever, and the robot has to
do everything by itself.
So it's somewhat unprecedented in that in the history of
robotic driving, there's never been a vehicle going 100--
this gets 142 miles by itself.
So this was the challenge.
The first Grand Challenge was a bit of a disappointment.
This is video of the DARPA course.
It gives you kind of of a feel for what type
people showed up.

Was anybody here at the first Grand
Challenge, physically there?
Wow, cool.
Two people, great.
You found a mix of other people, from top notch
research universities like Carnegie Mell, who's a really
big player.
Companies, mostly defense companies, who wanted to make
sure that their work is being acknowledged.
And a good number of people, who are thus referred to as
mainly car nuts, people who had no clue about robotics or
building large-scale systems, but figured they had the
computer in the car.
This is an open team.
People on a drive from Silicon Valley.
It is a good job.
This is a defense contractor that only practiced in the
parking lot and they had a control problem.
See this car flipped right over here.
DARPA shot all the video.
This is the biggest competitor, Terramax of Iowa,
a defense contractor, at 30,000 pounds.
They had a bit of a back up problem.
They backed up more than they went forward.
And then the smallest-- is there anybody here that
studied at Cal?
OK, if you're from CAL and you want to leave at some point
during the talk, it's fine.
Cal had the smallest entry.
Anthony Levandowski.
[LAUGHTER]
SEBASTIAN THRUN: So Anthony--
I have a lot of respect for him and I can tell you later
how cool it is to be on a motorcycle.
But I asked him after the race, what's your
goal for next year?
And he said, I'm going to double my performance.
Which most shareholders are excited about.
So Grand Challenge finished--
you can tell this is not a serious talk- finished at
-7.3, and many of the journalists said it was a
complete failure, because it was only 5% of the course, but
DARPA had put the hardest stuff up front, and some very
serious mountainous terrain.
And Carnegie Mellon, who had the best performing vehicle,
went all the way they are, basically blindly followed the
GPS points, got about a 1.5 meter error on GPS, and drove
on to this ridge over here, got high centered, burned up a
rubber tire, and that was the end of the race.
And it actually was over before most of us woke up that
morning in California.
So in 2004, we had 106 teams competing, and in 2005, when
we at Stanford decided to join, we had 195 teams. So a
little comparison here.
Of which Stanford is over there somewhere.
It's misspelled, with a D. That is something about the
average typing skills of certain agencies.
So we had about a year, roughly, maybe a little more
than a year, to put together a car that could drive itself.
And we started thinking it can't be that hard.
I mean, we drive cars all the time.
And then, as we started getting into it, we realized
there were certain difficulties
that we had to overcome.
So I'll tell you about those.
So this is our timeline.
And I start at the very beginning.
It's right over here somewhere.
So, in the very beginning, we had dedication to build team,
and Mike Montemerlo was my post-doc person who could be
the internal team leader with a couple of grad students, and
did most of the software stuff.
We had a no vehicle and no money, and nothing.
So, we had to find a way to finance an internal team on a
crazy idea.
So, the very first thing we do as a college professors, when
you don't have money to hire people, you teach a class, and
you give the students course credit, right?
So they pay tuition to take your class, and you teach your
class, you recoup your salary, and give them
an A or B or whatever.
So we created a course, DARPA Grand Challenge, which was a
project course, without a
syllabus, without any materials.
It's just one thing.
We're going to be 20 students, and by December 1st, we have a
car that's going to drive itself one mile through the
Mojave desert.
And students signed up.
40 came for the first class, and 20 for the second.
Kind of an exponential decay.
And they stayed on.
We then got in touch with Volkswagen to give us a car.
So they gave us a Touareg.
In fact, they gave us a number of those over the time.
So here's one of the students, David Stavens, who was a key
player in the car.
And the very first thing we did is equipping the car with
the type instrumentation that they usually used for these
kind of projects.
Everybody who does autonomous navigation in the air or on
the sea, equips robots with inertial guidance systems,
which are nothing else but the pairing of a GPS receiver,
that tells you roughly where you are, and an inertial
measurement unit that measures accelerations and rotational
velocities, so you can make interpolations between the
different measurements of GPS.
It kind of gives you roughly a feel for where you are.
And this is actually something that we work with
various people on.

Then we started devising a very simple control mechanism
to get to the state where the previous Grand
Challenge feet had been.
And the most basic question is supposed you're given a route
defined by GPS breadcrumbs by DARPA, and you care about
controlling the vehicle so it stays on the route, how do you
turn your wheels?
So it's a one dimensional decision every driver makes
all the time.
A very simple controller that actually we used on the race
day was designed by a fellow named Gabe Hoffman, which goes
as follows: if you're on the reference trajectory, then
make your front tires parallel to the reference trajectory,
Then you stay on it.
But if you get off it, then measure your crosstrack error,
and steer in proportion to that crosstrack error, so you
can see the wheels being slightly turned in because
they have a certain amount of crosstrack error.
That makes you go back to the course
OK, If you do that, you get a bit of a drunk squirrel,
because it's not oscillating left and right.
And for those of you who have a control background, and you
do PID control, differential term for dampening, an
integral term for drift and so on.
But essentially that's the picture.
So we put this together, and we tested this in early
November in Arizona on--
Volkswagen has a big proving ground there.
And this is basically driving blindly at the GPS points.
Kind of the same that most of the competitors in the
previous year had achieved.
And you can see it's going 25 miles per hour, relatively
fast. The steering wheel is relatively calm.
A bit of oscillation still.
So we're able to trace these points really, really well.
So we basically reached the level of 2004 in this class in
about five weeks.
Stanford students are amazing.
As I'm sure you all know.
And then we had to worry about cushion lines.
And then things became more interesting.
So we put lasers on the roof.
For those of you who don't know how lasers work, here's a
little animation.
So, a laser beam goes into a rotating mirror that gets
directed into the environment and to a planar area.
The laser light goes out, it's being affected by objects
being received by the sensor.
And we can measure the time of flight, and
thereby deduce the distance.
If you liked that animation, it's all stolen from this
video over here, OK.
Without permission.
And then you do motion planning And motion planning
could be as simple as the following which was Mike
Montemerlo's brainchild.
Basically, you try to follow a road.
You can go straight, but if you go straight, you might hit
an obstacle tat your laser sees.
And then you could, over time, gradually vary the lateral
offset to your reference trajectory.
For example, you can say, I want it, over here, I want it
driving a little more to the left side.
And then you can look at these trajectories, rule them out,
and evaluate them to the extent of which they violate
safety constraints or maybe corridor constraints.
You have the corridor in which you have to stay.
And if you do this at different speeds, you can look
at a fast bursts or slow nudges, and get a sense as to
how to move.
And essentially, what Stanley did is that.
So here's our first test It's not exactly desert, it's more
like a parking garage at Stanford, but that's the best
we could afford.
So, here is an evil little obstacle, that pop up in map
as little dots.
Another one.
And a little bit in the class, we had a little system that
could sense these things and drive around them, and not in
a very elegant fashion, but in an effective fashion.
So we took that to the desert.
Here's my class in Barstow, California, December 1st. And
we did our test. And the test was a bit of a disappointment.
We managed to get our one mile drive in.
In fact, we drove 8.5 miles, about one mile further than
Carnegie Mellon had driven, which, of course, we kept
talking about to our various friends in the field.
But a mile later, just after Carnegie Mellon failed, we
failed as well, so we didn't get very far.
And the car, because of the many, many different obstacles
got pushed around a lot.
This is where Carnegie Mellon failed.
We love to show this video.
That's exactly where they burned up their rubber tire.
We have a lot of respect for Carnegie Mellon.
But we eventually failed back there.
Which we don't show.
But the many obstacles in the desert pushed the
car around a lot.
And it was really hard to go fast. You could only go like 8
miles per hour.
And Carnegie had gone much faster than us.
Too slow for racing.
But we had something together, which was the first entrance
system, within two months, and we could now go and critique
it and improve it.
So the idea was, from that point on, we would only take
pick modules that wouldn't quite work, like pretty much
everything, and then replace them by a better module, so
that eventually we had a good driving car.
So the next thing we did is we actually got
together a real team.
And these are the people who actually did the work.
Mike was the software chief.
The people from Volkswagen, Sven and Cedrick.
Mohr Davidow is a local venture capital firm
that give us money.
We told them they have no return of investment.
They didn't mind.
And then we had a computer vision team.
And we had a planning and organization team and so on.
These are all Stanford students, mostly, and some
alumni, and so on.
I had a lot of fun just throwing together about 60
people to do this.
And we got sponsors.
So Android is one of the sponsors.
They provide all kinds of stuff like software, vehicles,
camera, caffeine, what have you.
And we came up with an architecture.
This is the most boring slide I have but it summarizes all
of Stanley's architecture.
It goes from sensors to perception modules to do state
estimation, to planning and control modules that control
the vehicle, and then to an interface.
And it's also pipelined in real time through the system
in about three hundred millisecond delay
from left to right.
In fact, for those of you who have an AI background and were
engaged in the artificial intelligence debate as to what
the right software architecture is, Rod Brooks in
the '90s came up with this subsumption thing, how to
organize robots and cursed the traditional approach, but we
are completing the traditional corner, see.
We are highly back in the 80's in how we build these systems.
But, in April, we had the team together, we started improving
our software models.
We had to face our first big hurdle, which was a called a
site visit.
DARPA had 195 submissions, entries, and they wanted to
find the 40 good ones that they could then further
support all the way into the race.
So we had to submit a video.
This is our video, and again we show the place where
Carnegie Mellon failed, just by coincidence.

And, we weren't the only one.
So Carnegie Mellon had a really strong team.
They actually were really smart.
They read the rules and realized, you could actually
submit two vehicles, not just one.
You could double your chances.
So they actually built a second car, which we didn't,
based on their Hummer, which was thought to be a really
sturdy off-road vehicle, and submitted their entries and
they passed with flying colors.
Just to give you a kind of a feel for what type companies
and what type individuals--
because I have two more videos I want to show you--
the first one, these were like more car nuts.

So some people clearly are like one thing that were
really good at, like maybe building a track vehicle, and
so saw the Challenge come along.
And they didn't think of it as a software challenge.
It had nothing to do with intelligence.
Just we have fastest in going from 1 to 65 miles per hour.
That's enough.
Software doesn't really matter.
And they had there.
And the next one is the most watched video.
I've been told some governor even saw it every evening.

I'm just going to play the beginning.
It's on the web.
They did actually quite well in the race.
AXION TAPE: This is a DARPA site visit video for Axion
Racing's entry into the 2005 DARPA Grand Challenge.
We want to thank the United States Government, DARPA and
it's director, Dr. Tony Tether.
HI Tony.
SEBASTIAN THRUN: So we can tell this is the only team
that brought an inflatable palm tree to the race.
It was really cool.
We weren't quite as humorous, I guess.
We're just engineers.
AUDIENCE: Who's team was that?
SEBASTIAN THRUN: It's called Team Axion.
It's coming out of Louisiana or something.
If you go on Team Axion, you can buy
calendars with these girls.
These girls were in Fear Factor before they--
they're twins.
So, this is the first time we had a car
without a driver inside.
This was the site visit.
DARPA had given us a very short course
when it was in place.
And even there was an obstacle.
Also known as trash bin.
And the car had to drive around it.
And it never was really elegant in doing it.
It was very short-sighted, But it was very effective.
I'll tell you later on why this is the case.
In fact if you look for me from the overhead, this is a
map of the vehicle built, which I'll tell you more about
in a second.
White means drivable.
Black means unknown.
Red are these little traffic cones over here.
And you can see the car kind of projecting
its steering direction.
And over here, you're going to see the evil
trash bin coming up.
Of course, the car doesn't--
I mean, you can see the steering direction.
Good, no one's in the car.
OK, so that left 43 semifinalists, which Stanford
was one, so we achieved our goal.
And so was Carnegie Mellon and Carnegie Mellon up here,
called Red Team, after their team leader, Red Whittaker.
And then we went into a phase where we really tried to
hammer it out of the box, so the goal was to build a system
that could run hundreds of miles without intervention.
So, at the time, you could do about a mile, and then we had
to intervene and we had to go to at least 500 miles without
intervention.
And that turned out to be tricky, because desert is
actually complex, and it's very alive too.
There's lots of rabbits and birds and so on, to be killed
along the way.
So we did a lot of testing.
Just go out and hand out in Barstow in 125
degrees in the summer.
Rosemary goes back there, has been no some of the trips,
driving all kinds of terrain.
And to be a little more technical, I'm going to show
you the typical stuff that would happen.
We would drive in the car.
And the car would do just fine.
We are all standing there
chatting, not paying attention.
And all of a sudden, even difficult terrain like this,
it will go crazy out of no apparent reason, and drive
maybe into a ditch, or a berm, or down a cliff, or something.
So we would not be able to catch it, and the car couldn't
be rescued.
What happened is, there were all kinds of data processing
problems where the car, in attempt to model the
environment, would model a road just fine
as shown over here.
And all of a sudden, you get these massive obstacles that
are completely fake and the car drives off the road and
kills us all.
And even though Stanford has a steady supply of great
students, we can't sacrifice them all.
So let me tell you a bit more technical, what's under the
hood, and tell you about three different innovations we made
or three different things we worked on during this race.
And one is laser interpretation.
So I told you about the laser going out and scanning the
environment.
As you scan the environment, at any point in time, you get
a scan line about 70 times a second, and you can assemble
them into a 3-d model, using a GPS inertial pose estimation.
Not very hard.
60 estimations.
XYX is also your yaw and roll and pitch
angle of the vehicle.
As you do this, you get these point clouds.
There's four of them here.
One for each laser: one in blue, one in
green, and one red.
And now you have to interpret that data and
ask where's the road?
Like there's a big mountain over here, and there's a big
ditch over here, and so on.
The way Stanley does this is, actually it uses a scanning
laser to build a 3-d model, and does inference
over the 3-d model.
So here's our laser going forward, scanning the road.
If there's a big pole in the way, the pole is being scanned
vertically, and if you start comparing the bottom point to
the top point over here, then you realize there's a Z
elevation difference and that constitutes an obstacle.
No more or less than that.
It's very, very simple logic.
OK, so something vertical, could be a--
I mean Stanley was very environmentally friendly--
like if it was a tumbleweed, it would drive around the
tumbleweed.
Anything.
It would just drive around it.
If there was an insect flying up, it would
drive around the insect.
Never mind.
Whatever the logic.
Now the problem was if the vehicle pitches, then the
scanline goes in reverse order of the driving.
So you might go forward, but the scanline goes backwards,
because the vehicle pitches down, and as a result, the
terrain get scanned twice, and sometimes the scanning is
consistent.
So when your pitch estimation is really accurate, you might
get these moments over here, like this one over here, where
the Z arrow is not large.
But sometimes, your pitch estimate is very inaccurate,
when your vehicle pitches a lot, for example.
You project out where the scanpoint is, so even a
quarter degree error in your pitch angle has huge effect on
where you think that point is.
So you have to be really certain about the pitch angle
that is way beyond what the vehicle could do.
So this is an extreme case of extreme pitching.
And this is the the resulting trace of one of the lasers on
the ground.
And you can see this is a flat terrain, but if you compare
this point to this point, just because you've got pitch angle
error, you think there's an obstacle.
OK.
So that was a bit of a bummer because maybe every couple of
miles, the car would just completely, without
motivation, pitch a little bit, and
then go off the bulge.
And then pitch a little bit- and, so you looked really good
at making a video that looked great, but you couldn't drive
it, at least not the Grand Challenge, so.
So this is where our first probabilities came in, and of
course, you know more about probabilities than me.
But the classical way to look at the data is just taking
rock solid and say they are all correct, right?
So you have a sequence of poses in this kind the mark-up
model, and you have that being fed by the inertial system,
and then you extrapolate by the points on
the physical world.
And then you can compare two of these points, for example,
and ask yourself, for two nearby measurement points, do
they have an elevation difference of 15 centimeters?
But in reality, these are erroneous.
So the truth is there's error between these pose
estimates over here.
There's error between these pose estimates over
here and over here.
so what do you got to write down is a published a full it
says no given all this error over here, what's the
probability that these two points are really in vertical
alignment of 15 centimeters?
And then, you only accept your obstacle if that probably
exceeds a certain threshold.
So if you are really confident, that's 95%
confident that there is an obstacle.
Now the problem with that was that now we had this really
complex, probabilistic model that we had to kind of feed
and populate, and no manufacturer ever tells you
what the actual error of their unit is.
So you have to guess it.
So how do you guess what errors you have in your unit
to get this test right?
Well you could sit down and measure it, which is almost
impossible.
So we decided to do something completely different.
We said we can do this in a day by using discriminative
machine learning.
Now how does this work?
So, discriminative machine learning, in this case, works
the following way, and it's not very sophisticated.
If your error model is correct, then it's going to
label flat terrain as flat and non-flat terrain as non-flat.
OK, now how can we train a machinery model to do this?
Well, lets label the terrain.
How do we label the terrain.
Well, you can have a grad student sit there and label
every pixel and say, this is an obstacle, this is not.
Or you could just drive somewhere, declare what you've
driven over as flat, because you chose to drive there.
And then kind of falsely assume that the side stripes
over here are non-flat, because
there's lots of bushes.
It's not quite correct, but good enough.
And then you can tune your parameters inside the model,
so that they minimize the number of pixels here that are
labeled drivable and maximize the number of pixels over here
labeled as non-drivable.
This is just a discriminative machine learning way in search
of a mark off chain parameters, blah, blah, blah,
blah, blah, to find the optimal error model.
So basically, you learn the mark off chain using
discriminative learning.
And the results were actually quite amazing.
So, it might look insignificant for you, but for
me was a big day.
You can see over here fake obstacle.
This is side by side, before and after.
And over here no fake obstacles.
But with this error model in place for the probabilistic
test for occupancy and our machine learning routine with
a couple of minutes of training, we were able to
basically eliminated all false positives.
There's one over here, that's missing over there, without
affecting the correct positive rate.
And some benefits.
We went down from like 12.5% false positives
to like one in 50,000.
So, all of sudden, we could actually sense terrain.
And what's surprising--
I mean, I've published books on probability in robotics,
And I've forgotten all about this when I made this car, and
then I realized, wow, probability is
actually your friend.
So they kind of come in again.
The next thing came up because the lasers
don't look very far.
They only look like 20 meters in distance.
And we wanted to drive really fast. And to drive really
fast, like 35 miles per hour was our set goal.
We had to look further.
We couldn't stop in time before the
laser caught an obstacle.
We measures that.
So this is a distribution of speed limit in the original
Grand Challenge.
And you can see it goes all the way to 60.
But if you go to 35 miles per hour, you have more speed
limits caught.
These are DARPA-dictated speed limits.
So we did something else.
We used a camera to find roads.
And you'd think that a camera was, of course, could see all
the way to the horizon.
And so the question then becomes how do extract from a
camera image where the road is?
And after 50 years of research, there's got to be a
piece of software that finds your road in the
camera image, right?
AI just turned 50 this year.
We should be able to do it.
It turns out it's not easy.
There's no paper that I know of that can find you all the
world in all camera images.
OK.
Now we started something simple and said maybe roads,
say, there's a bit of diversity of roads.
Just to give you a feel for it, this is often the original
Grand Challenge course of 2004.
We started saying maybe roads are brownish.
OK, so take a region of the color spectrum and call it
road, right?
But, of course, you drive over a paved road that's black, and
it doesn't work quite well.
We fight over stuff like this every year.
And then we said, maybe the road is the smoothest
thing in the image.
In fat, the less smooth it is, the less you want to
drive on it, right?
So the smoother it is, the better for you to drive.
The smoothest thing in the image is the sky,
unfortunately, That's kind of a bummer.
And then Hendrik Dahlkamp and a couple of folks from Inter
came up with a decisive idea, which is was just, we don't
have to solve all of what we're finding, we just have to
solve some of what we are finding, which is given that
we know how the road looks here, maybe we can dictate
road over there.
By that I mean the following: we already had a laser system
in place that was perfectly capable of fighting
roads, but too late.
It could find roads right in front of the car, but not
further out.
But if you could extract from this, see the video here, from
the laser system, you might be able to extract samples of how
the road looks right now.
Because this is what the laser sees.
It doesn't look very far.
If you could extract a region that's drivable, and use all
these pixels over here as training examples for a
mixture of Gaussian, and then apply that mixture of Gaussian
to the entire image in hopes of finding the road all the
way to the horizon.
And it's a typical result here, of a mixture of
Gaussian, where red means drivable, and I guess, blue
means unknown.
And, all of a sudden, we could would bootstrap our perception
range from the short laser range, using this little
machinery trick all the way to the horizon.
And it was a big day too, because all of a sudden, you
could drive faster and you could see
stuff in longer distance.
It's not perfect.
Like if the road changes its color, then for a moment, you
think, wow, that's something else.
But then you just slow down so the laser can catch the
obstacle, and that's fine.
So here is a typical example.
It's one of the early versions.
And this just shows, as you take the straight road over
here, and do the trick, the laser goes
about 20 meters out.
This car's driving down.
The dark red stuff is laser.
But the vision goes all the way to 60 meters out.
So all of a sudden, we could see much, much further.
We could detect obstacles much earlier.
So that gave us the speed.
I'm going to show you a video of the adaptation in action.
This is taken from the what is called the qualification
event, where the vehicle had to go from pavement to grass.
And as you can see, as it goes into the grass the first time,
it doesn't quite know what it is, so it
says look not drivable.
But as soon as it gets onto the grass, the laser catches
the vision of the grass, they vehicle adapts to it and is
able to then find grass very reliable.
So here is another transition going on.
Those are of course obstacles.
They are flagged correctly.
Here's another one.
This is a typical section with hay bales.
You can see the adaptation going on, and so on.
And was able to drive quite reliably.
So some of the results that we got with the adaptive vision
was really amazing.
I mean, I teach Computer Vision at Stanford, and I tell
the students, vision doesn't work.
Because it's really hard to make work.
But this really adapted all the time for lighting
conditions and road colors, and if the sun goes down on
the horizon, we just have a different model, and so on.
And it worked and quite amazed me well.
The last thing we did is, after we were able to do 35
miles per hour.
This was very short.
The problem we ran into was the vehicle was too fast. It
was way too fast. So if you go into 280 and drive 30 miles
per hour, that feels slow, but if you go into a very
treacherous mountain pass with a big cliff on one side and a
mountain on the other side, going that speed makes you
feel that you're crazy.
So we had to find a way to make the vehicle go slower.
Specifically, what happened is if we go to rough terrain,
like big ruts or so, like washouts from the rain, and it
would just go at exactly the same speed, just like a robot,
as it does on nice and benign terrain.
So we had to find a way to make it so down.
And to cut a long story short, we worked on adaptive
mechanisms for speed control.
Before I show you those, let me one last time highlight
what one of our competitors did at this wonderful place in
Pittsburgh where I used to work for eight years.
They actually recognized this problem and had a team of--
[VIDEO PLAYBACK]
-Four editors for this one management station.
[END VIDEO PLAYBACK]
SEBASTIAN THRUN: --like 24 people in a trailer that would
go through every inch of terrain and label it and tell
it exactly what speed it should go, based on area in
which they collected and they had a vehicle that does it for
months on end to get lots of ground data.
And they succeeded in labeling it but we didn't have the
manpower to do this, and we didn't feel like we wanted to
really hand label all of this.
So what it is we use again, machinery.
Being excited now about was machinery.
Where we showed Stanley how fast to drive, by just driving
a stretch in the mountains, OK.
And it was me driving, so the car got like a distant German
style of driving in the end.
And then build a controller.
This is the same figure.
As you will see, the yellow stuff is human driving.
The pink stuff is robot driving.
But the controller kind of copied that.
And the premise of the controller was it was
monitoring the amount of shock that the vehicle got, the
steepness of terrain, and the narrowness.
And it would then slow itself down to meet certain guideline
parameters.
And after it, say, received a shock, it would then slowly
forget and get faster and faster again
until it got new shock.
And get faster again, new shock.
Faster again, new shock.
It wasn't quite human driving.
I mean, typically, humans slow down before a rut like this,
would kind of would slow down after the rut, but still, like
ruts come in groups, and so on.
Whatever.
So with that, we had a system together in September where we
did thousands, hundreds of miles of testing.
In fact there was a constant balance between spending more
time in Arizona testing, and making sure we don't get
divorce at home.
I mean, wives still recognize us when we came back home.
Or we would fly them in and they would hang
out with us in Arizona.
And the car was actually quite competent.
It was not elegant.
So the steering was usually a little bit later
than people do it.
But you could drive hundreds of miles without any problem
whatsoever.
We did the 200 mile run in one day.
And you could average speeds of about 22
miles per hour, roughly.
Which in a desert, trust me, is a lot.
So if you put yourself in the car, 22 miles is a lot.
In fact, as Mike always observed, it's illegal to
drive drunk, but it's completely legal to program
you're car when you're drunk and then be driven by it.
So this is a good, kind of fun video.
Carnegie Mellon was a very serious competitor and by all
means they have won just the same way we won it at
Stanford, and I comment on it later on.
But they did fantastic with it in the desert, but they have a
lot of industrial wasteland in Pittsburgh.
So they practiced there.
Berkeley, my friends from Berkeley, practiced their
motorcycle.
They did a new one.
They didn't quite get the---
[LAUGHTER]
SEBASTIAN THRUN: Now, I mean, I show you this video, as I
got a comment that I have an enormous respect for the
people in Berkeley, and Anthony Levendosky in
particular, in that if you control a car, and you want to
steer it, all you do is you drag around
your steering wheel.
That's turning, right?
So we do this on the motorcycle, as Chris knows.
If you just drag around your steering
wheel, you fall, right?
Because you lose balance.
So what you do is, you steer in the opposite direction, you
start falling, and then you catch your fall.
That's the way you turn a motorcycle.
So working this out in the control center is quite an
achievement.
But these our videos on the background.
We had a lot of fun doing this.
And we weren't the only ones practicing this maneuver.
Carnegie practiced the same maneuver just before the race.
And we had also our problems. So we had a day when we came
into the Mojave Desert after a major rainfall, and the road
was basically turned into a lake.
And just by driving over that water, it would splash up and
you couldn't see anything.
And the only thing you could see something where the lasers
on the roof that were making all the decisions.
So we were sitting inside there realizing if There's a
mistake in the software right now, we won't even be able to
see it before we die.
That was the only moment where you could really say that the
robot was better than human driving, just for this very
short moment.
And clearly, the humans were insane to accept that.
Well come race time, just to finish up here, the race
itself took place in Fontana Speedway, and
then in Prim in Nevada.
Fontana was an event to basically
select the 20 last finalists.
So we came in blue T-shrits.
They are because Carnegie Mellon picked red as a color,
so we couldn't pick that anymore.
So we looked like Berkeley guys.
We had two cars.
We had the full support of Volkswagen of
America, and so on.
DARPA had made up a course that was about 2.2 miles long,
on the speedway.
It had all kinds of challenges, I'll show in a sec
here, that kind of resemble the challenges you'd find in
the desert.
It wasn't really desert racing, but you could claim if
the car wouldn't be able to make that course, then it
wouldn't be able to succeed in the Grand Challenge.
And so they had a narrow gate, that resembled one of the
gates on the course, very narrow.
Which already half the teams couln't negotiate that first
gate, it turned out.
And then they had an abandoned car over here, that could be
another robot sitting there while you're passing it, maybe
a disabled robot, and so on.
They had a big open high speed section where you could go as
fast as you wanted, up to 50 miles per hour, so Stanley is
going about 38 here.
That's the case where Carnegie was
consistently faster than us.
We were somewhat more conservative
on the speed side.
And then the most difficult obstacle was actually a tunnel
that they put up that emulated an underpass under I-15.
It turns out we crossed I-15 but we couldn't go over a real
highway, and they couldn't shut down the highway.
So they made us go through these really narrow
underpasses.
The problem with the underpasses is
you lose GPS coverage.
So you lose track of IR and you have to locally navigate.
So that can kill you because you lose your sense of IR, but
what can also kill you is when you emerge from the tunnel,
where you regain GPS, and then you realize you're actually
somewhere else.
And if you don't very, very carefully integrate your data,
your map would look very, very funny afterwards.
So this was an underpass.
This is an emulation of a power pole.
They wanted to test if you can go around the power pole,
because on the race course was the power support for half of
Los Angeles.
And Tether had this fear that we might just knock out one of
those, and Los Angeles would be dark and DARPA would be
famous for that, and so on.
And DARPA had to test for that.
And of course, it went fine.
Just to give a feel for how difficult it was.
This is just some random video from the
web that NOVA recorded.
This is local team DAD, which did actually
very fine in the race.
But in the beginning, they had some difficulties
with the hay bales.
and they dug their own nest. That was pretty much it.
They had a fantastic laser scanner on the roof
that we're using now.
Here's our motorcycle again.
And be prepared for some amazing--
They got the control worked out but not the
perception, it turns out.
This is the most amazing moment.
It is raised from the dead.
The ghost rider bravely forged on.

And you know some people had an interesting solution to the
GPS problem like these guys, when the went out of GPS, went
to full throttle, so in the, tunnel, they accelerate to 60
miles per hour.

And I mean, we had to get an ambulance for the Carnegie
Mellon person sitting there.
This is one of the very few videos that Carnegie Mellon
did something foolish going over hay bales.
They had an argument on this.
Not us, but Carnegie Mellon did.
The centers are miscalibrated.
The fixes afterward were where we went just fine.
But in the beginning, it his the straw and the judges said,
no, you're going to lose points for this.
And Red said no, I'm not going to take hay bales.
Didn't see the problem
And then there were lots and lots of different teams that
just didn't make the first gate.
Or they made their first gate, but didn't make the second,
run into a car.
And you might laugh, but it just shows how hard it is for
these distributive system to do this.
This is a moment where we had two unmanned vehicles on the
course and so on, stuck in the underpass,
rearranging the hay bales.
whatever.
This is an interesting team.
So they had about 20 laser scanners out there.
A university team that couldn't quite decide how to
organize the software.
So every student got one laser, and one
left them in the trunk.
And in the end, they all clicked electrode together and
they took a majority vote drive.

The race itself, which--
I'm getting a little more serious here--
unfolded on October 8th, and it was a big day for us.
So here's Mike, our chief software guy, a mechanic from
Germany, and on of the chief Volkswagen people.
Yeah, we got the race course on a CD.
Yeah, we had some car problems, too.
We got to the race course at 4:00 in the morning, and we
had exactly two hour to process it.
So coming into and going to the trailer and doing the
hand-labeling 80,000 of way points.
We each just basically smoothed the course using a
lease square smoother, and then look at the data mines.
It's defined.
Well, it looks fine.
Plug into the real world, and then talk to journalists, get
some coffee, and so on.
And then at 6:30 when the sun started
rising, the race began.
We were seeded second after Carnegie Mellon.
Carnegie Mellon had beaten in us on speed in the pre-runs.
And Carnegie Mellon was in third.
They had took us in the race.
And this is a video of the race, parts of it, from this
NOVA movie here.
And it's you a video that still to the present day
evokes some emotions for me, because after working a whole
year on this car and spending so much time inside the car,
this was the first time the car went by itself.
You you see it sitting there, empty in the morning, you
train it everything you know what driving, at
least, so you think.
And now it has to to stand as his own man, as its own car.
It makes a little mistake in the beginning, a little nudge
that was wrong, but then
confidently goes off by itself.
And then in like thirty seconds, you lose sight of
your car, and it just disappears.
It's kind of a bizarre feeling.
I mean you are all young people.
And when I give this talk to older folks.
I often compare it to raising a college kid, like raising a
child, making it really perfect.
And then someday it's going to go to college in a different
city and you have to trust it to make the right decision to
come back without dents and scratches or whatever, or not
pregnant, whatever can happened in college.
It's kind of the same thing.
You try to optimize this car and really anticipate every
contingency, and now it's on its own, right?
So it made a couple of stupid mistakes in the
race, but it survived.
So this is all footage that we didn't get, but a helicopter
collected from NOVA, which made a documentary.
As you can see, Stanley driving by itself without a
person inside, lots of advertisement, of course.
We were actually quite lucky.
There were about three teams that directly took off,
Carnegie Mellon and us--
at about the same speed.
In fact, we were slightly slower than Carnegie Mellon's
fast speed.
We just saw this one over here which is a website.
And somewhere over here, Carnegie developed an engine
problem with their fast vehicle, and it slid down
backwards down the hill and never
became really fast again.
And it allowed us to castellate it on.
This is Cartech.
Cartech failed at a very sensational moment.
A little GPS glitch.
Team DAD had gotten stuck behind a rock the year before,
and not a enough throttle.
So this was a way of determining as much throttle
as they possibly could when they were stuck, so they did.
Team Ensco is another one.
They actually had a laptop come loose and lost all of
perception in mile 10 but went all the way to mile 90.
So this is the case when Carnegie Mellon got stuck, and
Stanley just right behind it.
closing the gap of originally five minutes difference in the
starting time, so Stanley is kind of informally
taking the lead here.
And it was a complete act of a randomness that Stanley
actually won.
It was really a failure of Carnegie Mellon's engine that
made us win, and no more, no less than that.
So we were for a while, chasing Carnegie Mellon.
And at mile 103, a really important moment for us, the
organizers let us pause Carnegie Mellon
and let us pass it.
The way it worked, is when you've been paused, time
wouldn't count against you.
You'd just save the time.
But that way, a passing vehicle would just pass a
static obstacle, not a dynamic obstacle.
So this is video from our car looking at that,
which we got later.
And we were just in the tent, sitting there, and all of a
sudden a voice came, Ladies and Gentlemen, Stanley has
just passed Highlander and you can imagine what effect this
had on us and on other people.
The last obstacle was this treacherous mountain pass over
here, which only five vehicles reached.
Stanley was the first to go down.
And as you can see, it's fairly dangerous.
There's a big cliff over here.
It's about 200 feet low.
It's harder than the first year's mountain pass, even
though the entire course was easier than the first year.
And Stanley has to make life or death decisions, I guess,
and to basically control its speed, to not get too fast,
and avoid the berm, and do this for a couple of miles.
This is the DARPA chase vehicle that just observes it.
You can't stop it.
And then, when it happened, we all knew there was a chance
for Stanley to actually come back, which is amazing.
So we all lined up over there in the finishing line, and at
some point you saw in the distance a helicopter, then
you saw a little dust cloud, and then you saw
a little blue dot.
And all of a sudden, you realize, wow, your car is
coming back.
After 130 miles.
It was completely unbelievable and mind-blowing.
And our car was first to come back, which was even better.
So here is Stanley going to the finishing line.
There's some rover there, celebrating this really
historic moment, I think, for all of us in robotics.

I want to comment a little bit later on what it really means
for me to have been involved.
But this is just a little video that shows our adaptive
vision an action.
And this is the only man made obstacles we encountered.
It happens to be Carnegie Mellon's car, which is paused,
so you can check whether the video actually works, and we
actually catch it.
You can see it's been classified as non-drivable.
Which a Hummer is.
I think AMG discontinued the H1 production since.
You can see the shadow over here on the map and so on,
just for amusement.
But the interesting thing is-- and I mean this very
seriously--
if you look at this chart, five teams finished, four
within about half an hour of each other.
All of these are equal winners.
I mean, it's good to have the check, the
money, and the fame.
But technically, these differences of 11 minutes are
completely insignificant, and it was competing random luck
of the draw.
We were all driving right at the speed limit, Some had an
engine problem, some didn't.
And that made the final decision.
So it's really an amazing achievement for the robotics
field, for which people said certainly, this is absolutely
impossible, and you've got five cars finishing.
Now, DARPA has begun to realize there's a new
technology in the make, that actually has some hope for
society, and created what's called the Urban Challenge.
I'm not going to say anything about it.
You can read up on the web.
It's basically a challenge that deals with the city, and
you have to drive from point A to B to C, and you encounter
other cars along the way, and you have to interact with
other traffic.
And it eliminates one of the shortcomings in the Grand
Challenge which is there was no traffic.
So this is really a test in traffic.
So DARPA hasn't told us what city it is.
It might be Mountain View.
It might be Palo Alto.
Hopefully it's going to be some old military base maybe.
But November 3, 2007, together with Google, we're going to be
competing, and hopefully finishing.
Yes?
AUDIENCE: There's going to be live traffic?
SEBASTIAN THRUN: My interpretation is there's
going to be two types of traffic.
On is other robots, and the second
one is military vehicles.
But if it's military vehicles, it's going to be tanks, so
that if your little Touareg runs into a tank, the tank
won't even notice, right?
So you have to worry about traffic detection and so on.
Let me spend the last two slides on the big picture, why
we actually do this.
I mean it's fun to get engaged in a competition, but there's
a much broader picture behind all of this.
I told you the military perspective and there's a good
amount of utility to this, given the number of people we
lose, for example, to roadside bombs in Iraq these days.
But what drives me more than that, even, so is the societal
perspective, and what we can do to humankind to accept
driving cars.
And people sometimes don't realize this.
Cars are deadly instruments.
We kill about 42,000 people every year in the states
because of traffic accidents.
90% of those are caused by human error.
OK, so just in proportion, it's about 15 times as many
people as September 11.
And it's about as many people every year as all of the
Vietnam War for the United States.
So every year we lose the Vietnam War again,
and again, and again.
And if you don't look into this, there's many other
dimensions of which this kind of technology can help people.
Commuting is another one, right?
So, in the Bay area know about this, the beltway, Chicago,
Los Angeles.
People spend, on average, I think, 1.25 hours per day
commuting in a car.
And that takes away whatever 10% of the work time.
So if you could free that time.
Or people could still sleep, that could work,
they could do email.
I guess we do e-mail all the time.
But they could do more productive things.
They could actually make people's living more
productive, which I think would be a
great effect on society.
And there's many more dimension of which you can
turn this and spin this.
I talk about saving lives and saving money making people
more productive.
Here's the last one I want to really talk about.
I guess that's obvious, aging population.
The people who can't drive right now, drunk people, old
people, people. with disabilities, blind people,
children, and so on, could possibly drive.
This is an interesting one.
Highway throughput.
If you look at the nation's highway systems. it's many
times beyond capacity, like specifically in the Bay area
and Los Angeles, and so on.
There's almost no new construction of highway
infrastructure because people live there.
You can't take their property away, so almost all the money
goes into mass transit.
That is being invested, yet less than 1% of us use mass
transit as a way to get to work.
So if you take a picture of a busy highway at peak capacity
when it performs the best, which is if people go about 45
to 55 miles per hour, and in the picture you count the
number of pixels that are taken by car.
Those that are still free.
Only about eight to ten percent is actually taken by
cars and 90% percent is still free.
Why?
Because we're lousy drivers?
Maybe we can he control he could very well in a space to
put us on the sites, right?
So what if you could increase this to say, 16% by a self
guided car?
You've doubled the capacity of the US highway system.
You did something that otherwise would require
trillions of dollars of investments to reach, and for
which we have no current plan.
There's almost no new construction in this country,
and there's a steady increase of 3% per year of highway use,
which means 20 from now, it's going to be even more
disastrous than it is today.
So that certainly a vision that I think this technology
embraces that I think we should all care about.
These are the type things that we really pursue
as motivating factors.
At Stanford we are now completely, deeply engraved in
winning the next Grand Challenge.
It's going to be hard to do. it's harder than
the previous one.
It's going to be be driving in traffic.
if you see a funny Touareg coming through Mountain View
that acts like a drunk squirrel, stay away.
That's us.
And visit us on November 3rd, when we hopefully, do a good
job on the next Grand Challenge.
Thank you.