How Google makes improvements to its search algorithm

Uploaded by Google on 24.08.2011

Rajan Patel: The Google search algorithm is made up of several hundred signals
that we try to put together to serve the best results for each user.
Amit Singhal: Just last year,
we launched over 500 changes to our algorithm.
So, by some count,
we change our algorithm almost every day or almost twice over.
Scott Huffman: We really analyze each potential change very deeply
to try to make sure that it's the right thing for users.
Mark Paskin: The first step in improving Google Search
is coming up with an idea.
Scott Huffman: There are almost always a set of motivating searches
and these searches are not performing as well as we would like.
Ranking engineers then come up with a hypothesis
about what signal, what data,
could we integrate into our algorithm.
Amit Singhal: We test all these reasonable ideas
through rigorous scientific testing.
Mark Paskin: The first is with raters.
These are external people that have been trained to judge
whether one ranking is more relevant and higher quality than another.
Rajan Patel: We showed these raters side-by-side for queries
that the engineer's experiment might be affecting.
We also confirm these changes with live experiments on real users.
Mark Paskin: And we do this in something that's called a Sandbox.
We send a very small fraction of actual Google traffic to the Sandbox.
We compute lots of different metrics.
Scott Huffman: In 2010, we ran over 20,000 different experiments.
All the data from the human evaluation and the live experiment
are then rolled out by a search analyst.
Sangeeta Das: For each project,
it's usually one analyst assigned from the moment
that we're talking to the engineers trying to learn about their change.
And the impact is quite small, as you see.
Mark Paskin: We then have a launch decision meeting
where the leadership of the search team then looks at that data
and makes a decision.
Amit Singhal: That surely we should fix.
Sangeeta Das: Ultimately, the goal of the search eval analyst team
is just to provide an informed, data-driven decision
and present an unbiased view.
Amit Singhal: OK?
So, not approved.
The team will understand what's happening.
If our scientific testing says this is a good idea for Google users,
we will launch it on Google.

Mark Paskin: For many years now,
Google has been offering spelling suggestions for queries
that contained typos or misspellings.
So, sometimes you'll type a query and you might see,
"Did you mean," and then an alternate query.
If you type a misspelling of your medicine
and you don't click on the "Did you mean"
you might be getting results that contain that misspelling.
And they tend not to be high-quality results.
So, we thought about a different kind of interface
that we call "full page replacement."
And instead of "Did you mean,"
you'll see at the top of your page,
"Showing results for."
And in the case that we made a mistake,
there's another link, "Search instead for"
and it has the query that you typed.
We call that link the "escape hatch."
For every time a user had to click that escape hatch
because the spelling algorithm made a mistake,
we wanted to make sure that there were 50 other times
that they got the right spelling suggestion
and they didn't have to click the "Did you mean."
And they were also looking to see in the live traffic data
how often were users clicking on that escape hatch
to make sure that the user signal that we get from live experiments
was lining up with the signal that we get from our regular evaluations.
We brought it to Launch Committee
and based on the rater evaluations and the live experiments,
it was pretty clear that engineers had done what they were supposed to do.
And so we launched it.
Amit Singhal: When you align Google's interest with user's interest
as we have aligned,
good things happen.
Mark Paskin: We've put a huge investment into understanding what works for users.
Rajan Patel: Is this change gonna help users
not only in the United States or in English,
but all over the world?
Scott Huffman: I think we get excited
when we feel like we've hit on an idea that really helps a lot of users.
Amit Singhal: Users keep coming back to Google
even though they have a choice of a search engine
every time they open a browser.