Inside Google's Search Office (hosted by the Churchill Club)

Uploaded by Google on 05.08.2011


DANNY SULLIVAN: It's a real pleasure to be here and doing
a panel for the Churchill Club.
We've been talking about doing some kind of a session about
Search for just over about a year.
And when they got closer and we came up with some ideas--
this panel really came about from Stephen Levy's excellent
book In the Plex.
If you haven't read it, my short review is go read it.
Not now, but maybe in about an hour and a half.
And engineer, David Bailey, is quoted as being assigned to
work in an office at Google.
And he's supposed to be working in this office with
Amit Singal, Ben Gomes, and Matt Cutts.
And he says, it's definitely the cool kids office.
And that really struck me when I read that, because I'd been
to this office many times over the years.
I typically would go into Google and have a day's worth
of briefings, and then at the end of the day I'd usually end
up talking to Matt on webmaster issues.
And we would be in his office and it would kind of turn into
this Search bowl session, which is a lot of fun to hear
all of them interacting about, well maybe the results should
be this way or this particular issue.
So this kind of session is kind of bringing that office
to life and some of the things that go on with it.
If I had to describe the areas that each of these men
oversee, when it comes to Search, Amit would be the
brains, and Ben would be the looks, and Matt
would be the brawn.
So I hope you're happy with your choices in life.
Amit oversees the ranking algorithm at Google.
How Google decides what content should be shown in
response to searches that happen.
Ben overseas features that help you search better, as
well as the user interface look that lets you interact
with Google Search.
And Matt's the bouncer.
He's the chief of police.
The person in charge of keeping the people who would
spam and pollute and harm Google Search results and
bring disorder there under control.
And they're each going to tell you a little bit more about
We'll start with Amit because he's the baby of the group,
having been at Google for only ten and a half years.
AMIT SINGAL: Thank you, Danny.
So what Danny's referring to is the fact that of the three
Googlers you are seeing up here, I joined Google last.
These two were already there, and I've been here ten and a
half years in a twelve and a half-year-old Google.
So that's the composition of the office.
We have been there for ten and a half years together, eleven
and a half, and somewhere in their for Matt, and we have
worked together since the day I arrived.
I have had a fairly long academic background before I
arrived at Google.
I got a Masters in Search and then a PhD in Search.
Yes, there is such a thing.
And this is back when it was the sleepy field that
librarians used to study.
And what computer scientists thought that wasn't really the
prestigious thing to do operating systems
or compilers were.
This is all geek talk.
And I ended up getting a PhD in Search and went to AT&T
Bell Labs, which became AT&T Labs, to pursue an academic
career, and published a lot of papers and so on about Search.
And then in 1999, my good friend Krishna Bharat, who is
father of Google News, in a conference over beers told me
that he's going to this company court Google in 1999.
And Krishna was single then, and just to give you some
color on the conversation.
And I, and my lovely wife, [? Shilpa ?]
sitting there, we had one child and the
second one on the way.
And I said, Krishna, Google what?
Like, you are single.
You can afford to do this.
Google shmoogle's all going to die.
I have a family to feed.
work for AT&T. [LAUGHTER]
And next year, I was here, working with Krishna and these
wonderful people.
And the rest is, indeed, history.
BEN GOMES: So I think common thread is Krishna, because I
went to high school with Krishna.
We both had chemistry labs at home and that was our common
love, actually, doing chemistry experiments.
Now this is something that both of us, being
brown-skinned boys, don't do in this country anymore.
That was our common thread.

That our common interest from a long time ago and both of us
went into computer science.

He told me he'd interviewed at Google, and a lot of sharp
people there, and seemed like a fun place to work.
I was like, sure.
Why no.
I could be there, too.
So I interviewed there.
I got the job.
And I remember my boss at Sun asked me, he said, do you
think Google is ever going to be one tenth the
size of Alta Vista?
I was like, I am not sure, but you know it looks like a fun
place to work.
And his boss made fun of me in the group meeting saying, they
bring to work for a company that has an exclamation mark
in their name, because Google, if you'll remember, used to
have an exclamation mark.
Both of them now work at Google.
And the scuttlebutt in the valley at the time was Google
had a secret plan, because it clearly wasn't Search.
These are smart guys.
there's no business in Search.
And so I remember when I first joined Google, I asked Urs,
who was my boss at the time--
Urs Holzle, the VP of Engineering--
so what really is Google's plan?
And when he told me, it's Search, I was so disappointed.
It turned out to work out OK though.
MATT CUTTS: Hey, everybody.
My name's Matt and I was actually a
computer graphics guy.
I was working on my PhD at the University of North Carolina
at Chapel Hill--
yeah, woohoo.
A couple go Tar Heels--
and got a job offer to go work for Google in 1999.
And my girlfriend at the time said, well that sounds like a
really good opportunity, but I'm not willing to move all
the way across the country without a wedding ring.
So we actually eloped and drove across the country.
Took a quick honeymoon.
Showed up and the very first project that I got a Google
came about because my manager stopped at my cubicle and
said, Matt, how do you feel about porn?
And I said, well it depends.
Why are you asking, exactly?
And so they wanted me to write a familY-safe version of
Google, which is called Safe Search.
So I wrote the first version of that.
And in the process of doing that, I found out that there
are bad people on the Internet.
And so for the last 10 or 11 years, I've been dealing with
bad people on the Internet.
And of course, there's a huge number of good people as well,
but that's been a lot of what I work on.
DANNY SULLIVAN: And Matt has a corn field that he can wish
those bad people into.
You don't want to be one of them.
We're going to dive into some questions.
Just a really bit of background just to set the
stage on some of the technical stuff.
It won't be that much of a technical issue.

Search engines, the essential pieces of them, there is a
crawler that goes out and find pages from all over the web
and stores the content that this crawler finds in what we
call an index.
It's like a big book of the web.
The crawler's constantly updating this index, keeping
all the pages fresh.
Some of them it visits a lot, maybe every few hours, maybe
every few minutes depending on how frequent that page.
Some it goes back as it needs to.
Your personal blog, maybe it only needs to
drop by once a month.
When all that stuff is put into this index and we come
along, we do our searches, that's where the search
algorithm kicks in.
And it goes and it flips through that giant book, if
you will, and it finds all the pages that it thinks are most
relevant and should be shown in response to a search.
Now, for me, and for others, when you talk about what an
ideal search engine should be--
and this is how the big search engines
are generally operated--
they are akin to being a newspaper.
You have editorial content that is supposed to represent
what somebody feels as a fair representation.
Some of the best stuff that they can put together that is
being shown to you independent of any advertising influences
or any sponsorship that's going on out there.
You can get ads but they go off to the side, just like you
might get in a regular newspaper.
If you follow through with the newspaper metaphor, search
engines have a much more incredibly
hard job than a newspaper.
The newspaper will assemble this content each day.
They're in their office building.
Maybe they're getting phone calls that come in.
They're getting story tips or whatever.
For a search engine, it's as if they're in the middle of
Times Square and they are surrounded by people shouting
things at them saying, I got a great story for you.
I got a great story for you.
And some of those are great ideas, but the people don't
shout very loudly.
And you need to hear better and bring them forward.
Some of them shout very loud and they're terrible ideas.
And some people are just purposely being misleading.
And in the midst of all this, you have people who are
literally just dumping garbage all around
where you're working.
Just junk all over the place.
So go out there and come up with some good search listings
off of that.
With that kind of background, all these things going on,
people actively trying to mislead you, people who don't
even understand that when they've rendered their entire
web site built out of, say, flash, it was
kind of hard to read.
How do you figure out who to trust, who not to trust?
How do you go about that core part of the job, coming up
with the right listings?
AMIT SINGAL: That's a great summary of how search engines
operate, Danny.
Imagine you have a book with billions and billions and
billions and billions of pages.
You do have an index like you have at the end of a book,
where it would say which word appears where.
And that's the index, a very similar index that most search
engines, including ourselves, built.
Now when you type a query into Google, you give us a few
words and we use that index to find the few most relevant
pages for your two, two and a half, three word query.
This is clearly a deeply scientific process.
It's incredibly hard as a science.
Has been studied as an academic
field for over 40 years.
And at its heart, there are simple heuristics.
Simple heuristics like if you said Danny as your query, then
a page that contains the word Danny many times is probably
more relevant than the page that contains Danny just once.
That's the first simple principle of search.
And then there are other principals like words like the
are not very important so don't use them as boldly as
you would use a word like Danny.
And you take such simple principles of how language
behaves, and you build algorithms based on those
principles in an environment that's the modern web, and
that's with the basic
description of a search algorithm.
Now clearly, in the real world, everyone wants to come
up number for the query Danny Sullivan.
And therefore, they give us pages which say Danny Sullivan
one thousand times, where as Danny is so polite.
He only uses his name one or twice on his website.
So in the real world, all these assumptions, all these
principles are challenged like we are fighting through all
this complexity to get to Danny's page, which
we do get our there.
So that's roughly how it works.
So we have taken simple principles of how language
behaves and coded it in into algorithms to return to you
what we believe are the most relevant pages on the web.
That's the words of the web, according to our algorithms.
MATT CUTTS: And in the same way that all good things in
moderation, according to the ancient proverb, the first
time or two you see Danny Sullivan, that's great.
But if you see it a whole bunch then it starts
to turn a lot worse.
And so you have to realize that the web is like a giant
torture test. People will do crazy things on the web.
Any program you write to parse a web page will break by the
time you get web pages that are long, that are
complicated, people don't close their tables, and also
malicious or deceptive people.
And so you have to worry about not just ambiguity, like
there's a Danny Sullivan who's a race car driver, turns out,
but also just people who want to rank for everything.
And so it is a really difficult problem.
But through those sorts of principles, those computer
programs, those algorithms, rather than having a single
person saying I think this ought to be number one or I
like this result for that.
It's much more scalable to have the computer
programs and say, OK.
They can work 24/7.
They don't have to sleep.
It tends to work a lot better in different languages.
All the sorts of things.
BEN GOMES: And I think one of the early insights that Google
had was that the link structure of the web was going
to be really useful for this.
And so the page rank algorithm, which was at the
heart of Google so early is such search was actually
fundamentally leveraging the link structure of the web to
actually find out which is the real Danny Sullivan, in
addition to all the other things that we were doing.
AMIT SINGAL: So you can imagine when you're going on
the web and someone says, click here to go to for Danny
Sullivan's home page.
That link--
hyperlink as we call it--
is a tiny recommendation for Danny.
And we take these people's voices which are encoded on
these web pages as links and we have built an algorithm to
actually harness the power of the web.
These voices of authors who had built this web to build
whats Google today.
DANNY SULLIVAN: Now the links especially were the big claim
to fame when Google came along.
People had been using links, but you really took
it up to a new level.
And suddenly you could find this needle in the million
page haystack that was happening.
And over this period, if you go back to say 2000, it's
largely what I'd say is the Google decade.
You were this unquestioned leader when it came to search.
That was the gold standard.
And then towards the end of last year and through this
year, it's been really amazing the amount of attention that's
been focused on search quality.
We had the sunglasses merchant who the New York Times
profiled because he was convinced that because he was
such a rotten merchant that all the bad reviews were
giving him links that helped him do better.
We had complaints that people were saying that, wow.
I've written this content somebody has simply copied it,
or scraped it, and put it on their own website and now
they're out ranking me.
I'm the originator.
You're not finding my original content.
There were concerns that what's been dubbed content
farms-- and won't get into the debate about what those are--
but people were concerned that there were these content farm
things and they were flooding Google
with low quality listings.
And then suddenly it seemed like there was this turn that
oh, well Google is just terrible and all
your results are junk.
And you've reacted to that with a series of updates and a
series of changes.
The merchant prompted this unprecedented five-day shift
in your algorithm that I'd never seen happen so quickly.
You've had other updates on the scrapers.
Probably the most talked about thing had
been this Panda update.
Google often gives nicknames to when they make these
algorithm changes.
The most recent algorithm change they had
was nicknamed Panda.
So you're making these changes.
How do you decide what's broken, how to fix it, and
assess whether or not your fixes are working?
AMIT SINGAL: This is a very good question.
And the heart of Danny's question is how do you decide
if I should change my algorithm in this direction
and is that the right change or not?
And what we do at Google is that have a few principles on
which we operate this search team.
And the first and the foremost principle that we have put in
place is do what's best for the user.
And once you use that principle and develop science
then to measure what's best for the user to the best to
scientists' ability--
I've been a scientist in this field for 20 years.
Ben and Matt have worked here for almost a dozen now.
So what they have done internally is developed very
rigorous scientific methodology to evaluate if
turning of an algorithm right is a good thing or turning it
left is a good thing, and by how much is it going to do a
good thing for the users or not.
And it would be really good to set up the
context of the team.
How we operate.
Let me just use a very, very simple example of how search
And I'm going to use a dumb example to just prove a point.
Suppose one day an engineer walks up to me and says, Amit.
I'm going to promote all pages that have pink background.
Matt says, no.
Blue backgrounds better, OK?
What does this engineer do with this idea?
The engineer goes and writes an algorithm that incorporates
his or her idea, which either promotes pink pages or
promotes blue pages, depending on who won the
argument in our office.
That's usually Matt.
And then that algorithm is unleashed in a sandbox, which
we have of the entire web inside our offices.
Not really in our offices.
They are in data centers, but only
available to our engineers.
No users are subjected to that sandbox.
And the engineer runs this blue-page promotion idea in
the sandbox.
So the new algorithm is blue page ranked higher, other page
raked lower.
Now you have the new algorithm and the current algorithm.
The two ranking systems. We take these two ranking systems
and put them through amazingly rigorous testing.
Rigorous testing involves things like we will take
hundreds and thousands of queries from our past logs,
run the old algorithm, run the new algorithm.
We will show it to an independent human being
outside who doesn't even know what
algorithm is being tested.
And sometimes we do this on them so that they don't
develop a favorite, left or right.
And computers keep track of whether left was the new or
the right was the new.
And statistically after you have run thousands of queries
through this left- right blind test, first number emerges
that says, no, promoting blue pages is not such a great
idea, Matt.
It was a thought.
AMIT SINGAL: So, there we go, right?
Now that's the first test you have to pass.
Once you have passed that level of testing, we take a
tiny subset up over live traffic and we take this
algorithm and we put it outside in a real data center
where users are doing their queries.
And we take a very, very tiny subset of over live traffic
coming into that data center and subject it to this new
algorithm, if it was good.
The blue one didn't make it to live testing.
And if users are now liking the new algorithm, where their
liking is described by they are clicking much higher in
the ranks in that relevant documents relevant documents
are ranking higher, and they are spending less time
searching through results and click on multiple results--
they find the right result quickly with speed--
then the new algorithm's better.
And with these tests in mind and this much statistics
behind every change and we make about 500 changes to our
algorithm every year, and we run 20,000 plus such tests
every year.
With this level of rigorous scientific testing, we put a
statistical report together put by a statistician, not
limited to the engineer or his or her team.
That report is brought to a group of people, all three of
us included, to make a decision whether this
algorithm should be launched to the world or not.
And in that committee that you may have about in Stephen's
books and articles, is where a whole lot of debate happens.
It's not just these two numbers that will force you to
launch something.
There are many more considerations that go into
launching a system like, is it good for the web
ecosystem at large?
Would it benefit authors?
Would it benefit high-quality content?
Would it keep our system simple so that we can maintain
it much longer?
And with all these considerations put together,
that committee says, yes, we should launch this or not.
Hey, what did I miss?
MATT CUTTS: Very good description.
And there's a couple things you might not realize.
One is that there is no way to change a computer program, to
change how Google ranks and scores things, that's going to
be perfect.
That's going to make every single query better.
Because we get over a billion queries a day.
And if you get hundreds of millions of queries, you could
make 500 of them great and only one worse, but you're
never going to make all of them better.
So there's always this tension.
There's always this trade off where you're trying to have
the goals of the best results for users, the best quality,
helping the web ecosystem.
And then there has to be room for intuition and experience.
So, for example, I work on spam.
That's people who tried to cheat.
And it's definitely the case that users often click on spam
because they see something that looks enticing.
Oh, this is what I looked for.
And so we've seen examples where one person will click on
the same spammer eight times in a row.
And you just want to--
how can you do that?
After seven times you couldn't tell this was not good stuff?
And so by the looking at just the raw clicks are just the
raw statistics it might look like this
is a horrible change.
So you have to take those factors into account.
But absolutely, the statistics and those launch reports carry
so much weight because they've built up this intuition about
what are good changes and what are not as good for users.
BEN GOMES: And some ways this is essential
for us moving faster.
Because if you don't know that you're making progress, you
can constantly end up second guessing yourself and so on.
So being very rigorous is essential to our speed of
moving fast. So we are constantly sure that we are
doing the best thing we can for users at any given point.
And we go through the same kind of rigor for our
interface changes, too, because those are sometimes
even more complex to evaluate because there are more moving
parts in the system.
So the live experimentation and so on.
There we augment it with user studies where we have a
usability lab when we watch people use the product and
make sure that it actually works in
practice by watching users.
But they also go to the same level of rigor for all the
interface changes that we make.
AMIT SINGAL: So that's a long way of saying we are in the
process of scientifically determining what's the best
thing for the user.
This measurement in itself is a science.
We are in the cutting edge of that science.
And that science is evolving as well.
But to the best of science's ability to predict whether it
change is good for a user or not.
We use that science to make changes to Google.
DANNY SULLIVAN: I'll come back in a bit to the determining of
the relevancy and if you've got it right.
But Matt, I want to pick up on what you
mentioned on the spamming.
You are in this hostile environment.
What's the conference that you have?
It's like information retrieval.
Information retrieval, which was a science, where people
would be like, how do we get our stuff out of the lexis
nexis database nobody's mentioning. and then you have
like these hostile information retrieval--
There's an entire conference called
adversarial information retrieval.
It's where the goals change because the people are trying
to cheat and deceive you.
DANNY SULLIVAN: So you're in this environment.
How do you measure up what's wrong?
How do you prevent getting the wrong person?
You just said that you've got people who will maybe click
eight times on a spam thing.
You have to use some intuition.
But then some people make click eight times
on the right thing.
DANNY SULLIVAN: So how do you do that?
MATT CUTTS: Absolutely.
So we try to provide very clear guidelines.
If you search on Google for quality guidelines, we
actually have instructions for publishers and webmasters
about the sorts of things that are good and the sorts of
things that are not as good.
And hopefully they make sense because we want to judge the
same page that a user sees.
And so, by that principle, you shouldn't hide white text on a
white background, you shouldn't show us a page about
cartoons and then show hardcore porn to users, things
that you would think would be intuitive.
And so what's good and bad, what's the curse, is that once
you know how to see spam, you will always
see spam in any system.
You'll look at the cheaters, you'll find the people.
You'll recognize the people who are
trying to game the system.
And the nice thing is that most of the
time it's very clear.
Spammers tend to be lazy, and they tend to go all the way
out and try to get as much traffic as possible as quickly
as possible.
And that leaves some footprints in some ways that
you can spot.
And what's tricky is when people get more
towards that gray zone.
So, for example, we have this category of stuff
that we call web spam.
And we have very clear guidelines.
And some of the stuff that happened in the last year
regarding content farms was stuff that you might consider
just outside of the guidelines.
They didn't necessarily keyword stuff.
They didn't necessarily do something horribly bad for
users, but it was still really low-quality content that
regular people would complain about.
And so by just stepping barely outside that zone, it fell
between the cracks for a little while.
And the nice thing about Google is that by being in the
same office, you can turn around and you can say, hey.
is this your job or is this my job?
OK, we'll tackle this.
And that has worked very well.
At Google is also the concept of a war room.
We try not to go for big war-like metaphors, but that
one actually dates back a long time ago.
And whenever you have a crisis you say, OK.
Get everybody in the same room.
And that makes such a difference for collaboration,
such a difference for teamwork because if somebody is a
minute away, you might walk three or four times a day to
check in with them.
But they are in the same room or you can see through the
glass walls, and our offices have glass walls at Google,
you can see if they're at their desk.
You can walk right over to them.
You can see whether they're looking unhappy looking at
their computer.
And so you can say, hey, is something broken?
Do we need to fix something?
And that really does make a big difference in
productivity as well.

DANNY SULLIVAN: Mercury news had had this
article about, hey.
I've been penalized by Google and I've had this big network
and I didn't even know it happened.
And we've gone back and forth on this before.
I'm like why don't you just tell everybody
if they have a penalty.
You guys will tell people if they have some penalties, but
you won't tell everybody.
So why not just say, hey.
You know what?
You're doing something bad and I will report it to you in our
Google webmaster central system.
MATT CUTTS: Absolutely.
So Amit talked very well about algorithmic search and the
vast majority of what happens is all involving computer
programs. My team sometimes has to take manual action.
Because if you type in your name and you get off-topic
porn, so you write an angry email to Google and say, I
would not like this porn result showing up for my name.
I've never been in a porn film in my life.
And we write back and we say, well it's going to take us six
to nine months and we think we might have an algorithm that
might help, that's pretty discouraging.
So my group is one of the very few where we actually are
willing to take manual action.
And so we've been trying a communication experiment this
past year where if we have taken manual action, you can
do is known as a reconsideration request, so
it's basically an appeal.
And we will tell you whether we have taken manual action
against your site are not.
Now if an algorithm is ranking your site
lower, well I'm sorry.
With over 200 million domains there's no way we can talk
with every single web master or publisher one-on-one.
Literally everyone at Google would have
to do customer support.
There would be no one left to actually run the computer
programs and write new algorithms. So we think that
that's a relatively good compromise in that if there
has been manual action you can now start to get information
about that.
You can say, here's what's different.
Here's what's new.
Please let me back into Google.
I've taken off the hidden text or whatever's involved.
DANNY SULLIVAN: And if you're logging in it's become more
broad, more of the thing.
But there's still some things you just we're not
going to tell you.
MATT CUTTS: Well previously we hadn't revealed everything we
knew because there are some really bad guys out there.
Al Gore has had his web site--
DANNY SULLIVAN: Al Gore's bad.
MATT CUTTS: Al Gore's not bad, but his web site has been.
Donald Trump has had his website hacked.
And so there are a lot of really malicious people out
there that will install malware, viruses, spyware,
trojans, whatever you want to call them, stuff you don't
want on your computer.
And you don't want to clue those folks in.
So it is a tension.
But we've absolutely been moving more towards
As much communication as we can figure out how to do.
In fact some Google employees in this room have worked on
trying to improve that process.
And have really done a great job of it.
DANNY SULLIVAN: Now people not doing well in Google have
sparked complaints.
And in fact, people not doing well in search engines have
sparked complaints since before we
even had search engines.
And there's been various things that have gone-- the
most common complaint that I've heard in my time has
been, well you're not ranking me well because you're trying
to get me to buy an ad.
But lately, now it seems to be that the reason that you're
not ranking people is because they're all
competitors to you.
And that to preserve the Google monopoly, you're
blocking them off.
So what's the deal?
You guys are wiping off competitors?
MATT CUTTS: So that the nice thing about working in search
quality is we don't worry about ads or revenue at all.
We have a very clear mission of doing what's
best for the user.
So that's not in our area or scope of
worrying about at all.
AMIT SINGAL: And so there's a clear church and state
separation between search and adds.
No matter how much money an advertiser pays Google, and
that kind of goes into our revenue.
They cannot improve their ranking.
That's fundamentally how it works.
And then the question that Danny poses is hey, how are
you now putting this stuff up?
Because I'm competing with you.
And you have demoted me.
This is stuff, your stuff.
That's what they call it.
And I go back to our first principle.
Do what's best for the user.
Our job is to give users the answers to their queries.
What they ask for is what we need to answer.
Now in a most simplistic form, if the user types the query
two plus two.
Should we return a list of pages that have the words two
plus two on it?
Or should we say, four.
What would you expect?
If you are writing a search engine, what would you do?
Take the next example.
When someone says the query 1600 Amphitheater Parkway,
Mountain View California.
Would you not show them a map pinpointing exactly what
they're looking for?
Our job is to answer user queries.
And that's what we do.
Everything that we do-- someone types GOOG.
They're looking for what's the stock price of our company
today, at this point.
And we return that value right out there.
When you work with our first principle that it's all about
the user, and our job is to answer your queries.
Everything else falls in place.
BEN GOMES: I think we think about it in terms of the time
it takes you from the time you enter your query to the time
you get the information you need.
And it's going to be a lot faster for you to see the
number four over there when you typed in two plus two.
So that is our goal.
To get to the information, the answer you need
as quickly as possible.
And that guides--
we believe that's what's best for the user.
And that guides our decision making.
MATT CUTTS: And if you go to the very extreme, if someone
comes to Google, and types in poison control.
You really want them to get the phone number for poison
control as quickly as possible.
So you want to get them that answer.
Whatever it is they're looking for.
DANNY SULLIVAN: So that gives an answer.
Some of the direct answers.
And then you get into this issue of people saying, well,
I just did a shopping search.
And instead of you showing me, listing a bunch of shopping
search engines, you're sending me into Google Shopping or
Google Product Search.
So now you're just trying to keep yourself there.
AMIT SINGAL: This definition is actually somewhat absurd.
If you look at Google Product Search, that you're talking
about Danny.
It takes pages out there on the web.
It's just a search index system.
It organizes that information better related to your task.
If your task is to figure out how much does something cost?
How well is it rated?
Should I buy this?
Is this merchant high quality?
And Google Product Search takes information that's out
there on the web and merchants can feed it to us for free.
For free.
There's no charge to anyone who is in
Google Product Search.
They can feed us their prices, their availability, and so on.
And at the same time, they can tell us what web pages they're
selling that item on.
So Google Product Search is just a different lens or a
different interface search that's far more effective for
query [? and ?]
And yes indeed we send users to that interface.
From there, they can do their research, and go to that
merchant to complete their transactions.
BEN GOMES: In the end they are still going to Amazon or
whatever that merchant is.
AMIT SINGAL: I think you can squint at it hard enough, and
say this is Google's own stuff.
But the truth is, it's all the pages out there on the web.
And merchants out there on the web, feeding us
information for free.
MATT CUTTS: Well and Danny, you had actually made a really
neat graphic a few years ago.
Because at one point, you might do a
search for Tom Cruise.
And then if you want a picture of Tom Cruise, you had to
click on a tab, images.
And if someone types in sunset, or daffodils, or
roses, you might have learned over time that people actually
want pictures of a sunset, or a daffodil, or roses.
And so Danny had made up a cool graphic that was like
Google in 2015 with tabs all over the place.
Look for people, look for whatever.
And really what people want is they just want to type
something in and get something useful back out.
And they don't want to have which of the 32 different
options of lenses do you want to search through?
DANNY SULLIVAN: Now if you go to see the purest form of a
search engine, to me, it's I've done the search.
I've clicked.
And I've gone outbound.
And so from the search engine to a destination.
And that the search engine itself is not a destination.
And when you talk about like with shopping search.
When I've heard these arguments, most of them don't
hold up with me because I think ah yes, Google sent you
from Google to Google Shopping.
Wherein you still left Google Shopping and went to a
destination merchant.
But you get these tricky issues where Google actually
hosts content and indeed becomes a destination.
Cases like Google Books.
Or Google Places, where you are aggregating and
consolidating a lot of information.
So if I do a search, rather than going outbound to the
merchant, I may go to Places.
And perhaps the best example of this is YouTube.
Where I do a search, maybe I'm going to get a YouTube video
that's coming up, and you are a destination.
So there's inherent conflicts in that.
How do you deal with that?
How do you deal with those conflicts?
AMIT SINGAL: So we deal with our conflicts with the same
first principle that we have ever had.
Test, test, test, experiment, scientifically test, and make
sure all your changes are good for the users.
And once you have that principle in place, and you're
designing your result's page, because our job is to return a
results page that's really, really valuable for the query.
You're designing your results' page and testing it
Then for the query evolution of dance, you would see yes,
evolution of dance site.
And yes, you will see some YouTube videos as well.
Because that's what the users are looking for.
So we keep going back to our first principle.
When ever a conflict arises, we go back to our first
principle: is this good for our users?
If it is, we'll do it.
MATT CUTTS: And at the same time, it makes perfect sense.
I remember AltaVista used to be where you'd search for a
person's name like Jeff Dean.
And it would say buy Jeff Dean on eBay.
And you couldn't really buy people on eBay.
So it wasn't a very good thing for the user experience.
If we were always showing something-- like every single
time we showed a result from Google Books, that would be
really annoying.
So we know it's not in Google's best interests to
annoy users, to do things that are bad for users.
Because then they get turned off.
There's plenty of other places to go and get
information on line.
DANNY SULLIVAN: But as these other things have expanded,
have you ever thought, I wish we didn't have that?
Because my life would be simpler.
Because then I wouldn't have to deal with
these kinds of questions.
AMIT SINGAL: We like your questioning Danny.
They're fine.
But fundamentally, right, every time we build
let's say Maps.
It was the most innovative product when we launched
Google Maps.
The first product that allowed you to scroll the map in using
Ajax technology at the time.
Everyone else followed suit.
To the degree that now, if you land at a page which has an
embedded image of a map, you try scrolling it right there.
So we feel proud to build these innovative products.
And when we build these innovative products then
giving users answers right on the result page is absolutely
the right thing to do.
So clearly our job is going back to our first principle.
Giving users answers to what they asked for.
And sometimes we have to lean upon innovative products like
Maps to answer a query.
BEN GOMES: I think the core part of our competency is
actually ranking a variety of these
different information sources.
I mean it started with PDF's.
When on the first time, we started
crawling PDF's in 2000.
PDF's are long documents.
And so they can begin to dominate all the results.
And so as soon as we started crawling PDF's, we saw results
full of PDF's.
So we had to deal with this issue of [UNINTELLIGIBLE]
content there.
And I remember.
I was working on crawling and
[UNINTELLIGIBLE], and also ranking.
And then I realized, well we just hired a
world expert on ranking.
Maybe I should ask him.
And he has not yet even come to Mountain View.
I remember calling him up to say, Amit, what you did your
thesis on this stuff.
What do you do?
And so I remember talking to him about this.
AMIT SINGAL: So Ben calls me.
And he says, hey, we just started crawling PDF's.
And they're long documents.
They have the word over and over again.
For every query I'm seeing, just PDF's.
For the query IBM, I'm seeing PDF, PDF, PDF, PDF.
And you have written a dissertation on how to deal
with varying document lengths in such systems. And I had
joined Google in its New York office back in 2000.
And Ben was in California.
This was around late 2000.
He calls me up.
We were driving back home from Jersey City from having dinner
at a friend's house with our two kids in the back.
My wife was driving the van.
And I'm talking to him over the phone, solving, giving him
formulas that basically solved the document length problem.
And at the same time like trying to keep the kids quiet.
You know, take it easy guys.
It'll be OK, fine.
To the degree that we didn't pay attention to what was
happening to the car, and we ran out of gas and had to call
a tow truck.
MATT CUTTS: So maybe focus matters a lot.
Whenever you're trying to figure out these apples and
oranges and how to blend.
DANNY SULLIVAN: Matt, you alluded to this earlier.
But you don't manually control the rankings.
If there's results that are showing up, it is all down to
the algorithm.
With the exception of a tiny little experiment, we won't
get into right now.
But you just haven't done that.
So things show up.
And there have been times when--
the best example I think was one of the first examples
where you would do a search for Jew.
And you got this site called Jew Watch that came up.
It was a hate site.
And people were like, how can you allow this?
And in the end, you left it up.
And this goes on all the time.
There are these things here that a lot of people would
have a consensus of saying, this sucks.
You shouldn't be listing that.
Why don't you take it out?
So why don't you take it out?
AMIT SINGAL: Let me take that.
Right Danny's referring to a principle that we have held
dear to us in our search team.
Which is that we would not manually promote, demote, or
remove results.
Even if our judgment is saying our algorithms are doing the
wrong thing.
The extreme case in point was query Jew.
When someone did that query, it made all of us tremendously
sad to see our algorithms fail by putting an anti-semetic
site at number one.
We were in pain.
We were really, really hurt by this.
This is not what our principles are as individuals,
as a team, as a company.
And people said, why don't you just blacklist that site?
And get rid of it?
However, we said no.
It's our algorithms going wrong.
And we will find a solution to this algorithmic problem by
algorithms. Because this judgment was so clear-cut in
this case, that everyone's instinct would be to go
blacklist that site.
But in the real world, not everything is
such black and white.
There are lots of slippery slopes.
There have been many other queries.
Where various interested parties have wanted us to
shoot results out of Google's search engine.
One such query is Scientology.
Where we have both perspectives return in our top
two or three results.
And clearly one group doesn't want the other
perspective to be there.
If we start intervening manually, first of all, we
would make such arbitrary judgments.
Number two, we may paper over one hole in our algorithm,
which would not fix similar problems in many more queries
that our algorithm may be causing, and in many more
languages that we don't even read.
So our principle that we have held near and dear to our
heart is we let our algorithms reflect the voice of the web.
And sometimes our algorithms will be imperfect.
And we work hard to make them better and better.
But our subjective judgment, though mostly right, is not
always the right thing.
And that principle has actually served us well over
the last 10 years of us doing that.
And that is reflected in our users coming back to Google
and liking what we do.
BEN GOMES: I think that's a principle that's come down
from Larry and Sergey.
In fact, with the query "Jew," shortly after the problem
arose, Sergey said, we will not be changing this.
And then the web changed, and actually the
ranking changed around.
And he came back, and he yelled at us
like, why did you change?
And he's like, no, we didn't change anything.
The web did change, briefly, and then it
flipped back again.
But over time, the algorithms have gradually improved to get
rid of many or most of those sorts of problems.
And the side effect of that actually, like you referred to
briefly, is that we have been very good in languages we
don't even speak, because we've relied so much on
algorithmic approaches.
Because otherwise, we'd have to do the same thing in every
language in every country that we're in, and that's an
impossible task.
A consequence of taking this algorithmic approach is that
we can then be excellent in small languages, in languages
that otherwise are not
considered maybe that important.
And it just works across the world.
AMIT SINGAL: Yeah, a normal company would set their
priorities based on revenue in a language.
We don't pay attention to that.
We say, our job is to serve every query everywhere, in
every language, whether it's in Swahili or in Hindi or in
English, of course.
And this algorithmic approach allows us to dish the best
results for English.
But the same algorithm, since languages are not that
different in general, when you dive deeper-- there are lots
of differences that I'll not go into right now--
but that algorithmic approach, then, any algorithmic
improvement made in English not only serves English.
It serves Japanese, Swahili, Hindi, you name it.
And the entire search system in this world gets better.
And we feel proud to have had this principle.
DANNY SULLIVAN: I'm sweating, because I've got five
questions, and 10 minutes to get them in.
So we'll go into lightning round.

You've got, also, listings that have impacts on people.
Rick Santorum would probably be very happy if you could
make an algorithm change for searches on his name.
But you could argue, well, he's a public figure, did
certain things, upset other people.
Maybe that's just what happens.
But you've got people who are not public figures, who are
not known in any way, do searches on, say, their names.
And they go, I can't believe this thing's coming up, and it
needs to go away.
This is terrible.
So why not let them take that stuff out?
MATT CUTS: It's a tough call, because it's a
slippery slope, right?
Whenever your faced with a he said, she said sort of
situation, you really don't have the time or the staffing
or the judgment be able to make the right call in every
single situation.
So normally, we say, look, Google is attempting to be a
reflection of the web.
If there's someone who's important in real life, we
want to reflect that on the web.
And if someone is libeling you in real life, you can either
send them a cease and desist, you could get a court case.
There's lots of ways to take care of that.
And online, there's lots of great ways to get
your message out.
You can start a blog.
You can tweet.
You can be on Facebook.
So there's many, many ways where you can have other
search results that can show up, rather than just this one
negative search result.
But at the same time, I would say we feel the
Because we hear complaints, we hear people who are unhappy.
And it's safe to say, with so many users and so many
queries, there's always someone who's unhappy.
Or there's always someone who you can point out a really bad
search result.
And so, I can't speak for these guys, but a lot of the
times, I go to bed at night thinking about how can I fix
that problem.
Or you spend that time in the shower thinking, OK, how is
that going to get fixed.
But as long as you have hundreds of people behind the
curtain who are thinking about how to improve search quality
on all those different dimensions, then in general,
it tends to go pretty well.
AMIT SINGAL: Let me just briefly add, we have spent a
substantial amount of our careers devoted to search,

and we think day in and day out about search and how to
make it better.
If someone has seen some content about them that they
don't like, we hurt.
But then, there is that case when you are searching on a
doctor, who you learn had his license revoked three times.
Surely that doctor doesn't like that result.
But our responsibility's bigger than that.
And that's the principle we operate at.
MATT CUTS: Because it always comes back to the user.
What is in the best interests of the user?
DANNY SULLIVAN: Now, I talked earlier that you've had these
attacks on relevancy, and we've got to improve things.
And you've made these sorts of changes.
But one of the changes, and it's not even new, you've done
it for a couple years, to try to increase the relevancy of
results has been to personalize things.
So where someone is based out of, if you're doing a search
here versus down South, you're going to see maybe slightly
different results.
Your search history, the kinds of things you searched on,
sites you've been can come in and influence it.
So we have these personalized results.
And now, you've got people like Eli Pariser and his
Filter Bubble saying, well this is terrible, because now
you're just showing us things we all like.
And we're not getting the diversity.
So what's your reaction to that?
Do we have this kind of a bubble?
Should there just be normal results for everybody?
Is this is a harmful type of thing?
AMIT SINGAL: So you can just imagine, a normal result for
searching for a restaurant in Newport Beach would be all New
York City restaurants.
That would be a normal result, because by population and
masses and consumption and usage, that's
probably what wins out.
So we do personalize results.
And there are cases, like restaurants, where we
personalize them tremendously.
Or weather, we personalize weather results
to where you are.
And there are other cases, like the query
"banks." Guess what?
If I started returning Lloyds Bank in the US, US wouldn't be
very happy.
Or if I returned Bank of America in the UK, they'll say
something snarky about us.
So there is that level of personalization which makes
results tremendously relevant.
And then the type of personalization that Eli's
Fliter Bubble talks about is the view personalization, that
I'm just getting my view in first. And the truth in the
search is that we are returning relevant results.
But our algorithms are so finely balanced to return some
relevant results for you, and some results that have the
opposing point of view, as you have seen time and time again
in the controversies that we have just talked about, that
our algorithms are tremendously balanced to give
you a mix of what you want and what the world voice says you
should at least know.
I can give a personal example, the query "Lord's." You guys
are thinking Dungeons & Dragons
and swords and whatnot.
But I'm an Indian who loves cricket, OK?
And there's this cricket ground in London, Lord's.
For me, that's the first result.
However, all the swords and Dungeons & Dragons are right
there, number two and three.
And that's, I would say it's an ideal page for me.

I'll go with you on that.
I speak a little cricket.
I'm going through a Sophie's Choice of which question to
throw away.
You don't.
You all go.
No, I've got two, and then we'll go into the Q and A. And
we should make it fit.
So we all not only have personalization of results, we
have socialization of results, which is sort of a subset of
that personal, who you know, who you're friends
with, and so on.
You've been using social search for about two years now
and having that influence things.
But now you have your own social network.
So how is that going to have an impact?
In particular, Google Plus and the +1 system, on the search
results that people are seeing?
AMIT SINGAL: That's a very good question.
So Google was a pioneer in launching social search.
We were the first one to launch it, where based upon
who you know, we can bring their recommendations to right
top of your search results sometimes, or we can just
bring them to your attention.
And this is great.
Because if I'm searching for a plumber or a locksmith, I
would much rather have these businesses reviewed by one of
you, who I trust, as opposed to the New York Times Square
problem of everyone yelling, I'm the best locksmith.
Please hire me.
So social search is a great advance in search technology.
And social search is based upon who knows who, and who
knows about what.
And when you put these two things together, a very
powerful system emerges.
You basically marry the real world, how the real world
functions, into search.
And that search is more relevant, because we can bring
to you experts, and sometimes experts and friends you may
know, who may know something about what you want to know.
And that's a much better system than it is today.
With Google Plus, we would indeed--
Google Plus has been a very successful system.
We are very excited about it.
And with Google Plus, we have much more data on who you know
and what they know, so that we can improve your search
experience going forward.
DANNY SULLIVAN: Have you notched up the social dial a
bit, since it's launched?
AMIT SINGAL: We experiment all the time with numerous things.
MATT CUTS: It's early days, but it's pretty exciting--
DANNY SULLIVAN: That's Google for you guys.
MATT CUTS: --to look at the potential of how well this
might work over time, so.
DANNY SULLIVAN: OK, I'll squeeze in this last thing,
and then we'll go to the Q and A. And also, because we also
want to talk about some of the team building stuff, I want to
catch you after the end of the Q and A. Come back to you
about some of the things that have helped you.
You started as a team of 10 people
around a Ping Pong table.
And now you've got hundreds of people all over the place, and
yet you still make it together.
So we want some ideas on what people can take from that.
But Ben had-- you had mentioned a statistic when you
were talking, something like two humans only agree on
relevancy 80% of the time.
And that in fact--
I think you had said this, maybe [UNINTELLIGIBLE] said
it, but-- and that if it was a year later, and I showed you
the same set of results that a year before, you had rated,
you'd only agree with yourself 80% of the time.
So how do you know if you're relevant?
I know you mentioned the humans and things that are out
there, but can anybody know?
And can we independently assess whether or not you're
getting it right, when you make these kinds of
assumptions of, this algo treat will
make everything better?
AMIT SINGAL: Yeah, no, let me go to a sports analogy.
In soccer, you have tens of games in a season.
And if you lose two, you're out.
In baseball, you have 160-plus games in a season.
And winning or losing a single game isn't really going to
determine whether you'll make it to the World Series.
Just look at our Giants.
And what we do is, by giving thousands of queries, and each
query to multiple human beings, we can statistically
measure the inter-human being agreement on relevance that
you mentioned, which is about 80% in general.
I say this is relevant, with 80% probability he will say
this is relevant.
So we can measure that and incorporate that inter-human
agreement into our deep scientific measurement that
I've talk about.
And once you know that, then you can rig up statistical
tests to say, have I done enough samples through to
definitively say with 95% confidence, things like
t-tests and so on, that the new system is indeed better.
DANNY SULLIVAN: But does it skew a bit?
Because you ask these people, who are not experts on the
things that they're searching for, to tell you whether or
not they think things are relevant, so?
AMIT SINGAL: But time and time again, this 80%
statistic holds up.
So for your query, I'm 80% accurate.
Thus, for my query, you're 80% accurate.
And just knowing that and incorporating it into our
evaluation formula allows us to do the right thing.
BEN GOMES: And the live data actually has--
we have huge volumes of live data, even when we do tiny
experiments, right?
And so there, you can really make use of strong statistical
techniques to make sure you're getting the answer right.
And there, the person who's looking has a
real information need.
And so you get a good determination of whether they
are being satisfied.
MATT CUTS: But I think the part that might frustrate you
is Google has spent 10 years trying to figure out how to
evaluate potential changes.
But we haven't talked as much about that outside of Google.
And so to have a third party to say the relevancy is better
of this search engine or that search
engine is really difficult.
And unfortunately, a lot of people will do three queries,
and that'll say oh, OK, I found my homepage.
Therefore, this is the best search engine.
And so there hasn't been as much rigor in
the third-party analysis.
A lot of it is reduced down to metrics about who happens to
have more growth and queries this month, or
something like that.
DANNY SULLIVAN: OK, we're going to go to the open Q and
A. And I'm also going to ask you about where you think
things will be in five years from now when we get to the
end of that, too.
So I'll save the last five minutes there.
If you have a problem with ranking on Google, if you
could move to this side of the room.
And we'll get those questions to you.
So we have our first question over here.
In the last year or two, I've seen more and more of what
I'll call type-ahead, where Google attempts to finish my
typed query.
And I think I've seen search-ahead, where I actually
get results before I hit return.
Can you comment on how that's worked from your perspective
and where it's going?
BEN GOMES: Yeah, so we started doing Google autocomplete
about four or five years ago.
And the idea there is that we still think of the search
process in terms of the time from you have an information
need to the time you're seeing the information you want.
And so autocomplete actually helps you formulate the query.
Because if you get the query formulation wrong, if you've
got a spelling mistake, if you've got the wrong words,
it's very hard for you to get into the right information.
So autocomplete was a first step in helping you formulate
your query and essentially greasing the
rails to your answer.
Now the second step is, the best way that you can evaluate
whether this is the right query, is if
you can see the results.
So the idea behind Google Instant was to begin to show
you the results as you type.
So you can begin to evaluate whether your partial query is
what you wanted.
And we find that people get relevant results well before
they are finished, several seconds before they finish
typing their query.
And so, we are seeing a lot of benefit to users from actually
showing them results early.
And we've been very, very pleased with some of the
impact it has on the time it takes the users to get the
information they want.
AMIT SINGAL: Ben is being modest. He led Google Instant.
And Google Instant has been a huge success for us.
We are very proud of what we have done there.
MATT CUTTS: And in fact, Google obsesses about speed to
the point where we just launched something where, if
we really think it's likely that you'll click on the
number one result, then we might pre-fetch that,
pre-render it, so by the time you click on that result, it
shows up instantly.
You don't even have to wait for that to download.
So there's hundreds of people at Google thinking about every
aspect of the search session.
And how can we give you bionic arms to search faster, and
bionic brains, and all that sort of stuff.
And the goal is to get you to what you want
as quickly as possible.
AMIT SINGAL: They didn't buy my Google helmet idea with
DANNY SULLIVAN: Next question.
AUDIENCE: [INAUDIBLE] what you've done to earn your
success over the years, day-to-day in your development
[INAUDIBLE], how does it affect you knowing that
competition authorities around the world, including the FTC,
may well be [INAUDIBLE]
looking over you shoulders, oh and [INAUDIBLE]
expensive and time-consuming to have to keep up with other
companies, [INAUDIBLE]
reputation [INAUDIBLE]
yet it's always a roll of the dice.
What is it you hestitate to do?
What is it you don't, that doesn't get done?
Because of this [INAUDIBLE].
MATT CUTTS: Yes, that's a great question.
I think, having worked at the company for 10, 11 plus years,
the core essence, the principals of the company have
remained the same.
Because it's almost like Spiderman, right?
With great power comes great responsibility.
And I like to think that we've operated with the idea that
someone could be watching over our shoulder, and tried to
make the right decisions all along.
Now, you're right.
It can be a roll of the dice.
And so, some of the things that we've been doing are
trying to educate people.
How does search work?
What are the different policies?
So for example, we've made over 450 different videos that
talk about common questions and answers.
And if we can do a good job of explaining, yes, there is some
manual stuff, whenever there's spam.
But the vast majority of it is algorithmic.
And explain how these are best practices.
And how every search engine does them to some degree.
Then hopefully, that decreases the odds that there will be
some miscommunication, and some fluke
will result in a problem.
We have a very good legal team, and they think a lot
about all of these issues.
But I like to think that the core values at Google are such
that, we try to behave as if we were the underdog.
We try to figure out, how can you step into the other
person's shoes?
So how can you have a good appeal process?
How can you scale up on communicating?
How can you scale up on transparency and
But you have to be mindful that, as an engineering
company, we always have to have the idea
that we can be wrong.
We are a very disruptive company in a very disruptive
space, which is technology.
And so you have to be mindful of that responsibility.
And if there are people who have suggestions for ways to
improve, you absolutely want to be open to that
and listen to that.
Whether that's someone you meet at a conference, someone
who's Tweeting to you, or whether it's
someone who's a regulator.
DANNY SULLIVAN: Next question.
I had a question about what you do to deal with it the
quote unquote bad guys.
For instance, I understand searches are generated as a
result of an algorithm.
But, for instance, J. C. Penney, a couple of months
ago, that New York Times article about how
it gamed the system.
What did you do to affect their rankings?
And what can a company like J. C. Penney do to gain in the
rankings afterwards?
MATT CUTTS: Yes, absolutely.
So some of the Googlers who worked on that exact thing are
a couple tables away from you, as a matter
of fact, right now.
So for people who didn't read the New York
Times story, this was--
J. C. Penney had engaged an SEO company, a search engine
optimization company.
And there are certain things that Google prefers not to see
in our search results.
So if you are buying links that pass page rank, it's
almost like a form of payola.
It's like you're paying someone to say nice things
about you, but you're just paying them to link to you.
And so, that's something that's against our guidelines.
Just like you're not allowed to do payola on the radio.
And so, whenever we see a violation of our guidelines
like that, we are willing to take action.
We absolutely try to write algorithms to spot that,
counteract those links, to defuse and detect them so that
they aren't an issue in the first place.
And strangely enough, we'd already detected those links
several times.
I think, if there was a failure in that particular
case, it was that we didn't escalate, and take stronger
action earlier.
But then the flip side is, once the company tries to
clean things up-- and J. C. Penney did a very good job of
trying to make sure that they were doing ethical search
engine optimization after that incident happened--
then you have to make allowances and say, OK, how do
we let you back into the search results?
And at all times, the goal is to try to make sure that
you're ranking appropriately in the search results.
So it is a tricky problem in that, in some cases, judgment
does come into play whenever you're looking at violations
of our guidelines.
But the vast, vast majority of the time, you want those to be
handled with the computer programs, with the algorithms.
So that you don't have to bring that
judgement into play.
BEN GOMES: Our goal, in general, is to reflect the
real authority of a web site.
The goal of search in general.
It's not to be ahead or behind it.
And when somebody's trying to game the system to be ahead of
their real authority in the world, that's when we have to
take action.
DANNY SULLIVAN: Next question.
AUDIENCE: Thank you all for a great panel so far.
Ben Parr of Mashable.
So you had this cool product called Google Realtime Search,
where you could be searching, getting real-time updates and
feeds through Twitter, Facebook, et cetera.
Then the Twitter deal expired, and it disappeared.
And this is a question for my Google Plus
audience, by the way.
So the question is, when, or if, is Google Realtime Search
coming back?
And do you need Twitter for that, or is that going to come
primarily from Google Plus data?
AMIT SINGAL: So what Ben is referring to is, we launched
Realtime Search based upon data from various real-time
data generation, information generation systems like
Twitter, Facebook fan pages, and so on.
And our deal with Twitter expired.
We didn't come to an agreement.
And after that, we decided that the value that product
was providing was not enough for our users,
and we took it offline.
And we are actively working on-- as we speak--
figuring out, using our current G+ data and other
sources that are out there, to revive the same functionality
into Google search results.
So I could say, stay tuned.
We are working hard on it.
DANNY SULLIVAN: Why didn't we get--
and why don't we have now-- just, the ability to search
Google Plus?
You're a search company, and we don't have the ability to
search your own social network.
AMIT SINGAL: Your feedback is very well taken, Danny.
Believe me.
And again, we are on it.
DANNY SULLIVAN: We are always thinking.
MATT CUTTS: It's fair to assume there's always lots of
things we want to do, and a finite amount of people.
DANNY SULLIVAN: Next question.
AUDIENCE: With the addition of Plus One search results, does
this offer a new way to game the system, where people can
plus one, plus one, and get their searches results to the
top, or how does that work?
DANNY SULLIVAN: You can buy those
now, I've read somewhere.
MATT CUTTS: Not to be cynical, but every change in search
involves a potential way to game the system.
I've gotten a little jaded over the years.
But, the thing that's actually nice is, if you think about
ranking, as it existed a few years ago, it was primarily
based on links, anchor text, what's on the page.
So we have over 200 different signals.
The idea that the web might move from anonymous pages in
some dark corner, where you don't really know who wrote
what, to a web where you actually do have some sort of
annotation that says, this person wrote this, or this
person vouched for this page, and you can have the
reputation of individual people or authors--
so if someone who's a New York Times columnist shows up on a
forum and leaves a two line reply, that can be really
important, even if nobody links to it.
So it's absolutely the case that people will try to game
social signals, Google Plus, all of these things.
You already see people trying to sell plus ones.
But there's different ways, where you have new signals,
and different ways to intersect that, and hopefully
prevent that.
And the idea that you can get a big win from all of these
potential new signals is absolutely
worth giving it a try.
AMIT SINGAL: And no signal is used in its absolute form.
Every signal in its absolute form has its shortcomings,
like you mentioned for plus one.
We crossed hundreds of signals to build what's Google today.
MATT CUTTS: You can certainly imagine having fun like, oh
we'll by some plus ones, and then see what shows up.
You can play games like that.
DANNY SULLIVAN: The plus one data right now-- it is used as
a ranking signal for when you're logged in, correct?
AMIT SINGAL: When you're logged in, it is a social
signal if someone you know has vouched for something, indeed.
DANNY SULLIVAN: And if you're logged out, is it being used
as one of many signals?
AMIT SINGAL: We are experimenting with numerous
things, always.
DANNY SULLIVAN: Which is Google speak for yes.

AUDIENCE: So my question comes from the standpoint of image
and video searches.
So if I was to search for a moonlit beach, and if the
images or the video wasn't tagged with those specific
words, are you guys doing some--
advance the art of video and image search, which can get me
moonlit beaches, without those words being tagged?
AMIT SINGAL: So our image search algorithm is far more
sophisticated than just looking for those words,
either in the caption or nearby on the web page.
There's a lot of computer vision technology built into
our image search algorithm.
Some of the team members are here today.
Which, basically, use those images to say, hey, this one
looks like a beach, and so does this one.
And now, you have a positive loop.
You can find some great images of beaches.
And then you can find some other images of beaches that
didn't really say beach, or didn't see beach with the same
density as our algorithms would have liked.
So it's a good question.
We use a lot of that technology.
It works very well, and we are constantly improving it, as
you saw with our most recent launch of Search by Image.
An image comes to you, you say oh, what's that?
You just drop it into Google Image Search and, with very
high likelihood, we'll find it for you, what
you're looking at.
Using the same vision algorithms.
DANNY SULLIVAN: Next question.
AUDIENCE: I come from the mobile industry.
And I'm sure that mobile phones have different kinds of
search queries.
I'd be interested in a comment on that.
But the next thing that we think is coming is
machine-to-machine mobile communication.
And I'm wondering if you're preparing for machines to be
generating queries in that way.
And how different do you think machine queries will be then
people-generated queries?
BEN GOMES: Well, I think the first point is not actually
that accurate.
The mobile query stream is getting to be more and more
like the web query stream.
A while back, it was different.
When phones were really slow, when interfaces were not full
web browsers, the query stream was very different.
But today it's gradually approaching exactly the same
distribution as the desktop query stream.
And so, to answer that part of the question, I think,
actually, that things are much more similar that you think.
AMIT SINGAL: And let me also add, to that part of the
question, with the innovations that we have made at Google,
with things like voice search, where-- it's hard to type on a
mobile phone, so we gave you voice search.
Amazingly accurate voice search.
And we start seeing human beings behaving the same way
as they behave on desktop.
I would say that on mobile devices, we still get somewhat
more local oriented queries, what's near me.
And for that, we have launched numerous innovations.
Like on Google's homepage itself, there's a list of hot
things you can do.
There's restaurants, coffee shops and so on and so forth.
So mobile query stream is actually reflecting what users
need, which means it's coming closer to the natural
distribution of queries.
And mobile has been a great success for Google with all
the innovations that the search
team has made in mobile.
Not only by voice search, but things like, our buttons are
more touchable on a mobile interface.
Our maps are far more designed for this tiny interface.
We have done immense amount of work on mobile, and that
reflects in our success in mobile.
Now, the question about machine-to-machine searching.
Right now, we haven't observed that much of it happening.
And I think, once that system picks up, we would have to
analyze how that looks.
But in an ideal world of search for me, search would be
so accurate that you can just type a query, and the machine
should assume that the search engine, the other machine,
would answer it correctly, and then build a whole equal
system on top of that platform.
And that's our dream platform that we are building together.
DANNY SULLIVAN: Next question.
AUDIENCE: This may be a follow-up on the bionic arm
question, or a comment there.
Which is, I've heard people in Google talk about search list
search, which I interpret to mean that, maybe you can even
take the personalization to the extent of anticipating
what the user might want, without even typing a query.
So I'm curious about your points of view on that, and
what are the challenges that poses to you?
AMIT SINGAL: So it's a great question.
Because this kind of technology is what we, as
kids, dream of, right?
Computer would tell you what to do.
And-- the truth is--
MATT CUTTS: Maybe not tell you what to do.
But help you understand what might be possible.

AMIT SINGAL: And the truth is, first of all, you can see that
in a future that's possible, based on pieces that we
already have, you can build systems where a computer can
help you tremendously in making you a far more
efficient human being.
So my phone already has my calendar.
It knows when I'm free.
It has my to-do list, which says I have to
buy a baseball mitt.
And it has a map, based on Google Local, of all the
places that sell.
And it knows where I am.
So it's not too far out there that you can imagine that
computer can gently prompt me, hey, please do pick up
baseball mitt.
You are three minutes away from Sports Authority right
there and you have 30 minutes free on your calendar.
MATT CUTTS: And by the way, it's kind of
annoying I have to have--
yes, that would be nice.
It's kind of annoying that I have to have a to-do list at
all, right?
Because Google has announced something
called Google Wallet.
And wouldn't it be great if you could go to the grocery
store and you could buy things with your Google Wallet?
All you do is, you tap to pay.
And then, over time, if you wanted to, and gave
permission, Google could say, oh, you haven't bought cat
food for six weeks.
Normally you bought it every four weeks.
Do you want to just add cat food to the list?
And then, finally, I don't have to think about, oh, I
need A1, or I'm out of salad dressing, or whatever.
All these little traces we leave, if you're willing to
opt-in for those kinds of things, Google could be the
little tap on your shoulder that's like, hey, don't forget
to get wet cat food.
AMIT SINGAL: And the key there is that users have to opt-in
to these things.
It's a critical aspect.
I am a human being who deeply cares about my privacy.
And that's the key part.
Everything we do at Google, we think about that.
Sometimes we get it wrong.
We stand up, apologize, and move on.
But that's how these systems and technology will evolve.
If someone told me, 20 years back, that you would type into
some machine, what's the height of Mt.
Everest, and it'll spit out the answer, I'll go, you're
smoking crack, buddy.
Go on.
But, see, what's possible with technology--
we all have to dream it together, and then build it in
a privacy-preserving way.
DANNY SULLIVAN: If only you guys could answer our email,
that would be a real solution.
Next question.
AUDIENCE: Brian Fox, Western Union.
First of all, Google has to know that my cat died last
week, because I blogged it.
And you shouldn't send me that.
That'll be your job.
I'm curious about your vision--
your image recognition technology.
Do you use the power of the people doing the queries, when
they confirm that the beach really was what they were
looking for?
Does that loop back into your algorithm and improve its
AMIT SINGAL: Numerous factors go into deciding, in image
search, what images are most relevant, and
most liked by users.
And there is that positive loop that, based on users'
choices of what we return, we can improve the
system going forward.
So when you combine the powerful research algorithm
based on words with the power of vision algorithm and this
wonderful loop, that's when you get what's Google create
Image Search today.
AUDIENCE: Lauren [? Chaude, ?] also with Western Union.
We're a 160 year old company, and I'm intrigued with the
dichotomy of culture, which I care about.
So you're obsessed with statistics and algorithms, and
yet you talk about glass walls, and the power of being
one minute away from each other, and being physically
Can you talk about that in this global world, and web,
and blah blah blah, all that stuff?
Thank you.
MATT CUTTS: Yes, absolutely.
It's been enormously helpful to be very close to the
relevant people.
And Google has offices around the world.
And so, for example, numerous people in our office will have
a video conference unit right there, where the glass wall,
they can look behind them, or they can look through the
glass screen and talk to someone in New York.
And so, with very little work, it's easy to bring up somebody
and collaborate, from Tel Aviv, from
New York, from Zurich.
And that makes a huge difference.
You always have time zones, but just being able to have
that face-to-face connection makes a huge difference.
Short-circuit an email conversation.
Hop into some sort of face-to-face communication.
It really saves a lot of time and a lot of
AMIT SINGAL: And we have found this proximity of teams--
it makes us tremendously efficient.
We can churn out things, as Danny pointed out, in one
case, in five days, we could launch an algorithmic change
at this scale.
And this proximity doesn't always have to be physical.
And by video conferencing--
and in our office, there are two or three video
conferencing units that are open to the
world, all the time--
and people can just say, hey, can you unmute?
I need to talk to you.
And there we go.
We have a conversation.
BEN GOMES: And I think, in the history of Google, we went
through periods where we were very densely packed.
But we also found-- and that was not by design, it was just
that we hadn't gotten more space--
but we also found those were extremely efficient periods in
the company.
Where people were packed into an office and they
communicated a lot more than they otherwise would have.
It might not have been their first choice, based on their
Many people came from backgrounds where you had your
own office, and so on.
But it created a kind of energy that I don't think
arises otherwise, without that kind of density.
MATT CUTTS: There are some companies where every
developer has their own office.
Whereas Ben--
there was one office as that had three Bens in it.
And so they called it the Ben Pen.
But it really does make a difference to be able to just
turn around.
And we have all these cues, right?
You can do heavy-duty video conferencing.
You can do a hang out in Google Plus now.
You can do Google Chat.
And there's these cues that are subtle, like, well, I'm
red, but you can interrupt me if it's
really, really important.
All the way down to email, and meetings, and
those sorts of things.
So having that spectrum to be able to choose what's best to
get in touch with someone, whether it's something that's
a quick hit, 20 seconds, or a half hour meeting, that really
makes a big difference in terms of being able to
AMIT SINGAL: In the early days, we used to say, we pack
them tight and give them deodorant.
And that's how it works.
DANNY SULLIVAN: And we've got time for one more question.
AUDIENCE: So Shailesh from Citrix.
So thanks for the panel for wonderful
insights into the search.
So my question is about, when I do my search, most of the
times, I get results back saying that, 20,000 results in
two seconds or so.
But thanks to your excellent algorithms and principles,
first two or three links I get my results, many times.
So why bother spending time for finding and searching for
the 19,000 plus results, and giving them to us.
Can't you save time there?
AMIT SINGAL: So you have seen that number sometime appear on
Google's result page that says we have
20,000 or 200,000 results.
But the truth, indeed, is what Shailesh said, that if you
haven't found what you're looking for in the top two or
three results--
which shouldn't happen that often, or
you can send us mail--
then really, going down further is not that useful.
It's just that our algorithms do compute numbers for all
those results.
And so we give the user an indication of
how much there is.
But that doesn't mean you have to read 20,000 and feel pained
about that.
BEN GOMES: I think, in some ways,
it's a historic artifact.
There was a time when, when you did a misspelling, you
would get much fewer results than the real query, right?
And so people used to use it as an indicator.
It's no longer true today.
We correct your spelling, and so on.
But it's a historic artifact that has, I think, a little
bit of nostalgic value for us, too.
MATT CUTTS: And by the way, I'll give you one tip, which
is that, it is an estimate.
It's not an exact count.
So if you ever notice, we only give three significant digits
when we guess how many results there are.
That's a little cue to let you know, it's not really 982,000,
it's roughly 982,000.
DANNY SULLIVAN: And I'm grimacing, because I know I
could still do a search where I should be getting a smaller
number, but I actually get a bigger number, so--
MATT CUTTS: It's a rough estimate.

DANNY SULLIVAN: The last two things.
I want to come back to the team aspect.
So you've talked a little bit about it already, you can
network, but--
MATT CUTTS: Yes, there is one metaphor that Amit mentioned,
which is the baseball metaphor.
Which is, when you've worked together for so many years,
it's almost like having that many games in a season.
You don't get that frustrated if you lose one time, or if
somebody tells you to go back to the drawing board.
Because I remember, whenever I launched Safe Search, this
porn filter, the first time, I was all ready to go.
I was already to flip the switch.
And two engineers tested it out and said, you have too
much stuff labeled as port that's not really porn.
And I had to go back to the drawing board and figure out
how to make it better.
And at the time, I was [GROWLING].
I was kind of frustrated about that.
But over the course of doing many, many, many, many, many
launches, you build up that trust to where you say, this
person's looking out for the best interests of the user.
This person's looking out for the best interests of Google.
So take that as constructive feedback.
Don't get so caught up in one particular battle, one
particular controversy, because it's guaranteed,
tomorrow, there will be a new controversy.
A new point of discussion.
And that's helped a lot in making things more collegial.
AMIT SINGAL: Yes, and now I look back at my 10 years of
being part of this group-- we started around a Ping Pong
table with 10 people.
Now we have hundreds of people in our group.
And unwittingly, somewhere, we developed the principle that
has made this team out-innovate every
other team out there.
And that principle was, I would put leaders in place who
I respect technically.
So the entire management hierarchy of the group is
built of people like ourselves, who were engineers,
wrote code, can understand what an
engineer is going through.
Their happiness, their pain.
The entire group hierarchy is built from people who have
worked in the group for many, many years.
Hundreds of people report to us now, but everyone who
reports to us has managers who have been there many years.
The whole group leadership is built off purely technical
people with deep technical knowledge about search, and
deep understanding of what goes into this innovation
machine that is Google.
And that, we just put in place early on because I couldn't
find enough people to manage people as
the group was growing.
So I said, why don't--
Matt, you manage 10 people.
So Matt got 10 people.
And then, Ben, you manage 20 people.
And that's how we grew the group.
And it has served us very well.
This is a lesson in leadership that we have all learned in
We weren't designing for this.
But it so happened, that if you are in the innovation
space, you need to make sure that your leaders are so
technical, that everyone that works for those leaders
respects them as technical people.
And that has worked well.
BEN GOMES: And I think, all our leaders all stay with
technical titles.
And they think of themselves as, first and foremost,
engineers, as we do too.
Even though they do a lot of management and so on, their
self-perception is as technical engineers.
DANNY SULLIVAN: Can I just say, by the way--
your title is--
are you senior, assistant principle?
AMIT SINGAL: Principal engineer.
DANNY SULLIVAN: And your title is?
BEN GOMES: Google fellow.
DANNY SULLIVAN: And you're title is?
AMIT SINGAL: My title is also Google fellow.
DANNY SULLIVAN: Which pretty much has nothing to do with
what you actually do.
MATT CUTTS: But at Google, you can get anything you want
printed on your business card.
Literally, they do not care.
There have been some pretty crazy business cards printed.
Because, titles are--
at least, at Google, they don't make as much sense to
obsess about.
And so, if you can get whatever you want on your
business card, then you don't fixate.
You don't obsess about it.
You worry more about the job at hand, and that
tends to work well.
DANNY SULLIVAN: And then, in a Tweetable, or Google Plusable,
or Facebookable short statement, where are we at
five years from now, other than bionic arms doing all of
our searches?
AMIT SINGAL: So all of us have our views of where we want to
head five years from now.
I was raised on a healthy dose of Star Trek.
And I want my Star Trek computer.
That's what I want in five years from now.
I should be able to talk to it, ask it whatever, and it
should be able to have a conversation with me.
And search is, of course, fundamental to that.
MATT CUTTS: I want Star Trek in five years, but in one
year, I want the ability to get reminded that I need to
get cat food while I'm on my way home.
And also the ability--
Google Voice Recognition has gotten very good.
And it can't be that much harder to make a
well-punctuated email.
So I want to be able to do my email while I'm driving home,
I'm talking, maybe--
at a stoplight, or whatever--
and have it look as if I've very carefully
crafted it by typing.
That can't be that hard of a problem.
That shouldn't even be a three year problem.
So I'll go back and file a bug when I get back.
BEN GOMES: Yes, I think I share Amit's vision of the
Star Trek computer.
But I think, in the shorter time frame, I want the search
on my phone, which is with me all the time, to work really,
really smoothly and effortlessly.
And it's not quite there yet.
It's gotten a whole lot better.
Voice recognition is getting there.
Transmission is getting there.
You can now do these amazing things.
Like, you can take a photograph--
I was in South American recently.
I was in a restaurant with Spanish menus.
I took a photograph of the menu.
And with image recognition and translation, I could then
translate the menu into English.
I was like, wow.
That's amazing.
And it's all of--
you can see where you're going to go.
But you're not there yet.
And I want that reality to become completely fluid,
completely reliable, so that people all over the world can
actually communicate, and get information really easily.
And I think this matters, not just here, but particularly in
other parts of the world, where people don't have as
easy access to information.
Where people in India, and people in Africa, and people
in the Third World, really, have access to this
information the way that we take for granted
in many ways today.
And we have access.
And I think that will empower their lives.
And I think that that will change the world in many ways.
AMIT SINGAL: So let me communicate our enthusiasm for
search by saying, I feel like a kid going to candy store
every morning.
And you haven't seen nothing yet.
DANNY SULLIVAN: Well it sounds like all search in the future
will be mobile.
And we won't even be searching.
It will be happening for us.
Thank you all very much for being here.
It's really been a delight.
I wish I had another three hours to keep going at it.