The New "Bill Of Rights of Information Society"

MALE SPEAKER: It's great to have Raj Reddy here.
Now, I was doing a Google search of all things, and I
came across an interesting fact.
In 2004, Gross, Politzer and Wilczek won the Nobel Prize in
physics for the work on strong force that
binds quarks together.
Now, I also discovered that Raj Reddy had nothing to do
with that group, and he has never won the
Nobel prize in physics.
Although he has won just about every prize in computer
science that counts, including the most important one, The
Turing award, as well as the Okawa prize, the Honda prize
and the French Legion of Honor award.
Now, so why did I mention that stuff about the quarks?
It's because I personally have this theory that Raj is the
strong force that binds together the
computer science community.
And more than anybody else, I think he's the one who has
tried to make computer science get out into the lives of
people in the developing world and help them, and help us,
help bring computing and communication
technology to them.
He has also run the world's premier
robotics lab for two decades.
and the decade after that, he ran one of the world's premier
computer science departments at CUM.
I'm from Berkeley, so I have to say one of rather than the.
He's the founder of the PCtvt project, which I'm sure he'll
tell you about, and the Million Book Project.
And for the Million Book Project, he was the first, and
I believe only, person in the history of the world to get
the presidents of India and China to write a letter and
put their signature on the same letter to get them to
agree to a project.
So he is truly an irresistible force, and we're glad to have
him on our side.
And we're happy to welcome him to Google.

RAJ REDDY: Thank you Peter.
So I'm not sure I need the microphone, but
it's kind of noisy.
I'll try to use it.
So the talk today is not about the emerging world or Million
Book or a number of other things we
could have talked about.
But I came here with a mission.
The mission I have is, Google is not doing enough of our
organizing the information about the world.
And there are lots of things you're leaving behind, and I
want to tell you about them.
And then hope that some of you, the younger generation,
will actually pick them up and transform the way we access
and use information.
So this particular talk was inspired by my colleague,
Carbonell, Jaime Carbonell, who is the head of the
Language Technologies Institute, is one of the
leading figures in language processing.
For those of you don't know ancient history, the first web
crawler and search engine was built by one Jaime's students,
Fuzzy Malden, called Lycos.
The rest is history as they say.
And that was in 1994.
And so one of the things that Jaime said was when he was
trying to explain to someone what is the purpose of the
Language Technologies Institute, what will it do or
what does it do?
He said it is to provide the right information to the right
people in the right timeframe in the right language in the
right level of granularity.
And he had a few other rights, which over the period of time
we jokingly used to call the bill of rights of the
information society.
But I kept thinking about it, and I would bug him and say,
why aren't you doing all the research that needs to be done
in this thing?
He says, that's too big.
We don't have enough money to do all the things.
Not even Google has all the money to the all the things.
So the important issue is that when you look at the world's
information with this prism and say, what does it mean in
the first place?
What research has to be done to get there, and
will we ever get there?

And if you do, how will we know whether we have
succeeded or not?
So this is the challenge.
And so I don't think I have the answer.
I don't even think I'll be presenting all
the facets of it.
The purpose of my coming here and in giving this talk is to
stimulate each of you and perhaps even have a dialogue.
I don't think we can in this kind of forum, but we might
have a dialogue afterwards.
If you come to Carnegie Mellon, which is right next
door here, Carnegie Mellow West in the Nasa park, we can
have a brainstorming session.
So that's the purpose of this talk.

And there are more than five here.
You have the right information to the right people at the
right time, in the right language, in the right level
of detail, in the right medium.
And the last one is an issue that's very important if you
happen to be an illiterate person in a village.
You don't how to read or write.
Especially, you don't know anything about English.
And so you better be able to provide the information I'm
looking for in my local language, but more importantly
using audio and video, not text, OK?
And so the right medium also becomes an important part of
the research agenda that one needs to look at.
OK, so as soon as you use those words, they imply
certain kinds of research, search engines, classification
of information into the right chunks and right formats, and
right timeframe and support for the analysis.
Machine translation, summarization, speech input
output, all of these things, or video input output too.

But they're part of the solution, but
not the entire solution.
The right information just doesn't mean searching
something and displaying a list of all the possibilities.
In the end, it may be the right information without
clutter, right?
And the question of how do you get only the right
information, not everything else under the sun perhaps,
may be one way of looking at it.
And another way of saying it is, if you're a kid going to
school, the right information to you is knowing about the
things that were discussed in the class today.
Or if you're about to get married, knowing the
information where to get all the relevant wedding planners
and all kinds of the things or solutions for
that particular situation.
So the right information has many contexts and many
connotations to different people at different times.
So the right is very difficult to define, I'm finding.
And I don't what the right answer is, but I'm just giving
you examples of this.
The right people is again something that you can define.
If you're about to get hit by a tsunami, it doesn't matter
if you broadcast it to the whole country of Thailand.
It's the people on the shore that are sun bathing or
fishing or something that are going to be impacted.
They are the ones that need to get the information.
And by the way, they don't have
days or hours or something.
And furthermore, it is a push technology, not a pull
technology.
So you have to be able to figure out, how do you get the
information to them in ways that are not usually done
where I go and type in, is there a tsunami coming?
And then it tells you something about it.
So there are lots of very complicated, complex issues
that this set of rights that Jaime Carbonell proposed
seemed to raise.
And we don't yet have as a community--
language processing techniques community--
don't have a broad research agenda that covers the whole
spectrum of these issues.
That's the thesis of my talk basically.
So now what I'll try to do is take you through quickly what
we are doing at CMU, which is less than, as I was telling my
colleague, less than 5% of what we need to be doing.
And that's rightfully so because we don't have the
resources to do everything that needs to done.
So in search engines, the right information from the
future search engines.
How to go beyond the just relevance to query and
popularity I'll talk about a little bit more.
A second issue is eliminating massive redundancy.
For example, if you, say, type into Google, "web-based
email," it should not result in links to various Yahoo
sites promoting their email, not even non-Yahoo site
discussing just Yahoo email.
What it should just say is link to Yahoo email, Gmail,
MSN mail, a comparison between them perhaps.
That's much more what we call relevance rather than a
massive redundancy into the thing.
The second issue is harder to do, and we need to somehow
organize the world's information this way.
What information is trusted, and what information is just
pure marketing, and what information is snake oil?
Unfortunately, the web-publishing paradigm
permits all of them to have equal weight.
So we need to figure out how to begin to go towards trusted
information sources.
At one point, I proposed to my friend, Bob Kahn, who is one
of the inventors of the IP protocol and the internet,
that maybe we should have a Triple A web--
authentic, archival and always available information.
So how do you make sure it's authentic?
That means you have to register your information with
some trusted society.
Maybe Google could set up such a thing.
You submit your information.
You get a check sum or something.
And then every time you access this information, if you
changed anything, it'll say, no, it's not authentic anymore
but some other information.
You're not getting the original
information that is certified.
So if anybody had changed it, including yourself, then it's
no longer the original information that was certified
or trusted.
This is where the peer reviewed journals are
different than web sources of information.
And peer review journals cannot be changed.
They are printed and published, and they are
reviewed, and somebody certifies that this is
high-quality material.
And then it is there forever in that form.
They may be wrong or right.
It doesn't matter, it's there, right?
So we don't have trusted sources of
information at this point.
So I'll talk about these three subjects, maximum marginal
relevance, and novelty detection and named entity
extraction as three narrow topics within this right
information, which is what we are doing.
So most query systems simply retrieve everything that's
relevant to the keywords that you type.
Sometimes, that's not enough.
Things like novelty, timeliness, appropriateness,
validity, comprehensibility, density, medium--
all kinds of things might be also appropriate rathen than
just purely relevance to the thing.
Novelty is non-redundancy if you want to call it that.
If there are 20 different hits or 100 hits that are more or
less saying the same thing from different sources at
different newspapers at different locations, you ought
to be able to say, that's all one thing.
But let me give you five things that are really
different from each other.
And that would be a desirable outcome for the search engine.
So and there is more detail here.
I don't want to go into the detail, but I think you
understand.
So if you have a large number of things in the central
cluster there, and there lots of outliers, you want to pick
one out of each of those countries and present that
because then it'll be maximally non-redudant, right?
And that's what you're looking for.

So novelty detection is another idea that's usually
relevant in newspaper stories.
Usually, one of the things that happens in newspapers is
out of the blue, a word "Katrina" appears.
It was never there in the vocabulary before, and
suddenly it takes on a meaning of its own.
It goes on and on and on.
For a few days or a few weeks it's the top story.
And then it slowly goes away.
And it hasn't got away yet, but it will one of these days.
So the detecting of a new event turns out to be a an
important aspect of getting the right information
available in ways it is not otherwise.
Then there is a whole set of issues about how you do that,
cosine similarity, tfidf and things like that are examples
of how you might do this.
And so there are issues on how you do the first story
detection, FSD sometimes it's called.

And then categorize these topics, these words into
topics, and then see if these terms are maximally
differentiating in the topics.
And the second way of doing this is to use situated named
entities like "Sharon as a peacemaker."
So link detection is--
once you have the first story, you have detected it.
Now you need to track it over a period of time.
Why is this important?
It turns out that at any one point in time when you look at
a story like the number of people that died in Katrina--

on day one it might say seven people.
On day 17, it might say 3,000 people.
And then it maybe goes down or up just like the World Trade
Center, and the total number of people that actually died
changes every day.
So if you were trying to find out a fact of how many people
died in Katrina or the World Trade Center disaster, you
can't simply pick one of those things and
present all of them.
You need to find a way of summarizing or finding the
final number.
And sometimes it's called non-monotonic reasoning where
what was true one day becomes not true the following day.
And that becomes not true the next day.
At some point, it stabilizes, and you know a fact.
And at some point, it disappears.
You no longer have that fact, right?
So this become an interesting set of issues.
Another problem that occurs in language processing of this
kind is named entity extraction.
And this is important for the usual query type systems. Who
was mentioned, for example?

Peter was talking about the Okawa prize or Honda prize.
When I got it, I didn't know what this prize.
I got a letter saying you've been selected.
I said, what is this prize?
I've never heard this before.
But it turns out to be a very important prize in Japan, but
here in the USA, we never hear about it as much.

So finding these things like who was mentioned, what
locations, what companies, what products, turns out to be
an important aspect of finding the right information.

So here is a story.
And what you are trying to do is named entity
identification.
And there are a lot of names, and there are a lot of roles
they are playing.
And the issue is, can we have a system which identifies
different named entities and the relationships to each of
them between what it is that they're doing and how
they're doing it.
And so if you count, there are many new
techniques for doing it.
And the finite-state transducers and statistical
learning techniques are two of many.
So here are the people that were mentioned in that story,
Clinton, Kantor, Peng, Suzuki, Langford
and so on, and places.
How did you know these were people, and how did you know
these were places?

And it turns out this is not a big deal.
Once you have this thing categorized in your knowledge
base and with appropriate labels, you can do this.
But you don't always have all the relevant information.
You may come across a name that you've never seen before
or a name of a place that you've never seen before, and
it's not uncommon for many of us, when presented with a
foreign name, to know whether it's man or a woman.
And many times, we make the mistake of calling he a she
when the opposite is actually true.
So the roles that they play turns out to
be important also.
You wanted to say, who participated in this meeting?
Who is the host country?
Or who was the host?
What was discussed?
Who was absent at this meeting?
These kinds of questions come up all the time.
And it is not exactly stated in the story.
It is not explicitly there.
You have to infer it from the meaning of the language.
Most of us don't have any problem inferring those
relationships, but systems do.

So there are a number of emerging methods.

And here there is "who does what to whom" relation
extraction problem.
And this is very useful if you're going from unstructured
data to semi-structured data like in a database sense.
If you wanted to organize the information into your table,
then you need certain tools of going from unstructured
information to structured information or
semi-structured.

So those give you two or three topics that we are working on
at CMU, named entity extraction, novelty detection,
first story detection and maximal
relevance, marginal relevance.
So the next topic is "the right people." Again, "the
right people" is a very complicated phrase to define.
If a seven year old is working on a school project, and he or
she searches for "heart care" or "heart attack" or whatever,
then the kind of stories you might want to retrieve for
this right person would be very different than if you're
a doctor and asking the same question.
So each person, based on their context and situation and so
on, needs to be provided with a very different set of
information.

Once you see the example, it's obvious.
But the issue of how the system or individuals--
in other words people, how would they know what the right
information is?
And most of us have the benefit of knowing this person
is doctor and that person is a kid.
And therefore, you might be able to provide the right
information, but it's not always obvious.
And then, there's a whole set of other affiliations that
also force you to select or sub-select information in a
particular way.
If you're a family group or an organization group or
stockholder group or whatever, then the information that
you'll be looking for him and you might be provided would be
very different from each other.
So the right people, right information
to the right people--
and so the basic tool that seems to be needed here is to
somehow, given a text, classifying it, categorizing
it, and saying what is this about and how
does it group together?
Is it for a kid potentially?
Or is it for an expert?
And in-between?
So there may be many different ways you would organize this
information.
And those ways of organizing information
is not in the content.
You can;t search for it.
It's inferred information, and that's where the
complexity comes in.

So categorization, Yahoo for a long time, in the beginning,
used to do manual assignment of categories.
And Reuters, for example, used for a long
time hand-quoted rules.
And a lot of the rest of us use various machine learning
techniques, and they're getting more and more
sophisticated as we go on.
And then the issue of whether you can do a hierarchical
event classification becomes an interesting issue.

So the right timeframe.
Now, I used to think right timeframe is, as soon as you
know the information, send it to me.
That turns out to be the wrong strategy.
I only want the information when I want it, just in time
information.
If you send it well in advance, it'll sit there and
I'll even lose it.
That's the trouble with our education today.
You go to college, and you are taught calculus because 30
years later you might need it.
By that time, you have forgotten it so you have to go
back and relearn it.
So if I have to relearn it, why don't you just teach me
calculus in one week instead of one year, and then say, if
and when you need it, go to Wikipedia or something and
you'll learn the rest. Or, here is a learning by doing
method or learning calculus, and you can learn it
when you need it.
So that's the just in time information being provided
well in advance of when you might need it.
But this is not just that type of thing.
All of us get information about a seminar or about
something I'm supposed to do two, three weeks in advance.
If you're like me, which you probably are, you're probably
being bombarded with hundreds of emails every day.
And you just cannot keep up with it.
My senior colleague, and the only Nobel prize winner in our
field, Herb Simon, the late Herb Simon, used to say we
have the wealth of information but
scarcity of human attention.
We don't have enough lifetimes, and as human
beings, don't follow Moore's Law, right?
Our capacity has been constant for the last thousands of
years, and it's not going to change any time soon purely by
evolutionary means.
So the right timeframe--
defining it even becomes a very difficult issue.
So getting the information to the user exactly when it is
needed, immediately when it is requested, prepositioned this
whole issue of anticipatory information.
So for example, one of the areas of research some faculty
members have studied at CMU is what is called
anticipatory computing.
We cache results all the time.
Computers were designed going back to the '60's, the [?
Strech for example, would pre-compute lots of things and
then pick one of the right ones.
And that type of caching of the results is common in
computer science.
So the issue is if I know there are only 40 things I can
do and 38 of them are trivial and can be done
instantaneously, two of them require search or something
that may take a minute of time, and I don't want you to
wait a minute, you could pre-compute it.
But you don't present it.
You pre-compute it and you keep it.
And when I ask for it, you then give it to me.
So that's the issue here.
So things like push technologies, alerts, and
reminders and breaking news, are also
often considered harmful.
Just because the Steelers won the Super Bowl, I don't need
to know it the instant that a certain event has happened in
the Super Bowl.
I may be in the middle of a meeting.
I don't want the alert at that point.
And if and when I am ready to get it, I should
be able to see it.
So there is a whole set of things that alerts are not
always the right things to do, and that becomes an important
part of the right information.
The right language, there are lots of--
at CMU and many other places, including Google--
there are a lot of language translation tools and
techniques.
And unfortunately, all of them tend to be not that good.
And it's a hard problem.
It's been a hard problem.
It'll continue to be a problem, but there's a lot
that is known about how to deal with it.
And so one of the reasons you might want to do the
translation is for multilingual search.
There it doesn't matter if the translation is not perfect
because you're not asking the people to read the
translation.
It's mainly being used as a way of indexing and to various
materials such as trans-lingual research,
language identification.

Language identification turns out to be a
very interesting problem.
One of the things we're working on is this Million
Book digital library project, right?
That is well before Google print going back
to five years ago.
But China and India are partners US in this project,
and they are doing the scanning.
And they enter all the meta-data.
And the scanning is done by high school graduates at not
minimum wage, at some wage that I don't know about
because the government of China and India
are paying for it.
And I was looking at one book.
It was obviously a French title, but the guy didn't know
that and typed it in and said German.
I said, how could you be so stupid?
How could you make such a mistake?
So this young man was obviously super bright, who
should be hired by Google, came to me and brought me a
couple of books.
And one of them was in Gujariti and another one was
in Marathi.
So it happens that even though I'm from India, I don't know
either of those languages.
I couldn't tell the difference between the two.
So he says, you're a PhD, you should know this, right?
I said, sorry, you're right.
So it turns out that just because you're educated or
literate, it doesn't mean you know all the languages and you
know all the scripts.
But it's a very simple problem.
All that you need to see are one or two words to the title,
and then you can tell, most of the time, what language it is.
It is a sparse language machine learning problem.
Most people that do language detection go and look at a
whole page of text and then do a frequency count.
And then say, oh, this is this language.
That's an easy problem.
The hard problem is, supposing I only give you three words of
the title, can you tell what language it is?
Turns out you can.
It requires a little bit more work.
So there's the issue of regular translation like the
one that you can get from Google.
And there's also what we called reading assistant or
on-demand translation.
Let us say you know a little bit of Italian but not a lot.
And you are starting to read, but you come up against, in an
email, some word or phrase that you don't understand.
You should be able to just select it and get immediately
a translation, but one that is much better than simple
translation because now it's in context.
So most of us have this problem, especially if you're
a non-native English speaking speaker and you come across a
word or a phrase, or even if you're an English speaking
native speaker.
If you're reading Shakespeare, and there are lots of phases
there that are not used by us today, and you need to be able
to have a translation assistant or reading
assistance which will give you that.
It's an interesting sub-problem to solve and can
be solved for all the different languages.
Transliteration also turns out to be an important problem in
the right language because many languages
have different scripts.
So it is very interesting, especially coming from India.
It turns out, in India, the sounds of all these languages
are the same no matter what language, but
the letters are different.
But in Europe, the letters are the same, but
the sounds are different.
So when Unicode was invented by the people in Xerox PARC,
they thought in all the world, all you had to do was make
sure that for the same letter, no matter in what language it
is, you have the same 8-bit code or
whatever out of the 16-bit.
But that turned out to be exactly the wrong thing for
Indian languages.
You want to be able to say, no matter what language, it is ka
or ke or whatever the syllable happens to be.
And that's not easily identifiable in the Unicode
system as it's currently done.
OK, so there are lots of issues on right language.
There are many different techniques that have been
tried, statistical translation, knowledge-based
translation, example-based translation.
I'm not going to give you a tutorial on all of these, and
you can quickly find out about them.
Where things are going now is these so called multi-engine
machine translation.
It turns out no one of these techniques is perfect.
They work some of the time.
They don't work other times.
So rather than depending on the best technique using only
one, you say, can you actually build a hybrid system that
gets translations from all of them, and then figure out how
to use them.

That's the multi-engine machine translation, and here
is an example of different kinds of translation, EBMT and
semantic, sentence, syntactic [INAUDIBLE]
and so on.
So here is an EBMT example, example-based machine
translation.
And multi-engine translation.
I won't go through all of this.

You get three different translations, and it turns out
you need to figure out of a way of
selecting the right ones.
That itself turns out to be a hard problem.
How do you know this is the right set of things?
And that is determined by looking at the naturalness of
the sentences.
There are ways--
if you have ever worked on natural language generation as
opposed to analysis, this is one of the issues that you
spend time on.
OK, what we can do in new languages.
What we cannot do yet is an important set of things.
And at CMU we work on what we call orphan languages, obscure
language that Google may not be interested in or most
commercial organizations would not be interested in.
The reason we work on these is we get funded by DARPA.
DARPA wants to know how to translate Pushtu into English
or Swahili or some other language.
And that's not something that you would normally consider as
a research topic.

OK, the next interesting aspect of Bill of Rights is
the right level of detail.
And there a number of aspects of level of detail, and I will
present some of them here.

One of them is this issue of if you have a document, if I
have a report that came up on a Google search, and I want to
read it, I don't really have the time to
go through 300 pages.
If there was an automatic summarization technology that
would produce a summary for me, that would be very useful.
And there are such tools and techniques.
They're not good, but they're OK.
And people stop at that point.
It's really not adequate.
What you need is a hierarchical summarization.
You need a one line statement, a headline saying, what is his
book about?
And you need a paragraph, an abstract or something, and an
executive summary, and a regular summary.
And so you can have four or five levels of detail of the
same document.
And so one way we characterize it is we say, 10 word summary,
100 word summary, 1,000 word summary,
a 10,000 word summary.
And so information structuring here is not 10 words.
It's 20 I guess.
Headline, abstract, summary and document.
And the scope of the summary can also be different in
different places.
You can see what the issues are.

And this two by two matrix is very illustrative of the kind
of problems that you run into.
One of them is when you just go do genetic query like you
do on Google.
And there, if you're asking the question, do I want to
read any further?
What you want is a short abstract in the right column,
query-free.
And if I want a summary where I don't have the time to read
a 100 page book or a 300 page book, give me a summary of 10
pages or 20 pages, then those are the executive summaries.
But neither of them are appropriate if you're actually
looking for a highly focused response to a question where
you, say, want to filter the search engine results by
clustering them or grouping them to determine whether you
want to read any further at all.
And the query element summarization for busy people
is, I don't want to read this article.
I want an answer.
And it turns out--
I used to think this was only important for busy people--
it's also relevant to an
illiterate person in a village.
So if this person goes to a computer or pays somebody to
do a search for him and says, I have AIDS.
What do I do?
What they don't want is 10,000 or 100,000 links to AIDS
related articles.
They are not competent to read and summarize and analyze and
integrate the information that is there.
They are illiterate, or semi-literate.
So what you want is something that will actually understand
that knowledge and say, oh, you have AIDS?
You need this cocktail.
And then you say, where do I get it?
And you get another answer.
So it turns out the kinds of information that's needed when
you query a highly focused requirement, is essentially
you want the computer to solve the problem.
And so some of the types of work that Ask Jeeves and
others tried to do earlier is related to this.
And I think Google Answers might be related to it.
For whatever reason, Google Answers has never taken off,
or at least I haven't seen a lot of buzz about it, but
probably people use it still.
So maybe if we remove the cost angle, it'll be more
successful.
But solving problems is different than just providing
information.
One of this is passive knowledge, and the other one
is active knowledge, right?
So in the right medium.
finding information and providing information in
non-textual media turns out to be very important also.
And there are a number of research issues in these areas
that many of us are working on.
I'm sure there may be work here too.

So, in conclusion, the purpose of this presentation was to
highlight the kind of research problems and concerns some of
us in academia are grappling with.
No doubt, there are people at Google research and Google
engineering that are also working on these problems. And
the issue is, how do we make substantial progress?
And the only way that will happen, I think, is for us to
begin to have professional conferences in each of these
topics, maybe organized by Google.
A one day or two day or three day symposium that will
actually bring all the people working in this thing in a
public disclosure, without having to send an NDA or
something, where you can actually present the ideas and
go forward.
And that's one of the things I think we need, especially in
areas where things are still very fuzzy of what the right
way of approaching the problems are.
Thank you.

MALE SPEAKER: OK, any questions?
Comments?

Otherwise we can go have coffee.
I had a question.
You were talking about trying to get some diversity of
results that [INAUDIBLE]
from the main cluster.

AUDIENCE: But if you do that without having some good
[INAUDIBLE] capability, then people lose the idea that most
people are talking about this thing in the middle, and all
the fringe concepts are considered the same way as the
majority concepts.
You need some way to balance that.
RAJ REDDY: So the way to look at it is grouping together
relevant sub-topics into clusters doesn't imply you
have to throw away the details.
It might all be there.
But it's like Google News does, right?
It gives one headline, and one may be the main story, and
then it says 74 other stories.
And so if somebody wants to see how the same story was
reported in multiple different locations, you can get it.
But the interesting thing is in that scenario, we never
have the patience or time to go read all the 74 stories.
What I would like to see are those three stories that
provide a completely different
perspectives of the same story.
You take any of these stories when there are controversies
in Israeli or Palestinian newspapers, the same story
will have a very different way of presentation.
And that's not just there.
It can be between here and Mexico, it can be any place
where there are differences of opinion.
Between Google and Microsoft, the same story would be
reported slightly differently.
And what I want to see are those specific differences.
In what way are they different?
Because I don't have the time to read hundreds of pages to
figure out for me what the differences are.
If I had a human assistant, I'd say, you go figure it out
and tell me what the differences are.
And what we're looking for is the equivalent of [INAUDIBLE].

MALE SPEAKER: OK, thank you.