Quicksilver: Universal Access and Action

Uploaded by GoogleTechTalks on 08.10.2007

>> Good morning everyone. Thanks for coming. So people have been asking for this TechTalk
long before Nicolas even joined the company. So, here it is. This talk will be on Google
Video, so try not to reveal any confidential information in the Q and A afterwards. Else
good. >> JITKOFF: [INDISTINCT] Okay. Yeah. So, I
do still get pressured into doing this by some of the teams. I don't normally talk to
people in public, or about anything I do. But, it seemed a lot of this stuff seemed
relevant to Google so I thought that it would be worthwhile and interesting to people in
the world at large. To start off with this, I wanted to give sort of the basics of Quicksilver,
sort of introduce it to people who haven't used it before. It's kind of unapproachable
because the interface is basically nothing. It just tells you to search and there's no--almost
no indication of what else you could do. At a basic level, which most people use it before,
it's a launcher. So, I can type things like iTunes, and I can jump back and forth through
my apps pretty easily. This thing that almost everybody got a grasp of. Additionally it--it
is sort of like a--it's a mixture of the command line, and the Finder, and lets you get at
information normally you wouldn’t be able to see. So, if you, well, if you go to the
applications folder, you can use it sort of as a navigational tool. So you can navigate
into things. You could actually navigate further down in them, to sort of look and see what's
inside of it. So I go in my address book and I can see, sort of contacts there. If I choose
to like go to, iTunes and navigate into that, you sort of see the content that iTunes has.
So it's sorted by the things you'd expect to see in iTunes itself. This sort of, acts
sort of like a filed browser, like the Finder, but sort of without having to force you into
the Finder, or into iTunes, as you choose to do it. Additionally, there's this sort
of extra field. So, when I'm in iTunes, when I pick an album, for example. I can choose
to play it. I don’t actually know it. You can do the audio from there. Okay. Now, I
can use it to play albums. I will show you that it's playing. It will start playing in
the background and that's some of the basic thing you'd expect to do when you have a song.
The right-hand half of Quicksilver really is based on, what are the other things you
could do with this item. So I found this album, I can choose to play it in the party shuffle,
or I could choose to play it next. So rather than remove the track that I'm currently playing,
I could say, instead I want you to put this afterwards and sort of queue these things
up and eventually this builds a--a playlist without having to interrupt people. So these
two things basically make up all of what a Quicksilver can do. You find something you
want, and you choose what you want to do with it. Additionally, the third one that you might
see sometimes, like if I choose to mail something to somebody. I can choose the people I want
to mail to. And really, this is sort of building a grammar-like structure. Where we have a
noun, we have a verb, and potentially an indirect object for that verb. And with these sort
of basic building blocks, you can do almost anything and can chain things together. So--oops,
sorry. So these are the basic sort of things you'll see Quicksilver do. The first one is
sort of search, just for finding stuff across your computer--sort of what items match this.
But the more powerful aspect of that is, summoning something. Which is a little bit like search,
but more, "I know exactly what I'm looking for, give it to me now." When I say iTunes,
I'm speaking about a specific object. I'm not looking for iTunes in the content of some
file and, in this case, just simply getting me to that object as fast as possible is the
goal. Sometimes you don’t know about the--where the object itself is but, you know how it
sits in relation to other objects. In the iTunes case, you know where your tracks are
based on what albums they're in, or genre they're in. And, you will jump to that album,
you will jump to a genre, and then you can use browsing to sort of pivot down, to find
the item you want. This works especially well in cases with people, and we'll talk of a
little bit more about this later. The last thing is acting, which isn’t really sort
of a way of navigating, but in acting upon something, you can cause new things to be
created. So if I take a file, I can press it, I create a new object and basically navigate
from the original object this compressed version of it. And using this sort of four things
you can get around and do almost anything you try to do in an application but faster.
So, to start of with, I wanted to talk a little bit about philosophy behind Quicksilver. This
isn’t really stuff that's actually implemented, but more the general guidelines that I followed
or thought of, as I was going along. These three basic--make up the basics of it. First
are the Fast Universal Access is probably the thing that most people see and it's more--it's
mostly about how finding anything on your computer or on a web, just sort of instantly.
It's--the major point is to be able to search anything not just files or not just web pages.
And Google and Spotlight are very good at spotting--or searching those specific things
but not across both. Additionally people are also really important, and being able to search
for people and use them as, sort of, ways of finding things is important. It's--additionally
the--the place these things live shouldn’t matter. My data lives in the cloud it also
lives in my computer, and being able to search across the web and across the content in my
computer makes it more valuable. Google desktop is sort of, moving towards this by letting
you look at your Gmail and also look at your files. But really these things have to be
added on like a case by case basis. It's really hard to sort of, let you search across multiple
things. Quicksilver makes a point of indexing a variety of web services. Usually ones that
are focused on content. So things like Backpack or Stick-it, were people will store "To do"
information or just contact information they want to look up. And even at this sort of
basic level, being able to say, "I wanna search for this page I know of that is my content
and just jump to it", makes them a lot more useable. The second part of, sort of, ignoring
boundaries between things, is ignoring the boundaries of the filesystem. We talked a
little about iTunes as sort of a container for stuff. If you actually look at iTunes
in the filesystem, it's full of garbage. Like the application package itself isn’t really
meaningful and in the Finder you wouldn’t even be able to get to--in to this. But it
would be interesting if we could see what happens when you sort of drilled further down
into things. Quicksilver sort of--or abstracts the filesystem out and says, instead of showing
you what the real contents are, I'm going to show you what I believe the contents should
be. And it's done on a case by case basis for apps so, Safari shows you bookmarks, iTunes
shows you music, iPhotos shows you albums. Almost all the media apps are sort of filled
with content and, representing them as their content and not just for the dead end is the
best. If you watched Aza Raskin's talk about "Away with Applications" you're gonna recognize
a lot of stuff because our goals are pretty consistent with each other. If you look at
a lot of tools nowadays they're all focused on going places. So, Spotlight, Google Desktop,
Google.com, they're really about finding something and getting there. And getting there is half
the battle. And it's a huge help to be able to, sort of, find this thing instantly and
not have to worry about where did I put it. But once you get there, you still have to
fight with the UI or whatever app you're going into. And you end up taking all these steps
to actually accomplish what you want after having found the thing you're looking for.
And what might be more ideal is to simply do the thing you were trying to do initially
without actually having to go. In cases of Photoshop and the things are more complicated,
it makes sense to go into an application and sort of work in it for a little while. But
if I'm simply trying a jot off an email or sort of add something to my To Do List, I
shouldn’t have to interrupt what I'm doing. If you look a little bit, there's a lot of
tools that work on just making it easier to do things that you do frequently. So something
like Automator lets you build up chains of actions to specify, "Oh, I'm going to compress
this file, upload it to my web server and then send it to my friend." And if you do
that often, it's sort of worthwhile. Pipes as well lets you mish-mash data and--into
a format you can sort of look at it. But all of these require upfront effort. And you basically
have to do planning to, sort of say. "I'm gonna do this frequently make this easier."
Some of the more, on-the-fly ones, are things like the Services Menu which is probably a
bad example because it's so hard to use. I will put this in to basically let you manipulate
any content that you have. So--so I can use the--yeah, Services Menu--I should stop that--basically
look up in dictionary and it takes the context into account, takes me in the dictionary and
executes that service. One of the first things I did in Quicksilver was take that whole body
of information and include it. So this--that same action appears here and I can use it
as sort of a starting point to add functionality. Eventually, I have to sort of dig deeper,
then sort of add more data, but at a basic level the OS provides these things already.
Some other stuff that takes context into account is the Enso Launcher for Windows. It's actually
quite good at looking at what you're currently doing, and taking that into account. Rather
than saying--rather than having to explicitly ask it to define a given word, you can just
simply say define. And it says, "What are you trying to define? Oh, this is the selection
I'm going to define this for you." And if you wanted to you could overwrite it, and
write something else. But taking this--taking the current context into account, makes these
things much more flexible. In addition to taking the context into account, it's also
nice to not have to worry about--not worry or--it's nice to not have to leave the context
to accomplish something. Sort of--say I'm giving this presentation to you and I realize
that I have to get milk or something, and I'm going to add this to my To Do list, which
is sort of a simple thing a lot of people do at Quicksilver. That will simply add the
text in, but I wanted to show I could go to it and then I'll actually know what's in this
file, because I've been doing it all day. So, it adds get milk. But the idea is that
I don’t actually have to leave the presentation--I don’t actually have to leave the presentation
to do this. I can do it in context, I don't have to worry about it. And then I wanted
to email it to somebody, I could do that and just stay here, and I'm done and it'll go
off eventually. The last thing is--this--Act without effort or Act without doing, is based
on a Dallas philosophy. I sort of wanted to take the opportunity to babble on about it
for a bit, just because it was such a guiding principle in the way the app was built. "Wei
Wu Wei" basically translate as--translates as act without doing. "Wu Wei" is non-action.
And the idea is not--you don't actually do these things, but rather you work with the
natural flow of things, so that it is an effortless form of action. You're, sort of, in harmony
with the way the system works and in the end you don't leave a trace. And basically things
will--you can resolve things and get back to what you're doing, without having to do
any interruption. Now I'm gonna talk a little about implementation. So that was sort of,
the overall goals, this is more what--the what and how, these things are actually done.
Going back to sort of these high level goals, each one of them sort of dictated how the
system was built. As far as how fast you can go, really the limit on search is eventually
not going to be optimization, it's more the user input. So Quicksilver's to the point--it's
not actually searching a lot, which makes it faster, but it's at the point where the
user inputs the thing that's slowing it down the most. I can't give you the application
you want because you haven't typed enough letters for me to understand what it is. And
the only real way to optimize this is to decrease input or eventually read the user's mind.
And there's not really a clear way to do this, it's simply a matter of allowing them to feel
comfortable with what they're typing and understand that that's going to take them to the right
place. If they have to worry about, "Is this the right thing?" it's going to slow them
down. This is originally--most of this, sort of benefit comes from abbreviation matching,
at least in Quicksilver. And this is based on something that LaunchBar did a while back.
But not requiring people to type the full um, either spaces or even prefix matching
slows it down so. So, I can type APHO and I'll get Adobe Photoshop even though those
letters are spread about different words and ranking in these things--ranking these things
correctly, is sort of a fuzzy affair, that I don’t really understand how actually works.
It's all been tweaked and basically will learn from the user over time. But every time I
type APHO, I'm going to get Photoshop. And if I re-change my idea of what, that abbreviation
means I can choose a different one and the system will learn it. The whole point is for
to be flexible enough that it doesn't get too stuck on what it's been doing before,
but still will learn new things. The second thing is being able to [INDISTINCT] the second
thing is universal access. So how much stuff you can get at, and Quicksilver really has
to get into tons of applications that are spread around your hard drive, including things
that are even on the web. And it [INDISTINCT] get at the data as well as the functionality
of them so, eventually--it's been growing for a long time, sort of, what are--what is--What
is the content of this app? How can I add it to Quicksilver? What's actually useful
to search? And almost anything is valuable--is valuable to be able to search. Things like
airport networks and network locations, I sort of, added on a whim, because I thought
it would be nice to be able to just type it in. And that's not something you would traditionally
search for but having it mixed to into this catalogue is valuable. Additionally once you
start to get in to these things, to get to their data, you can also learn how to control
them and directly manipulate the data rather than simply getting at it and piping the data
back and forth between different apps. So, the initial version of Quicksilver were all
building this in and it basically started to get really, really big really quickly.
Because every new app involved the ton of the more code, and understanding all this--these
different interfaces. And it was all done in sort of shell scripts or Apple scripts
and each one of them--those scripts had to be dumped in. Eventually I realized if we
had move it to a plug-in architecture, and that's the sort of way it is now were, each
of these things deals with specific app, they can provide sort of the basic objects within
that app, but also the actions the app provides. And I learned later that people don’t deal
very well with plug-in and having to worry about installing them, and figuring out which
ones they have. But as an architecture it works very nicely. And really each of this
plug-ins just provides these different objects and builds this huge searchable index of items.
And the index doesn’t know what these items are, it just knows that they are things that
have certain types. And the plug-ins themselves provide actions that will look at these items
and say, "Oh I can handle a URL and I can email it to somebody else." And in effect
create new objects by applying actions to them. One other nice thing about the way the
system is built is that the actions themselves are objects that are in it's data base. That
it treats nouns and verbs as, sort of, the same ways. So, I can add shell scripts, and
I can add other things to make the system more powerful. It allows people to sort of
collaborate together so, a lot of this--the activity on forums, was based around how can
I sort of, mix-and-match these things or add additional functionality. Okay. So, the last
thing that isn’t--is part of the implementation that doesn’t--it sort of helped, and it's
sort of a strange thing, is that, the idea of magic and mystery. And I read a blog post
by Mike Kuniavski, that's sort of about magic in user interfaces. And eventually technology
is going to be so advanced that it doesn’t really help people to understand how works,
and simply being surprised or delighted when it always provides the right answer, is the,
like the correct approach. And a lot of Quicksilver is based on idea of just, sort of, make it
seem like it just works, and even--even forcing this area of mystery when it wasn’t worth
while. Like, it probably would have been bad for commercial app to have a web page that
was quite so plain and unapproachable. But I liked it because it sort of added to this
feeling. And it--the alchemy thing was sort of designed around getting people to think
of this is a mysterious thing that they could learn and would have to play with but it might
be reliable, but it might be kind of interesting. I don’t know. It's--I think it’s sort
of--interesting way to go and I probably couldn’t pull it off if I was trying to sell it. So,
as far as magic goes these two apps Abracadabra and Constellation, are extensions to Quicksilver
that I did, with the help from other student at CMU when I was there last year. And I wanted
to show them of, just because their fun to show off. Because the regular interface is
not particularly interesting. Anyway, so, Abracadabra basically is the simple gesture
system--let me get rid of these things--it's not particularly intelligent. It watches what
you draw and will try to match it so some pre-programmed pattern. I’m using it right
now to switch back and forth from Finder and then back in to Keynote. Which--but additionally
it was built, sort of, to work with Constellation. They are both focused on null space interaction*18:09
which Quicksilver was sort of ignoring and I really like the idea of being able to browse
these stuff in Quicksilver. So, constellation was more about, how can I show the actual
things and let people sort of navigate through them and find things visually and then decide
what they want. And this has all the same functionality Quicksilver does, were exposes
the action, it exposes the hierarchy. It's just you don’t have to know what you're
looking for, ahead of time. You can just sort of explore it. Sorry--why is it such a rush
to get this stuff out of the way--okay. So, Constellation and Abracadabra were sort of
ways that I was able to play with the Quicksilver system. And one [INDISTINCT] I talked about
little bit was what other things am I trying to do now. In building those two--in building
Constellation, I had to build it in to Quicksilver. Because all this plug-ins and stuff are bound
into the application and there’s no way to actually get at that functionality outside.
Which is a bit of a pain because Constellation is not really that associated with Quicksilver,
the two are entirely different apps. And forcing them together was just a requirement of doing
it quickly. One of the things I'm interested in doing is building a new framework called,
Alchemy, that Quicksilver's basically going to set-up on. And it will allow it to have
access to all these plug-ins but still allow other applications to co-exist with it. Where
these plug-ins simply abstract data and the things that you can do with data in a way
that other apps can load them and it will take advantage the things that are already
exist. So, building conduits to Automator and the things it does, and building conduits
to Spotlight and the data it has access to. And it's more about, sort of, [INDISTINCT]
things together in a consistent way that one interface or multiple interfaces can access.
Quicksilver itself has sort of been back-burnered just because I'm more interested in working
on this underlying framework, and seeing what new things will come of it and then applying
them back to the original. Even so there’s some stuff that I'm sort of interested in
pursuing with Quicksilver, stuff like, hybrid search. Quicksilver is really bad at searching
content or rather it doesn’t do it at all. You can do it with Spotlight but it's a hack.
And one of the things I was interested in doing, is saying what happens if I just search
everything at once and get all the results back, and I will show you the ones that I
think are most likely hits. Ignoring where their actually coming from. So they can come
from google.com, they can come from Spotlight they can come from GD, it just pulls them
together. The other thing is contextual search. When I'm in an application, there’s probably
different things that I'm interested in searching that aren’t in, sort of, the global index.
When I'm programming, I might be interested and all the source code files that are currently
open. And being able to search the methods of them. Whereas normally that would be clutter
in the index, if I was going to find those. Additionally, sort of letting people do--search
for actions. Right now you can’t search for actions. You have to find a noun and then
you can act on it. But what if you could just find an action and then specify a noun that
you want. If not, just take the current context as representing the current noun. So, I would
be able to do stuff like define and it would just say,"Oh, define what?" Like that. Yeah.
So, the last question is sort of what--how does this apply to Google? What things could
Google do, and what things are--what things are Google doing already, um, to sort of accomplish
this? The main thing that Google has an advantage of is the interconnectedness of its apps.
It’s already starting to connect its apps together. You can see this in Reader where
it’s able to do things like email directly in to Gmail from Reader as well as share things.
And there are a few other places you’ll see this, but each time we do this, it’s
very hard coded. It’s like, it’s an implementation in these--in that app. And what will be more
interesting to say is, what if I don’t use Gmail, what if I use Applemail and I want
my web apps to be able to communicate back to back. And they don’t really care anymore
what their emailing it, they're just knowing that this is a conduit with which I can send
stuff. And part of this is making the data that the apps contain available to other apps
and Microformats can do this to some extent. But Google has an interesting opportunity
because they have control of such a large suite of apps already, that they could build
a system that would be sensible around this. One other thing that Google can look at is
the number of search boxes that are on my machine. All of these search different stuff,
but some of them have similar behavior. Spotlight and the Google search boxes--the browsers,
basically are both universal searchers. They’re all about, "find me something that matches
this" and they could be easily, easily be unified into one type of search box. There’s
also contextual search boxes in each application which are unlimited to the content of that
application and really do warrant standing alone as something else that you can look
at. But they could be tied in to the universal search. The third kind and these--these are
a bit of an exception where they are simple filter boxes within the OS--within the UI,
but they can be much more tailored towards a specific type of search or activity. This--lets
skip over this slide--this quick context picker in Gmail is nice because it basically let's
you search context lets you act on them and it’s results are very tailored to this and
the actions that it shows are very tailored to this, but it allows a huge amount of functionality
in a very, very small space. Other search boxes that Google's sort of doing nice things
with is the Docs one, which has amount of--basically the summon ability where you can go to a search
box, you can type some string and jump directly to the thing you’re thinking about. Potentially
with Docs you are--you know roughly what the name of the thing is you’re not trying to
do this, sort of, complex deep search on it you just want to jump to it. And this ability
to jump to things is something that could be carried across in the universal search
or in the individual app search boxes. All right. I think we’re done. Thank you.