Google I/O 2012 - The Web Can Do That!?

Uploaded by GoogleDevelopers on 28.06.2012

>>Eric Bidelman: All right. Are we ready? Android is processing my photo. Hold on.
All right. Guys, I really want to thank you for coming to my session. This is The Web
Can Do That, in case you've misplaced what session you should be in.
Apparently none of you want to get free Android hardware downstairs, so I do really appreciate
your time for the next hour. This is going to be adventures into HTML5.
Just so you know who you're talking to up here, my name is Eric Bidelman and I work
on the Google Chrome team. I'm an advocate for Chrome team, I teach developers HTML5,
help them use it by writing tutorials. Of course, if you want to contact me there's
Google+, there's the Twitter and there's also a blog that I don't write to very often, but
it does have some interesting things I think on it. So that's who is up on stage.
I get this a lot. People are really skeptical about HTML5 still and so they don't know what's
possible. And so part of my job is to come up here and teach you guys and you too, the
folks at home, what is possible on the Web. So today's talk is that the agenda is there's
going to be no agenda whatsoever. So I'm going to sort of highlight really cool,
interesting use cases that HTML5 can now solve that you probably didn't know were possible
on the Web today. Things that you probably need to build a real
world application on the Web. So a little bit of convention about this slide
deck. This is a living, breathing slide deck. I'm doing something a little bit different
with this. You will see in the top right corner there's a legend and there's a radar, and
that signifies that stuff is on the radar. Keep it on your radar, it's still pretty new,
it's changing quickly. I can actually make CORs requests, cross origin resource requests,
[indiscernible] XHRs, to the WebKit issue tractor, which is really nice. So as soon
as a bug is fixed, I know it's fixed, I can mark them, denote it with this little bug
guy as it gets crossed off. So you kind of know when you go back to this slide at
It's not up right now. I'll post it after the slides. But there are spec links and you
know the stuff is up to date. Okay. So without further adieu let's jump
into the number one main awesome, sweet thing that the Web can do. The first is that we
have CSS for Web apps. And you're probably like whoa -- yeah, Mythbuster style. You're
probably like, whoa, Eric, we've had CSS for a long time. What's up with that? Well, we
have. We had a lot of great stuff in CSS. We have floats and absolute positioning and
tables and all this stuff kind of working together. The real problem with that is it
was all designed in times of websites, not Web apps. So if you want to do complex Web
layout it's really, really hard until now. So I'm calling 2012 sort of the rise of CSS
for Web apps, and that's because the working groups have proposed a number, a slew of new
stuff that we can play with eventually as Web developers. Regions for things like flowing
and magazine type layout and design. Flexbox layout, which I will talk about, it's one
of my particular favorites, being able to easily position things on the page in a flexible
manner. Grid layouts and hierarchies in ASCII and
CSS so you can define where things show up on a page. It makes sense. If you're building
something on top of CSS you need engines and different mechanisms to do that for Web apps.
And then there's the bells and whistles, there's the hierarchies, there's the folders and effects
and other really cool stuff coming down the pipeline.
CSS variables. If you guys attended the Web component session earlier, that stuff is finally
coming to CSS. You can finally use variables in CSS. It's not that exciting, but it's super
exciting as a Web developer that you don't have to use a Web processor anymore.
So why am I picking on Flexbox out of all this new stuff? Flexbox is really great for
Web layout design and it's really going to solve and is solving the sort of common use
cases of a Web app. So that Holy Grail layout, that header, that footer, that three-column
tiered system, very hard to do with, notoriously hard to do with CSS today. Very easy with
Flexbox. And so the way Flexbox works is that there's
a new display property. So I can tell a container to be a Flexbox container and I'm doing also
something in this slide deck, nobody is going to catch me on vendor prefixes. I'm using
sort of a compile type syntax, a mix-in syntax to denote that any time you see a + for CSS
that means vendor prefixes are involved. I can tell, for instance, this right container
to be a Flexbox container with display vendor prefix flex. And we can do things like center
content horizontally and vertically super easy. With three lines of CSS we can do this
now. We don't have to use negative margins or JavaScript. So along my main access here
-- this is a little bit of Flexbox terminology. In my cross origin access, I can justify content
to the center, I can flex it to the start, I can say everything be adjusted far to the
right. And I can also align those items in that cross access, so I can align items to
the center, I can allow them to the bottom. I can do things like stretch to take up the
available content of your parent container. Very, very handy with layout.
And I can do also things like distribute yourself evenly in a container. Take up the available
space and put an even distribution of white space around yourself.
Another cool thing about Flexbox is that it completely takes away the sort of mapping
of your source to how things are rendered on the page. So typically what happens is
that you define your A, your B and your C div. These are just three divs inside of this
flex container. And that's how I've weighted out my markups, right? I have an A, B and
C in my markup. But maybe I don't want to render A, B and C in that order, and I can
do that through different properties of Flexbox. I have this flex direction property, so again
vendor prefix. So right now we're flexing in a row, but I can choose to change the order
of that row. I can reverse the order of the row using row reverse. I can tell these items
to be in a column so A, B and C are not in a row now, but they can be rendered in a column.
And then using the order property I can sort of change the order depending on what value
I set here. So by default these are all the same order as they're laid out in markup,
but I can tell the second div to come before A. So this is independent of how I've got
my markup out, just rendering it different next to the page using the orientation and
direction properties of Flexbox. I can also tell it to come after everything
as well. But another really cool thing that Flexbox
solves is sort of this type of use case where you have the same height columns. So in order
to do same height columns -- this is a great example of floating three divs across the
page. But if I want to do something like make these even, right, I have a really awesome
UI, I can do that with a little bit of JavaScript where I can set these and get the computed
style of the content and know exactly how tall they should be.
But what happens if my UI is being rendered through JSON, I'm porting in an RSS feed or
something, and -- my content is dynamic. As soon as that happens, I put a little bit more
content, my columns are now out of sync, I have to recalculate, I have to touch JavaScript,
I have to set properties in CSS. It just feels super, super icky.
So the way Flexbox solves that is through its bread and butter, flexibility. So it's
really nice. I can tell things to take up the available amount of the parent container.
In this case all of these items, A, B, and C, are equally distributed in the Flexbox
container. But I don't have to do that. I can tell the second difference to be two times
greater, grow to be two times as big as your siblings or three times as big or four times
as big. I can also tell it to shrink. So A and C will
be equal size with a flex property of one and then B is going to be a negative flex
property. So it's really handy when it comes to that
Holy Grail layout. So I want to show you an example of this.
This is a live page. Think of this as that Holy Grail layout, that header, footer, that
three column in the center. The first thing you will notice is this whole
thing, maybe not the resolution, but the whole thing is centered vertically and horizontally
in my page with three lines of CSS, right, the alignment and justification there.
Next thing to happen on this page is a really sweet navigation bar. You will notice that
that last button up there has more content than the rest of my buttons, but that's okay.
We can tell the Flexbox items to grow and shrink. So all my buttons are exactly the
same size. I didn't set anything in CSS, particularly the widths of those buttons, it's just flexing
to take up the available space of that complete Web app. But if I add more, all of these buttons
will continue to flex and be of equal weight and size and distributed evenly. It's great
for navigation, it's great for UI. We can also use some CSS transitions on the
flex property so I can do really cool UI flex here, little zippies here, so these are just
Flexbox items within a Flexbox container with the column orientation. And with the CSS hover
I can tell them to grow with the flex property to grow to be two times or three times greater
than rest. My center content in the center of the article
content I can use transforms to do things with vertical alignment there on the side,
my navigation on my side. If I wanted to I could combine the media queries,
and you remember I could independently order my source and how things are presented using
that order property. So that combined media source may be in my mobile UI. I have something
that is different. If I scale this down to mobile UI, notice in the case of my desktop,
the article is in the center. That's my main content, that's what users should be seeing,
but when I scale on to mobile maybe the articles should come before that navigation element,
that zippy on the side. So I can do that with that ordering orientation property, very handy
when combined with media queries. So that's CSS Flexbox. And that's the new
one. There is an older Flexbox if you've been following this stuff closely.
That's actually available in a number of browsers right now. So WebKit, Chrome.
IE 10 will have the new Flexbox as well. So you can expect to use this relatively soon.
All right. Number two amazing thing is dynamic CSS. By dynamic CSS I actually mean -- that
was confirmed too. I actually mean the calc function. If you
attended the session earlier today on Web components, they have CSS variables coming.
But we actually have the ability to do similar things now with the calc function. It's been
around in Firefox for a long time, just came to WebKit very recently.
What I have here is a live example of something that's in the Flexbox specification. So I
basically have a bunch of divs. I have a div within a div within a div, and I put a border
radius of 50% on this div, so that's what's creating the circles.
I've specified the parent circle container to be 300 wide and 300 tall. What's really
nice about this is I can use the CSS calc function, so again vendor prefix, to calculate
each of these inner divs to be 100% minus 4 EM. Each of these children divs is 4 EM
smaller than its parent container and that's producing sort of this target effect.
What's really neat about this is you combine it with what we just talked about, which is
Flexbox, and so that's what's doing the alignment of the center and justifying the content to
the center. So that vertical center, that horizontal center with three lines of CSS
now, combined with the calc function you can do something like this, which you haven't
been able to do before without a little bit of JavaScript.
Really awesome. This stuff now can work together really easily with a lot of new CSS stuff.
And support is actually really good for this as you can see by this slide.
So that's what's happening with CSS. Number three amazing, awesome thing that the
Web can do is data binding. And you're like what? The web can't do data binding. Oh, but
it can. This is actually old hat for JavaScript frameworks.
If you use your framework like Angular JS, for example, one of my favorites at the moment,
data binding looks like this. This is how they define a template in Angular. It's just
HTML, which is really, really nice. I don't have to learn a new API or markup.
If I want to do one-way data binding I can have an input, I can change the input, and
as I change that my template is being re-rendered. I'm just re-rendering the value of the input
as it changes. And that's done through sort of Angular's magic. My model in this case
is this val. That's what I'm calling the model. And as I change the input, val is being re-rendered
to the template. So that's one-way data binding use a JavaScript
framework. But we can actually do a neat little trick that I kind of discovered, which I'm
calling it poor man's data binding, and it's using data attributes, HTML data attributes.
So we don't need a framework to do one-way data binding. So the way this works is that
you have a data attribute. That's your data model. That's where you're going to be storing
your values. And then a lot of people don't know, but there's
this really awesome, amazing attribute that you can use to pull out values from the DOM
and use those values in CSS. So that's how we'll get the data from the
data model. Then our view from this case, what we're rendering this data to, is generated
content. So before and after pseudo elements and generating that model to those elements.
And so we don't have a JavaScript framework to work for us in this case so we have to
actually hook up Web event listeners to do this and watch for the changes.
So what you get back with the poor man's data binding is exactly the same thing. So I can
have an input, and as I change that input, I can re-render that to my template, my pseudo
elements, on the fly. So a small amount of code to do this. I set
up an event change event on my input. And then as that input changes, the user changes
it, I'm sending a data attribute using the data set object. So data-values is my thing
that I'm changing to the new value. And the really cool part of this is that I'm using
before and after pseudo selectors for this. So I'm using that attribute method to rip
out that data element, that data attribute, from the DOM, dividing it by the max attribute,
in this case 100. So you get the exact same thing as the Angular
JS case. I don't need a framework to do the -- this type of one-way data binding.
You can do the exact same thing with an extra span or DOM node and just render the inner
text of that. That's cool. This is a little cooler in my opinion and it illustrates that
we can do this now in HTML5. But we can also use something better, which
is the dataless element, something new to WebKit, and Opera has had this for a slew
of time. This is sort of a semantic way to do one-way data binding. So in this case your
data model is actually defined in markup. You have data list element, you can reference
it by an ID, so the browser's out, and each option is sort of the values of this data
list. And what we bind to in this case is an input
element, so we reference it by the list attribute of ID and so we're finding this data list
-- these values to this input element. So it's a great way to do things like auto complete.
So I have an example of that in here. I have the list of browsers. When I click on this
I get the values from that data list. We bound that data to this input.
If I do things like -- if I start typing, just as you expect I have auto complete behavior.
That's super handy. That's the HTML5 way to do one-way data binding. You actually saw
it in that blue column example, you saw me use this data attribute trick as well to render
out the heights of those columns as they were changing when I clicked on them. And both
of these methods for data binding are actually all supported on all the bottom browsers,
which is really, really sweet. But if you're doing something simple like I just showed
you, you can do one-way data binding just with HTML, just on the Web.
How many people think HTML can access a file system?
It's hard to see up here, we have some believers. We have a lot of naysayers.
It's cool. This one is totally confirmed. It's confirmed because I got really excited
about this one. I got so excited that I wrote a book.
[Laughter] >>Eric Bidelman: Shameless plug, I know.
What we have now in HTML5 is a file system API. This is something that you can do very
easily in a native app. You can read and write folders and persist data. That's very common.
We should be able to do this on the Web as well and we can do that with the file system
API. So there's a new property on the window method
called request file system, and this is vendor prefixes alluded to by this underline here.
You can open the file system and read and write files and folders. That's really huge.
The security behind this is exactly like the other offline storage API. So it's on a per-origin
basis. You can't read and write to somebody else's Web app and you certainly can't come
out of the browser and write to somebody's My Pictures folder or the system's My Documents
folder. It's all on a per-origin for your Web app. But it's super powerful. And the
real benefit of this is to get around issues that something like App Cache does. So if
you've used App Cache, it's kind of clunky, it's hard to use. With the file system API
we can easily and dynamically cache individual files and folders very quickly. Store them
in folders, arrange them in a hierarchy, nuke that if we need to.
So as a quick example of caching an image file, for instance, just a png file, we're
going to use a friend, XMLhtpp request, pull that guy down, and we're not pulling it as
a streamer, but we're pulling it as an array buffer, so we'll set the response type and
we're just going to get a raw byte array back from that image.
And we get the response and we'll open the file system using WebKit file request file
system, give it a store, so maybe we'll store a megabyte worth of data in this. We'll use
the file system's root directory, just the root entry of my file system, and we'll call
it image.png, we'll create it if it doesn't exist using this param here. We'll get back
a file entry, we'll then write that response of the XHR out to the file system and save
that data. So this is just creating a file writer object. We can set up events for when
the write has ended, if there's any errors that happened. And the important part is this
guy right here, this writer.write. We're gonna write out that data, the response of the XHR,
create a new blog from the response and that's going to be saved. The image is gonna be saved
locally to our Web app and we can use that as if it were a local resource to our Web
app. So that's the final product. You can see there's
a lot of callbacks that are involved here. I have at least one, two, three, four -- four
to five levels of sort of callback -- callback spaghetti that's happening here. Callbacks
are hard. People don't like to deal with them. This is a famous quote that I just said.
[Laughter] >>Eric Bidelman: Actually, I wanted to make
it simpler for developers. Again, I'm really passionate about this API. It's very powerful.
So I wrote this library called filer.js. You can check out the GitHub URL there. But essentially
what it is is a library that wraps common UNIX commands on top of the file system API
calls. So we have things like LS and CD and copy and the things that you know from interface
that make it super easy to use this API. If I want to copy a file, for instance, filer.copy,
the file path you want to copy to and then renaming it as well. So super handy.
It makes the whole thing much more approachable if you're familiar with this sort of development.
Let's see a demo of this guy. So this is called the file system API playground. Essentially
what this is is a Web app that sits on top of the file system API. So right now I don't
have anything in this guy, but you can easily use HTML5 Dragon drop to add some files, boom,
I can add some text files. I can open these files, right, so there's this sort of notion
of using this -- this resource locally with referencing it via URL. Can use that within
my Web app. Can also read this file and preview it if I wanted to using the file reader API,
and get some metadata such as the date and last modified times. Can use filer.js and
API to rename the file. The best part about this is I've actually cached these resources,
so if I refresh this page, my Web app, right, has those values and those resources saved
in them, so all the changes I made are being persisted, which is really awesome. Can create
files and folders. Create a folder using the API. You can drill into it. Import some data
here. Luckily it's after lunch so, you guys don't have to get too hungry. But this is
really great. I mean, this is native stuff, but it's on
the Web. We can write and read files and folders now using the file system API. So you're probably
asking yourself this: This is great, but if it's only in Chrome, like why am I going to
use it? Then I'm like, oh, yeah. So again, I really
like this API, you guys, so I decided to implement a polyfill stream library that sits on top
of index DB that then you can use in your Web app to have basically the file system
API in browsers that support index DB. So same Web app, right? Same Web app running
in Chrome is running in FireFox. And I can add files and folders to this guy, maybe -- it's
a nightly build, so I don't know -- there we go. And if I refresh this up, those files
that I just wrote to the Web app are preserved. So that library is called IDBfilesystem.js.
And if you're interested in this API, the file system API, check it out, it basically
means that we can now use this API in a relatively performant way in the browsers that do support
index DB. Super Stoked about it. Hope you guys get Stoked about it, too. Files and folders,
the Web can do that. All right, Number 5. This one looks suspicious. Totally is busted.
Just making sure you guys are paying attention. [ Laughter ]
>>Eric Bidleman: The real Number 5 is serverless downloads. So we got data in the file system,
we're storing data in our Web app -- where is my clicker?
What happens when you want to get data back out of your Web app, right?
So for a long time, if you wanted to download a file and trick the browser into downloading
a file from the response of your server, you would have had to do something like this:
You would send a content disposition header as an attachment, give it a file name, and
essentially what this did was when the browser hit this response, the browser would sort
of -- it would trick it into downloading a file and it would invoke the browser's download
manager. What we have now is a lot of people are writing thick clients, right? Maybe there's
no server involved whatsoever, so how do you do this?
How do you send a header if you don't have a server? It's impossible. So smart folks
in the Web community were like, hey, we can solve this, we've got this great idea, let's
make an attribute, we'll call it "download". That makes a lot of sense. And what download
does is you can use with an anchor tag for instance, and so what happens is -- normally
what happens is if you -- if a user clicks on this H ref to this logo.png, what happens
is the browser will open that image in a new tab and you can view the image. That's cool.
But what happens if you have download specified, this attribute, and you can give it the name
of the file you want to download as, is that it's going to trick the browser into doing
the exact same thing as the content disposition header, and then you can download that file
instead of navigating to that resource. So if none of that makes any sense to you, I
have an epic novel creator here, and this is just a text area that I'm typing into,
so let me bring this up a little bit. So just to show you that I'm typing live. I'm just
typing in this. I'm composing my epic novel. And then I want to have the user save what
they've just done, so I can do that using the download attribute. When I click this
button, I'm going to create a txt file, a text file in JavaScript using the block builder
API. Create a dynamic link in JavaScript. Attach that download attribute as my file
.txt as its value. When I click that, the browser is not going to open that resource,
it's actually going to download the file that I just created in JavaScript, and if I open
this, it's going to be native to my system, and as you can see, the text content that
I just pulled out from that text area has been saved. I've just downloaded a file in
JavaScript, all clientside, no servers. That's pretty sweet.
So we have the ability to save data in the file system, we have the ability to get it
back out using the down attribute. What about getting data in between apps? Now, there's
this thing called Web intense, and I encourage you to go to Paul Kinlan and James' session
later in the week. But one way to officially transfer data is rocketships, of course. But
another way is with post message. And in order to describe this and set this up for you guys,
I want to take you down sort of memory lane of post message. Post message started off
as a way to communicate with a Webworker or a window object, right? You could send a string.
You could send "hello world" to a worker, and then it could, I don't know, do something
interesting with that. But what happened is, hey, people are gonna -- the browsers are
like -- developers got smart, they were like let's just send JSON stringify data. So they'd
stringify some JSON, and pass some commands in and out of worker or to a window object.
So it was totally cool, but then the browsers were like, hey, like, we can do better, let's
just do that for them, just not have the ability to stringify JSON, but let's just do it under
the hood, so what happens is the browser would JSON stringify and JSON parse this message
out, so that was cool. That got us one step further. Sort of this post message evolution,
but then the third step that came along was the ability to send more complex data, right?
You just saw me send a file -- create a file in JavaScript. Why can't I send a file into
a Web worker and do something interesting with it, or a Web GL texter as an array buffer?
All right, we have binary data in -- on the Web now; we should be able to do interesting
things with it. So eventually , the browser is like, hey, we'll open this up, we're going
to use this structured cloning algorithm to pass a blob, a file into the Web worker and
process it. Or you can send an array buffer, so this is really, really cool, right? And
a lot of the modern browsers support this. You can send a massive, you know, data set
into a Web worker. Do some computation on it in this sort of multi-thread environment
and then get it back out and do something interesting. So the real problem with this,
though, is that these are all -- all the methods I just covered are copies, so that becomes
very inefficient when you're copying and sending amounts of data in and out of a Web worker.
It's kind of pointless to have this awesome, you know, multi-threaded Web worker doing
its thing, but if you can't get data in and out very quickly, then it defeats the purpose.
So of course I wouldn't be up here if we can't do better, and we can do better, and that
comes via transferable objects. Transferable objects are something that the Web GL community
thought up because they need this ability to do processing on large amounts of data.
And so you can see this guy is using your old friend, but it has different semantics,
and that's why this method is prefixed with WebKit post message. This is something that
is supported in Chrome. So what happens with the new post message is that it's a zero copy,
so if you're familiar with like C++ or C, think of it as a pass-by reference rather
than a pass-by value. And so the ownership of this data is actually transferred from
your main app to the worker or window context that you're posting to. And so what happens
is that this becomes actually really, really efficient, up to 50 times faster. I measured
this in Chrome 17. So Chrome 17, you know, that is sort of long gone now, but it could
even be faster. I don't know. But the important point of this graph are these last two columns
here. In Chrome 17 the difference between the regular post message, which is on the
left, the middle one, and the right, is orders and orders of magnitude. This is a logarithmic
scale, keep in mind. Faster. So basically what I did in this test was I sent a 32 megabyte
file into a Web worker. I didn't do any processing on that file, and then I sent it right back
just to see how fast that was. I measured that. And you can see the rate -- the round
trip simulated rate of that is super, super fast. So this is really sweet for things like
physics simulations and again Web GL, gaming. I have an example of this, and unfortunately
this is not super sexy to demo, but this is my awesome sexy demo for transferable object.
This page has basically set up a Web worker. I got a worker ready to be post messaged to,
and when I hit this run button, it's going to use that new WebKit post message to send
data into Web worker. I think we're sending -- we're sending a 32 megabyte array buffer,
so this 32 megabyte file into a web worker, and then sending that right back, seeing how
fast that is. So I'll hit this, ready? Boom, boom, boom, boom. 6 milliseconds. 5,000
megabytes a second simulated rate, right? I mean, that is kind of ridiculous, right?
That's awesome. That's awesome for sending large amounts of data. And that's a lot of
data. 32 megabytes is no joke. Pop up a full screen here. Just to show you
the difference -- I'm not picking on any browser in particular -- but a browser that doesn't
support transferable object, you can actually noticeably see the difference here. Let me
zoom in. So when I hit this, run, boom, boom. So, not only do the numbers speak for themselves,
157 as opposed to 5,000 megabytes a second, you can actually see the delay, right, visually
the delay that it takes for this Web app to make that round trip request. So significant
difference using transferable objects and performance. In my opinion this is something
that, you know, the Web path form and workers in window and post message should have had
available in the first place, but this is sort of a common thing in the Web, right?
We're sort of incrementally improving on what we already had, making it better and better
and better every time. This one's only in Chrome, again something that should be in
every browser. Efficiency is something we love in the Web. How many people think the
Web can access native hardware? Not using phone gap or something like that?
Okay. Got some believers. I love it. You guys know this is when it's going to be confirmed.
It's confirmed. It's totally confirmed. What I mean by accessing hardware is device APIs.
There's a whole working group dedicated to figuring this stuff out now. And what I mean
by devices are things like geolocation and accessing the accelerometer of this laptop,
using web GL to access the GPU, right, the graphics processor unit of the laptop, using
a very high level JavaScript API. I just showed you the file system API. We can read and write
folders and files on the Web now. Whether I'm online or offline, network connectivity,
all via JavaScript, battery API, game pad API, all super possible. Web RTC and Web audio,
I want to briefly mention on. There's two dedicated sessions to these two topics, because
they're very -- they're noteworthy and very complex and they should have a dedicated session.
Being able to access voice and video on the Web is sort of this coveted grail, right?
I mean, we've wanted to access the microphone for a long time or get camera for a long time,
but you've needed a plug in to do this, so this is one example of being able to sort
of -- the first step along this path to do that, with one attribute, this x WebKit speech
attribute, we can annotate an input, and the browser is going to annotate it with this
little microphone input. Instead of typing in this Web app, it can actually interact
with the microphone in this Web app. "I want some coffee". All right. It worked. Sometimes
it doesn't. Or you get some weird rendition. But this is really great. A different way
to interact with my Web app. I don't have much of an API to play with, right? It's sort
of happening under the hood. But I'm accessing the microphone of this laptop using one attribute.
So what we have now is something way better. We have camera and microphone access via Get
User Media. Get User Media is coming out of the Web RTC project, navigator dot -- either
Get User Media or the prefix version. Also Christian from FireFox tells me this is coming
to FireFox and nightly builds, so that should be on there as well. To the camera, the device
and the microphone device. With this call, I can specify that I want audio access, I
want video access, and what I'm going to get back is a stream object representing that
data that I get from both devices. And then what's really cool is again I'm using different
parts of the platform that have already existed for a long time. I'm using a video element.
HTML video is one of the core features of HTML5 when it first came out. Instead of setting
the video source to, you know, a movie file or an OVG file, I'm setting it to a blob URL
created from the data I get from the camera. So instead of a file, I'm just sending the
camera data and the microphone data directly into this video tag, so that's what's going
to give us the ability to render live to video tag. So a little demo. There's a lot of really
cool demos people created using Get User Media and the ability to access the camera.
This is my demo. I do trust this app. I made it. I wrote it. So I'm gonna -- I can't see
at the top. I'm going to allow access. We'll create a video element in JavaScript, and
there we go. My first stage -- you can see behind -- so this is great, no plug-ins, right?
This is just JavaScript on the Web, but again I can combine this with different parts of
the Web platform, so I can use CSS filters in real time to overlay this video with effects
like gray scale. How about a blur effect in real time?
That's pretty -- that is pretty trippy. Version effects, right?
Just using CSS, applying to this video, I can do all that and still really awesome,
looks great. I can combine it with a canvas, right, do a photo booth effect.
[chuckling] >>Eric Bidleman: All right. That's enough
of that. One of my favorites, though, is this one called the Web cam toy, so I'll grant
access to this. Okay, that uses my camera. This is not going to use CSS filters, but
it's going to use the power of camera, the power of Web GL, right? So with Web GL we
can access the GPU, that native access, and we can do things -- really awesome things
with shaders. Shader technology is amazing. Again, this is all JavaScript using Web GL.
No plug-ins. Look how fast it is. Really trippy things. Let me get to the cool stuff. There
it is. [chuckling]
[ APPLAUSE ] >>Eric Bidleman: Yeah. This one is cool.
Super trippy. Oh, yeah, 3D. This is after you've had a few drinks at the party tonight.
Just amazing things. People have done thing with augmented reality, right, holding up
a paper -- piece of paper with a QR code on it, the camera tracks you. So play around
with those. When I put the slide up later, just really awesome, amazing things. Support
for this is actually really good. Opera just released Opera 12; it has this. They've had
it in the mobile version for a long time. It's coming to FireFox nightly I'm told, and
Chrome just released a stable version with Get User Media, so we can do camera access,
we can do microphone access without a plug-in now.
HTML5 audio has had a little bit of a slack -- been problematic, to say the least. But
what we can do now with the Web audio, guys, make it -- make it sexy again. So HTML5 audio
is great. It's just like video, right? Without a plug-in, we can play audio on the Web, just
populate a source attribute of an audio tag. Play it, that's cool, we don't need a plug-in,
but what if you want to do something like this.
[Music Playing] What if you want to visualize that data in
real time, get the frequency analysis as it plays?
Visualize that on two canvas tags, plus Web GL transforms a little bit of a reflection
on the bottom there. Can't do that with HTML5 audio. What we have now is ability to tie
in to what's called a Web audio API, tie in an audio tag as a source to that API, so instead
of driving the Web audio API, via, you know, an XHR request or something, we can just send
data directly to it -- sorry -- via -- via this audio tag, so it's really, really important.
This doesn't have to just be a video. It could be a -- excuse me, an audio tag, it could
be a video tag, so imagine I had a HTML5 video playing, and that is the source to the Web
audio API, then you're visualizing sort of effects, as movie changes and dark monsters
pop out, you can really scare people. But the way the search works is really easy, so
we have the Web audio guy. I encourage you to go to Chris Wilson's session later in the
week about this, really amazing API. A lot of low level access to the core audio of the
system. High level JavaScript API on top of it. Create an audio context. Notifications
-- I will create HTML5 audio element, I will populate it with MP3 as its source, set up
a couple of controls, some auto play on it. The important integration point between these
two, sort of the old audio working with the new Web audio, the capabilities of that API
is this create element source, media element source. That's how the audio tag is sort of
filtered and piped into the Web audio API. Which leads us conveniently into Number 10.
So I get this one a lot. A lot of people ask: "Hey, how come HTML5 can't do, you know, streaming
audio? Or streaming media?"
And it actually can. It can do that through another old friend, WebSockets. So WebSockets
have been around for a while. It sort of suffered from the same limitation of Webworker, where
you can only send just basic data. It can send string data. That's not so handy, right?
We have things like files and array buffers and ability to use complex data. We should
be able to send and stream multi-media or files to our web apps.
So we can do that now with WebSockets. And WebSockets has a new sort of socket, that
binary type is the property you set on this. And you set it to a blob or you set it to
an array buffer, and that tells the WebSocket that's going to be speaking binary data instead
of just regular string data. What's awesome about this is on my on message event, I can
use that directly to set an image source, for instance, from a blog URL. Instead of
having to base 64 encode data on either end, decode it, which is, you know, the overhead
associated with that, we can just use the data directly that we get back.
So I have a couple of demos of this. The first is a Chrome extension. So I have a little
Chrome extension installed here and top right corner. I'm going to click this guy. Imagine
this is my viewer. I'm going to basically do a sort of screen cast from the Web.
This is my viewer, imagine this is somewhere around the globe. I'm going to open a new
window, pretend that's here. So when I click this guy again, it's going
to fire up a Websocket, binary Websocket, and use the Chrome extension, screen capture
API to capture the current tab. Sports, because sports is great.
So on the right side, I have my presenter, this is just capturing the current tab as
a png as I scroll. It's sending that data through a WebSocket to the remote party. You
can see it's very performant. I don't have to do any basic 64 encoding. As I'm scrolling
on the right on the live page, that image is just being sent across the wire in near
real-time using the WebSocket. That's really, really sweet, right? We can do this now thanks
to the Chrome extension API, yes, but also via a binary WebSocket.
Whoa, inception. [Laughter].
>>Eric Bidelman: We can also do other things with different types of media. So I just showed
you a little teaser preview of the Web audio API. The API allows you to analyze sound in
real-time as it plays. What I have here is a page that sets up a
binary WebSocket, and it's going to use the file reader API to essentially cut an MP3
file up into multiple smaller chunks and then it's going to connect to that WebSocket, send
each of those chunks across the wire, on my -- on my left side here, the viewer side,
I'm going to use the Web audio API to schedule those chunks at very precise times exactly
when they should happen. So essentially I reconstruct the audio on
the other end and just sort of use the sequence that it should be in and then visualize that
using the Web audio API. I'm going to load up an MP3 file here on my DJ machine, this
guy in the right corner. Binary WebSocket, web audio API.
No sound? There we go, there's sound.
[ Music ] >>Eric Bidelman: So again, streaming from
one client, streaming audio data using a WebSocket, using the Web to reconstruct it and then visualizing
in real-time. You can see the chunks coming in, I'm receiving
sort of mini chunks of that MP3 file as I reconstruct the audio and sort of the current
time is playing. Just to prove to you this is streaming, I refresh this, come back at
the exact same spot. It's using two brand new search capabilities on the Web platform
together, something that you probably maybe not hopefully didn't think was possible, in
streaming audio on the Web. No more plug-ins for this, you can do is now
using the Web platform. [ Music ]
So that was really quick. I covered a lot. I just want to point out that we did cover,
you know, sort of everything under the sun. The Web can do really amazing things, I hope
you do agree. CSS, whether it's the new CSS stuff for Web application layout and design,
doing data binding, doing things like file access, accessing the native hardware, using
-- officially transferring data once you have it in your Web app, and then doing really
awesome things with some of the older capabilities of HTML5 multi-media, and also some of the
newer stuff, combine those two with the integration points.
Just want to point out, we have a Google+ developers page. If you guys want to know
more about what Chrome is doing with HTML5, and the open web, check that out. If you are
not using Chrome Canary, or the developer channel, Chrome really does heart HTML5. A
lot of this stuff I just showed you again is very new. So play with the new Chrome file
bugs on Let us know what you think. Give us the feedback.
This presentation will be up on my GitHub account at HTML5can, as soon as I exit this
stage. You can also access these slides at HTMLfive, with the five spelled out,
There's my social stuff if you want to follow or Tweet at me. I'm happy to take questions
here and I will also be available after the show, as well as office hours tomorrow.
Thanks, guys, appreciate your time. [ Applause ]