Udacity: Teaching thousands of students to program online using App Engine

Uploaded by GoogleDevelopers on 01.11.2012


FRED SAUER: Hello, everyone.
I'm here in the studio today with Iein Valdez, colleague,
developer advocate on AppEngine as well as myself.
And today we have two really special guests with us.
If you've been following the AppEngine blog in the last
couple of days, you saw that we had a couple
of guest blog posts.
And we actually have those guest bloggers with us today.
So we have first Steve Huffman who is the founder of Reddit
and Hipmunk.
And Steve has been teaching a course online at Udacity, a
free online university.
And so we have Chris Chew from Udacity who's a senior
software engineer there.
And today we're gonna talk to them about their experiences
of building course material for Udacity, building the site
itself, building the course material,
working with AppEngine.
And we'll have a chance for you to ask questions on a
moderator page.
So let's bring in Steve first and let him to do a little bit
more formal introduction.
And then we'll get cracking.

FRED SAUER: I'm sorry.
You were muted for a second.
Go again.
Let's try that again.
OK, so.
I'm Steve Huffman.
I am one of the founders of Reddit, and more recently one
of the founders of Hipmunk.
I've been a web developer basically my entire career,
all seven years of it so far.
And then this past spring, I had the opportunity to teach a
web development class with Udacity.

At the end of this course, I really wanted students to have
the ability to build their own web applications
and get them online.
And Google AppEngine was instrumental in
the success of that.
Actually went into the class not knowing anything at all
about AppEngine other than basically that existed and it
might help us with want we needed to do.
And by the end of it, I came out a total fan.
The course was a success.
AppEngine is great.
It was a really fun experience.
And so now I've turned into something of an AppEngine
evangelist, I guess.
Well, thanks for coming to the program today.
I was gonna ask you what your feeling was when you started
out and how it worked out.
But it sounds like at least the end result
was a pretty good.
Can you tell us a little bit more about how you felt about
AppEngine going into it with--
FRED SAUER: --a lot of unknowns?
STEVE HUFFMAN: So basically what I knew about AppEngine
going in is that it was this Google product for building
web applications without dealing with the sys
So you get a basic web framework and you can deploy
your code and it would magically be online and it
would magically scale and you'd have this data store.
There's a lot of magic basically.
And as a developer, right, one of the things you learn is
that you should be aware of magic, right.
There's always something lurking under there.
But what we wanted to accomplish with the course was
I wanted to teach people how to build web applications, not
teach them how to be sys administrators.
Didn't want them dealing with installing MySQL and Windows
and that sort of thing.
So, it sounded like on paper, and after like a cursory read
through of the documentation, it sounded like it would do
just about exactly what we wanted.
But I hadn't actually ever really used it myself.
So, we went in with some question marks basically.
If this thing is as good as it says it is,
it's gonna be perfect.
But as a developer, right, the documentation and
implementation seem to be implemented by very different
teams of people.
But in this particular case, I was pleasantly surprised at
just about [INAUDIBLE]
at how things worked.
And when I was [INAUDIBLE] things the first time just to
see like will this basic application get online, I was
actually really blown away that the deployment process
worked on a first try.
Because I've spent an embarrassing amount of time
over my career deploying code.
So it was actually a really cool experience.
Can you maybe talk a little bit more about your background
as a developer?
Like, where did you start?
What was your road, like, to getting to this point where
you wanted to teach this course?
I mean, programming has been my life's passions since I was
a little kid.
I got started when I was, I think, eight years old or so
typing in programs out of magazines into GW-Basic.
And all through high school, I loved to code.
And I studied computer science in school.
Me and one of my college roommate, Alexis, we started
Reddit immediately after graduating from the University
of Virginia.
And that was actually my first time really
developing on the web.
I had a passing knowledge of HTML and CSS.
And I had dealt with a SQL database maybe once or twice.
So that was a pretty good trial by fire.
Fortunately, Reddit never had hockey-stick growth, right.
It grew slowly but surely over the last seven years or so.
But, we learned a lot of things the hard way from how
to scale a database, how to migrate stuff.
Going from one app server to two app servers was a big
challenge for me.
I didn't know what the heck I was doing.
And going from two to whatever they're at right now was also
a huge challenge.
And so, I've learned a lot of lessons, a lot of best
practices, the hard way.
And that whole experience was a lot, a lot of fun.
And I wanted to basically, in this course, try to share some
of those experiences so people don't have to make so many
missteps that I made.
And AppEngine, actually, it's funny because it abstracts
away a lot of those bad decisions.
It makes things a lot more straightforward so you can
actually focus on the actual software you're building.
And I feel like on Reddit, I actually spent a lot of time
focusing on ops and not on Reddit itself.
That certainly rings true with what we've heard from other
And that's kind of the mission we set out when building
AppEngine when we launched it several years ago.
So, I'm glad to hear that echo back that that was the
Ops are fun.
But it's one of those things, you know.
Leave that to the pros, right.
And Google, like you guys are the pros, so.
As me, it's like when I was 20 years old, why should I be
messing with this stuff?
I've never done it before.
Google, they're the world's experts.
FRED SAUER: I guess my own experience is ops fun when
you're on one server.
STEVE HUFFMAN: Yeah, exactly.
FRED SAUER: And as soon as you get to two, it starts to
become annoying.
STEVE HUFFMAN: I mean, it's fun when it's working.
I was actually curious if you could talk a little bit about
Python and how that was as a language, some of the
And then how it was like to actually build the content?
So, Python's great.
You know, when I actually started Reddit, Reddit was
written in LISP.
And when Aaron Schwartz joined us, he was a big
Python guy at the time.
I was thinking to myself-- or thinking out loud rather--
we should rewrite Reddit because the stuff we have
right now is trash.
And he's like, well, we should rewrite it in Python.
And I was like, OK, sure.
What the heck.
I had never done Python before.
So we rewrote Reddit in about a week in Python.
And basically, Python has been my main language ever since.
And I love it for a lot of reasons.
The main reason is that it is unlike LISP.
It is very difficult to write bad code in Python.
It's a great language for teams.
It's very readable language.
And when you're working together in a team, being able
to read other people's code is so important.
And we also developed these picky feelings about web
At the time, Rails was blowing up.
But it was just way too magical.
Like it hid so much from you.
I really didn't like it.
Django was just getting started.
There was a couple others we looked at.
But every time we tried one of them, we would find ourselves
within a couple of days feeling boxed in.
So we ended up making our own framework called web.pi, which
basically mapped some URLs to some handlers.
And you had to explicitly deal with get and posts and headers
and cookies.
And we felt like that was the perfect level of abstraction
to deal with.
And by a very coincidental twist of fate, AppEngine
actually shares some lineage all the way back to web.pi,
which was really convenient for me.
But I think it's the right level to be working at as a
web developer.
You should understand how the web works.
You don't need to have these simple tasks hidden behind
these complex abstractions.
And so a simple framework, a simple language, everything
nice and simple, I think, is a really nice quality about
Python the way we are doing things at Reddit, and then the
way things work at AppEngine now.
That all rings true that you want simple abstractions as a
programmer so you can put the pieces together.
But you also want to be close enough to the metal that you
could actually make the engineering trade offs that
you need to make in order to make it site
performance and scale.
STEVE HUFFMAN: 'Cause every site's a little bit different.
And what makes a good site great is how you deal with
those unique challenges you have.
And being able to do that is supremely important.
So I think you started talking a little bit how it was
building the course.
How did maybe Python play in, and how did AppEngine play in?
What was it like building this course and going through and
creating that material for the students?
STEVE HUFFMAN: So, it was actually a lot of fun.
And so once we realized that AppEngine was going to work
out, we constructed the course the way we did, which was
basically at the end of each lesson--
there were seven lessons in total--
at the end of each lesson there'd be homework.
And the overarching kind of arc of the course was
basically build a blog.
And so we could say, add this feature to your blog.
So the first lesson was basically get a hello world
website online, which if you knew exactly what you're doing
could take as few as five minutes.
And if you've done it for the first time, it still shouldn't
take more than an hour, which is really impressive for
getting anything online.
And lesson two was forms, getting forms and
validation and stuff.
And the nice quality about the way we've structured this is
to submit your homework, all you would have to do is submit
us a URL of your live website online.
And then we could have our grading scripts just go hit
your website and verify that it's working.
And so I think then we got into basic databases and then
user accounts and then some kind of scaling memcache
stuff, that sort of thing.
And I wanted the course to be immediately useful.
I wanted the students to have gone through these experiences
of implementing accounts, of implementing caching, of
validating form elements.
Because those are all the things as a web developer you
do over and over and over again.
And it was a really nice opportunity.
I feel like web development is one of those rare pieces of
learning where you can be instantly useful after a very
short amount of time.
Well, you certainly have something to interact with,
and it's live on the web.
It's very gratifying.
FRED SAUER: I went through the course myself, and I really
thought the notion of submitting a URL where you had
completed your assignment, or your mini assignment, was a
really good way of keeping me engaged and pulling me along
the different steps of the course and say, well, just do
this one more thing and you can get to the next
step and the next.
So I thought it was very engaging.
I like it.
STEVE HUFFMAN: I'm glad to hear that, really
glad to hear that.
FRED SAUER: So you talked about, I think, some of the
features you liked on AppEngine, the built-in user
accounts and the caching.
Were there other things that stood out in this platform as
a service offering that were really beneficial?
Either thinking back to the days of
building Reddit or Hipmunk?
Things that you use today that would have saved you maybe a
lot of time?
STEVE HUFFMAN: Oh, absolutely.
The big one is the data store.
At Reddit, and to a lesser extent at Hipmunk, I feel like
I've written our own kind of like ORM, no SQL thing on
postgres half a dozen times now.
And it gets a little bit better each time.
But the data store is really, really nice.
Once you start thinking in like I can't do joins, I need
to structure my data this way, it's really, really nice.
And data store's nice from two points of view.
And this actually really surprised me.
The first is that you just don't have to think about the
scaling issue.

It just works.
I was really surprised at that.
The documentation doesn't go into great many details about
how it works.
But I guess if you've built it a few times, you kind of
understand which strategy Google has taken with this.
And, the other thing was that it allows you to develop very
fast, right.
The whole no SQL argument, right, is that you don't have
to be constrained by a schema.
You can kind of add features as you go, which is part of
the culture web development, right.
Think of half a product and build it and get it online,
and then write the other half later.
If you have to do a data migration every time you add a
new feature, which we did for the first couple of years at
Reddit, it's so painful.
It's so painful, especially if you don't know how to do data
So not having to think about that would have been a
tremendous advantage.
I feel like I spent probably half my time over the course
of Reddit thinking about how are we gonna store this data?
How we're gonna access it quickly?
How are we going to scale it?
And if I didn't have to think about that, I could have spent
a lot more time just screwing around on Reddit, which would
have been nice.
So you mentioned not having to do data migrations.
And I think people who have built it scale or have built
sites that have been around for a little while as they add
features maybe have thought about this.
But maybe not all of our viewers have.
Can you maybe talk about how you in practice add fields to,
essentially, an existing scheme, or how you restructure
the data store without doing a migration, without
taking the site down?
STEVE HUFFMAN: So a big challenge when you're using
kind of a traditional approach, or the kind of naive
approach that you might learn from a web tutorial, is that
simply adding a column to a large database
is not always fast.
Actually most modern systems it's pretty fast now, but it
wasn't necessarily the case then.
But let's say you want to restructure Reddit comments,
for example.
They're threaded.
There aren't really good kind of tree-like data
structures in SQL.
So you have to kind of create the systems.
It's like, so, are we gonna pull out all of the comments
for a particular story and sort them in memory and then
display them?
How is that relationship gonna work?
How do we get a fast list of everybody who's
voted on this thing?
How do we sort all of these stores in real time?
Those are things that aren't really SQL problems.
And so you kind of have to work around SQL to do it.
And for optimization, what we basically do is we
de-normalize everything.
We store things redundantly in all sorts of different places
and in a different tables and on different systems so
everything's fast.
If you take the kind of the more the data store approach
where you can kind of your data as you're going to use it
or you can store your data in the place that it needs to be
and if you change the way you want to use it, all you need
to do is store it a little differently.

It's much simpler.
It's a much easier way of thinking about things.
You don't just grab this blob of data out of the database.
If you try to distill everything into a simple query
of basically by ID.
You can convert everything into a by ID query, things
cache really nicely.
Things scale really nicely.
And if you're doing that simple just get me item with
ID x and all of a sudden that item has all of the data
you're gonna need for x, it's really, really convenient.
I hear that one of the hard things for people coming from
a relational SQL background is this notion of giving up
And they fight it initially.
And I went through this myself when I was first going into no
SQL scalable databases.
I constantly was trying to normalize my schema.
And when I finally gave that up, then this whole new world
opened up of kind of what you're describing of hitting
the sweet spot of everything is a fetch by ID and the
systems performing fast and cache.
All those things you mentioned just suddenly come together.
STEVE HUFFMAN: And the code actually gets simpler, too.
Everything gets actually much more simple.
You just have to think a little bit.
Sometimes it's tricky.
Sometimes you have to fold your use case into this thing.
But now it's funny.
After doing it for a few years, the notion of designing
a table, it's just ugh.
I don't want to spend time doing that today.
So this one question that I always ask developers--
at least ones that I talk to initially when they're coming
to work on AppEngine and they're struggling a little
bit with the no SQL world, and because AppEngine is a little
bit different or the no SQL world is a little bit
different, it feels like an uphill battle because.
It's something new that they're learning that they
haven't been exposed to.
But I always try to come back, like six or eight months later
when they've built their site and had some time to play with
it, and ask them is this new world harder or is it easier
than it was?
And I'm curious what your answer is.
STEVE HUFFMAN: I think it's absolutely easier.
Because the hard parts about SQL is that SQL's designed in
the general case to solve problems
that aren't web problems.
It's really good for like dealing with
spreadsheet-like data.
But when you have this case where like--
on Reddit, we built this database called thethingDB.
And it's because everything was a thing, right.
Users and comments and links and subreddits all had a lot
of the same qualities.
Have you ever tried to do inheritance in SQL?

It's a nightmare.
And so once you can free yourself from trying to force
your problem into a spreadsheet, yeah, it becomes
much, much simpler because sometimes you have to--
There are challenging aspects to it which is, what
queries can I do?
The types of queries you can do can be limited sometimes.
So that requires a little bit of extra thought.
But it's a lot less thought then, how do I represent this
non-tabular data in a table?
I appreciate the answer.
The answer varies, but it's usually somewhere between both
worlds are about equally complex.
Or it's more along the lines of your answer where this new
world is actually easier.
And I think the answer tends to be if people are playing in
a world where they're building bigger apps and more scalable
things, they tend to find the no SQL world easier.
And people who are still building smaller apps for
themselves or for small groups of users, these tend to find
them of roughly equal complexity.
I mean if you're on one machine, SQL
works pretty well.
There's still some constraints, though, right.
Because if you make one schema design decision and then you
want to change that, that's troublesome.
If you have to care about data types, it's nice not having to
care about data types all the time.
Sometimes it's nice to just store a blob of stuff and then
not have to think about it again for awhile.
So if you put yourself in the shoes of other developers and
you're maybe coaching a friend or a colleague on what
technology stacks they should be using for maybe a site that
they're building, could you maybe talk about some of the
different things they should think about as far as choosing
STEVE HUFFMAN: When it comes to choosing technology, you
can't go wrong using something that somebody else really big
is also using.
And I usually use that as an argument against things.
Don't use Mongo because who big uses Mongo?
Don't use noJS.
Who big is using noJS.
Use Python.
Google uses Python, right.

Use postgres.
Use MySQL.
Use things that have worked for other people in the past,
the battle hardened, tried and true systems.
Because you don't want to be the test dummy.
You've got more important things to worry about than
will this thing scale for my needs.
You want to be one Google search away from answering all
of your problems.
You want to put yourself in a position where you will not
have any problem for the first time.
And so that's how we usually think.
And a really good measure about the quality of a piece
of software is how easy is it to install.
If that thing builds on your weird machine the first time
and the documentation is clear, that's a really good
sign people have been down that road a number of times.
And if you're using NODE and half the packages have to be
installed globally and half of them have to be installed
locally and it's screwing up your home directory, you
should be thinking, ugh, maybe this isn't quite as polished
as it should be for my needs right now.
So, before we kind of jump over to Chris to kind of hear
his take on kind of building Udacity and his experiences
there, maybe can I leave a parting question with you
which is would you use AppEngine for your next
Reddit, Hipmunk--
I don't know what the third in that series is?
It would absolutely be, I think, the
first place we'd start.
Because when you're starting a company, you should be
thinking about what is your company doing.
You don't want to be thinking about how are we gonna scale
this thing.
You just don't want to deal with those problems.
The first three months of a start-up are so critically
important, just launching.
Yeah, I would give serious consideration.
And certainly if it was Reddit or Hipmunk, both of them would
have of lived just fine on AppEngine, I think.
Every problems a little bit different, but that's
certainly where I would start my thinking.
IEIN VALDEZ: I have one more quick question.
Actually I was just curious, sort of, now that the course
is up, people have taken it, what's been the reaction from
people who have taken the course on the platform?
Overall, like positive, useful?
What's your take?
STEVE HUFFMAN: The sentiment that's gotten to
me is really positive.
Now, I don't think people are gonna be like, hey, Steve,
your course sucked So--
IEIN VALDEZ: Yeah, yeah.
STEVE HUFFMAN: --maybe I'm a little bit of a biased source.
But, I think the enrollment numbers are still pretty solid
and consistent.
I've met a handful of people, including a couple of our
employees at Hipmunk who took my course, which is really
nice because I kind of push my agenda on people before they
even get through the door.
So that has been really nice.
And I'm really proud of it.
It was a really fun experience and I love meeting people who
have gotten to learn from my mistakes, get a year or two
ahead of the curve, and not have to go through the same
pains that I went through.
IEIN VALDEZ: Very cool.
So we're gonna go over to Chris Chew, senior software
engineer at Udacity.
Chris is gonna do a little introduction.
I think he's gonna have a demo for us for Udacity.
And then I think we already have some questions queued up
for you Steve and for Chris, and maybe for a
couple for us as well.
So at the end we'll go through those.
First you want to start out and just do a little better
introduction then I gave you.
My name's Chris Chew.
I'm a senior software engineer for Udacity,
originally from Colorado.
I actually moved out here to join Udacity.
So it's been an amazing adventure to be part of
something that's making such a big splash in the world.
And it's been fun to work with people like Steve
and you guys at Google.

It's interesting.
AppEngine is actually a pretty good fit for Udacity, and I'll
talk a little bit more about what Udacity does.
But I would characterize a lot of the first half of the
discussion in terms of the application of AppEngine for
Steve's class is that everybody really
enjoys doing things.
And people enjoy learning by doing.
And that is absolutely Udacity's mission.
We want to provide a high-quality, higher education
based on the belief that people learn best when doing.
And we really want to try and redefining what that means.
And so we also want to give people employable skills.
And one of the fastest growing areas in the world would be
web development.
And so having a platform where you can get people just doing
immediately within the first five minutes of a course, you
can get people already working on a project by putting up
their very first website.
It's amazing.
It really is a perfect fit for us and for our mission.
CHRIS CHEW: So, Udacity was founded by some people from
Stanford, along with various other people that just really
wanted to try and disrupt education.
Our mission is to provide the high-quality, higher education
and to find ways to give people employable skills.
I have some screen sharing ready.
So I'll go ahead and start sharing my screen.
Do you see our Udacity home page on--
That looks good.
CHRIS CHEW: All right.
So our website is udacity.com, of the home page.
We have a lot of courses.
So we have--
I think it's five or six that are in development and some
where around 10 that are actually out and live.
They're all completely free.
And they're all built-- or have been designed and taught
by leading experts in the particular field.
We're focusing on science and technology, engineering,
mathematics, the stem curriculums, I guess, because
that's what we know and that seems to be kind of what the
world needs at the moment.
And how is it that we branch out, I think, it's still to be
One of our most popular classes is Introduction to
Computer Science taught by Dave Evans.
And throughout the course you learn Python.
And you actually learn to build a search engine.
And students have actually built all kinds of search
engines that started from zero to scratch, went through the
course and built interesting things.
Steve's class was Web Development CS253.
And people actually start with no web development experience
and put up a working website that adds value to the world.
A really interesting one that I was just looking again at
yesterday is somebody who built something that looks at
tweets and defines some sentiment and analyzes that,
particularly around the upcoming US election.
So here's the overview page for Steve's course.
It talks a little bit about Steve and what
the course will do.
And you can just start it any time by previewing the class.
Or if you enroll, we'll start remembering your progress.
And at the end, you can actually get a certificate.
Courses are based around small lectures.
We try to add as much interactivity as we can,
broken up by quizzes.
And quizzes aren't graded at the moment.
And it's just a way to make sure that
you're following along.
And the course builds up to final projects at the end.
In which case, Steve's course, it was
actually building a blog.
And we assess that by actually sending out a request to your
AppEngine app-- or it could be any app really--
and deciding whether or not the blog is actually operable.
So that's the, I guess, you asked the overview.
Do you have any questions in particular?
Well, to get started maybe, you heard Steve kind of talk
about his experiences.
And I wonder if the way you think about AppEngine kind of
aligns with Steve?
And what experiences did you have kind of along the way?
I think it does very much align.
Although, lately I've been focusing more on the longevity
of an AppEngine application as opposed to just
getting up and running.
And the interesting thing about AppEngine is it actually
does both pretty well, and better than most things.
My background, I've been a web application software engineer
for 13 years now.
I had started actually building Access databases, and
then got into the web by putting them online
using PHP and MySQL.
Since then I've been responsible for applications
in seven or eight different languages, probably 15, 20
different stacks.
And so I came to Udacity with a pretty broad range, I think,
of skills as it pertains to the stacks.
And, I also pretty strong in the [INAUDIBLE], as Steve
calls it, the understanding file
descriptor limits on a Linux.
You know, stupid things like that.
And I came to Udacity actually hoping to take us away from
AppEngine, quite frankly.
I was skeptical, like Steve.
I didn't know enough about it, and my experiences with it had
been vanilla, I guess.
And actually the opposite has happened in my time here.
I turned into a big fan of AppEngine.
And it is absolutely the best place for us to be.
And it's very surprising.
And it really does force you to make the right kind of
scalability decisions from the get go.
And whereas Steve can talk about that transition of
learning those things over the years of scaling is hard.
And so to be able to just do it from
beginning is really nice.
FRED SAUER: Good, good.
I'm glad to hear that.
Can you maybe just help--
I know you talked about this in you're guest blog post--
but maybe describe you're stack to our viewers today and
what does it look like?
I think you have some custom libraries in there.
CHRIS CHEW: Yeah, we do.

We've actually had to do a lot of work, I think, to improve
our code over--
so we've been live on AppEngine for about 10 months.
And we've seen some amazingly crazy spikes in volume.
So, it's been really nice to be able
to handle that traffic.
But even then, we definitely don't feel like we handle it
as well as we should.
And so we've been doing a lot of refactoring of our code
into some libraries so that we can kind of ensure that we're
following the best practices of AppEngine.
One of those is creating a persistence library that wraps
on top of the new NDB library.
We're in Python.
And that adds some things, like helps us manage
references a little bit.
'Cause with a data store you do kind of move away from
relationships, and you kind of look to have
things in key values.
But the reality is things really are related.
And we do have to manage some consistency around that.
And that's the downside to a bunch of key value pairs.
And so that library helps us manage some of
that a little bit.
Also a big component in that is some kind of event driven
sort of processing that allows us to do some deferred
processing so that when one value of one key changes and
we need to update a series of others, we have a framework to
run the code that can make the others eventually consistent.
CHRIS CHEW: On top of that, then, we have a REST kind of
framework we call TRAILS, which is also developed by
Kaminsky, one of our engineers here.
And it's a really nice--
it's actually, I think, the nicest REST
framework I've ever seen.
It's a joy to work with.
It's very easy to build families of RESTful resources.
And then after that, it's all HTML and JavaScript.
Maybe can you share some of the questions that you get
from others.
When you tell them that Udacity's running on
AppEngine, what sort of questions do you get?
So, a lot of times it's disbelief or surprise, I guess
surprise more than disbelief, but.
And the first question that comes up is,
what about the speed?
And I think the truth is it's a little slower than if you
were building something on--
if you put servers some place and were running things low
level with a simpler stack, it's
absolutely gonna be faster.

And that's a trade off that you get for the horizontal
And we do experience latency spikes.
And we're working through figuring out why those happen,
and then what we can do to minimize those.
And we're learning there are things that we can do to
minimize them.
And so that's the growing pains that you get with
AppEngine, I think, would be learning how to minimize
latency spikes and organizing your code and your data
structures such that you don't have that as often.
And so, that would be the first surprise is how do you
keep your site fast?
So, I remember in your guest blog post talked about a few
things that you were looking at to kind of reduce the
latency of various requests in your application.
I know you already mentioned NDB which adds memcache-based
caching for free, instance caching--
FRED SAUER: It has the tasklet support, that sort of thing.
Can you name one or two other things that maybe you're
doing, looking at?
Is it restructuring your code?
Is it doing more requests or RPCs in parallel
asynchronously, braking up task queues?
Like, what are your main strategies that you're kind of
focused on right now?
So the overarching strategy is to be able to fulfill each
request with the least amount of work possible.
And so that means we've basically taken the idea of
normalization and turned it to 11.
Everything is de-normalized.
We have a lot of duplicated data.
And we're really trying to kind of--
I keep calling it pre-materializing views--
kind of take it from an Oracle background--
where we try and get the data to be ready to be read as
quickly as possible.
And so, in order to do that, we have a ton of deferred
processing which sits on top of the task queue.
And we've actually monkey patched some of the NDB stuff
to allow us to be able so that when something's written to
the data store, we can send something into a task queue to
let other code know that it needs to update other parts of
the system.
We're starting to use the NDB task list, the co-routines, a
ton to try and fire off things in parallel, and also give the
process opportunity to handle other
requests at the same time.

And I think that's a trend that's happening in web
development everywhere.
I mean, noJS or [INAUDIBLE] and Python all those things
are really taking advantage of the co-routine of patterns or
cooperative programming.

Then I think the other thing we've--
so we actually have a couple layers of cache on top of NDB.
We actually have an in-process cache for some things where
consistency's not as important.
And we don't use as much memory.
Our apps are actually, surprisingly, have low memory
So we're trying to use more of that with in-process caching
on keys that are gonna be more predictably read.
Iein, I know you're already kind of looking
at questions here.
Do we have some good questions?
We have a couple questions.
Some of them have been touched.
Let's start with Steve.
There's a couple questions for Steve.
Maybe the first one here is from anonymous
person which is--
STEVE HUFFMAN: Hopefully not anonymous.
Which is, do you plan to continue teaching at Udacity,
perhaps a follow-up course to CS253?
STEVE HUFFMAN: It's something we've talked about.
And I think I'm on board with doing that and I think
Udacity's on board with doing that as well.
Presently, my head is basically
in the sand at Hipmunk.
But when my load lightens up a little bit, yeah.
There's definitely a few gaps in the current course that I'd
really like to fill.
Yeah, potential another course or two.
That would be interesting to explore
down the road certainly.
That's good to hear.
And then we have another question for you from Udi
which is, which concepts were the hardest or trickiest for
students to understand?
And then, what would you change to overcome that
STEVE HUFFMAN: For students to understand?
The database stuff is tricky because, I think, really to
get a good grasp, it just requires time, right.
You just have to be thinking in databases for a while.
So that's challenging.
Certainly some of the concepts are really hard to teach
because coming up with meaningful
quizzes can be tricky.
A lot of like the basic HTML stuff was actually really
tedious to try to teach.
And some of the scaling concepts are hard to express
in a video and hard to capture in either a simple quiz or a
simple web app.
But by and large, I think we managed to get to the high
points pretty well.
But as with anything, if you take a seven unit course and
spend maybe 100 hours on it or so, it still takes thousands
of hours to become an expert in anything, right.
So, some of the trickier concepts just
take time, I think.
IEIN VALDEZ: Cool, cool.
And then one more question for you which
is someone's asking--
[? Varun ?] is asking-- how easy would it be for someone
to learn PHP and MySQL instead?
I guess this is probably in the context of your course.
What do you have to say about that?
STEVE HUFFMAN: I think a lot of the
concepts are very similar.
We tried to not make this an AppEngine tutorial, but have
AppEngine facilitate the concepts we
were trying to teach.
Now some things will be tricky, right.
You've got to get MySQL running.
You've gotta get PHP running.
You've got to get those two talking.
That, I think, will actually be the hardest part.
When MySQL goes wrong--
or when you run out of--
as Chris was alluding to, one day you're gonna run out of
file descriptors.

Those are the type of issues that you're gonna have to deal
with at some point.
And those are tricky, right.
Those are challenging.
Things break in weird ways.
And that's the joy of being an ops guy, I guess.
This looks like a question for Chris.
I'm not sure if he can answer or not, but
I'll just go for it.
Basically, the gist of the question is this is around the
sort of copyright status of the CS253 course.
Is it Creative Commons?
Can it be translated into other languages and shared in
other websites?
This looks like someone in the Russian Republic potentially.
I'm not sure what the status is.
Do you have any comment?
So, our courses are all Creative Commons.
And as far as I know, we intend it to stay that way.
We do actually have several translations already.
And we absolutely encourage translations.
That happened accidentally, and it was one of the happiest
moments, I think, we had here at Udacity that people found
our stuff so useful.
There's a website here.
And I can follow up on a blog post on a Udacity blog or
And we have a team called Udacity.
So you can actually search for Udacity there.
Or I can provide a link.
And that's the best place to submit new translations.
And also the translations are then put into our subtitle
system in YouTube.
IEIN VALDEZ: Cool, cool.
Translations would be wonderful.
We'd very much appreciate that.
IEIN VALDEZ: Fantastic.
Here's a question for Fred.
IEIN VALDEZ: Which is, the data store quotas
seem to be quite low.
Doesn't that severely limit the free usage for AppEngine?
What happens if it goes beyond the--
FRED SAUER: So, yeah.
So with AppEngine, we've always said that we want it to
be free to get started.
We want you to kind of start experimenting with AppEngine,
be able to try it out, kick the tires a little bit, before
you have to give a credit card and enable billing.
And so we have two of the main components that most
applications who have billing enabled, they pay for, are
instance hours.
Those are hours that there are app servers running your code
so that's when you're getting traffic.
And the other is data store operations.
And other smaller things like storage
quota on disk and bandwidth.
The first two are kind of the key components.
And so the free quota for an application includes,
currently, 28 instance hours a day.
It's a little bit more than 24.
We want to give you some room to kind of occasionally bump
up to multiple instances.
But it essentially let you run kind of a small site easily.
The data store operations are--
what we did is we kind of looked at average use of lots
of small apps, and we kind of picked a number there where we
thought, OK, most applications should reasonably be able to
use the free quota and be able to run.
So I have a couple personal sites that I run under that
free quota.
They're just ones I use with my family.
They don't get hit very often.
I don't have to pay for those, so that's great.
There are a few kind of things that people starting with
AppEngine run into quickly with respect to data store
usage that catches them a little bit by surprise.
And with just a few lines of code changed, you can probably
reduce your data store usage very significantly.
So one of the decisions we made early on with AppEngine
in hindsight that was maybe not quite the right decision,
but what we decided was the properties on entities would
by default be indexed.
That is, so if you have a customer property with a first
and a last name, by default we create a first name index and
a last name index so that you can do queries like find me
all customers whose last name starts with Hoffman.
And hopefully Steve would show up in that list.
Or first name with Chris, and we would find Chris there.
But it turns out in this non-relational world, that's
usually not what you want.
You don't want an index in every single property.
You probably have maybe a dozen or two dozen different
properties on an entity.
And you're not gonna run a query against all those.
But every one of those indexes--
there's actually two indexes, an ascending and
a descending one.
And so kind of to wrap up the story is by default,
properties are indexed.
And there's an extra penalty that you pay as a user for
writing those.
And the change is very, very simple.
When you declare your database model, when you say first name
equals db string property, in parentheses you just put index
equals false.
And suddenly you've reduced the number of small write
operations by two just for that one entity.
So every time you kind of update that entity or you
create it, that's a reduction right there.
There's another area where when I get a chance to look at
code that other developers have written, I very often see
that they're doing data store queries rather than fetches by
key even when they know that the result is gonna be a
single entity.
And queries essentially cost twice as much work as a fetch
because you're first doing a query across the index to
figure out what entities you need, and then you're doing a
second phase of actually getting the entities.
So right there with that trick you can kind of cut your data
store big operations in half.
And then with the other one, you can cut the
smaller ones in half.
So, short answer, there's some low hanging fruit to kind of
address that.
And we think that the free quota should still be good for
small applications.
So we have time for about two more questions.
I have one here for Chris from Evan in Oregon which is,
Udacity is an awesome application and I'm excited to
watch it grow, but certain oddities persist.
Each view of each course requires greater than 300k in
monolithic JSON object.
Why not a more dynamic fractured Ajax response to
avoid loading?
So, hopefully that makes sense, that question.
CHRIS CHEW: I'm curious.
I think Evan might be a plant.

So, that has not escaped our notice.
It's interesting that--

yeah, I mean, everybody makes wrong design decisions at
time, and sometimes you just have to make a decision.
Right now, when you pull down a course, you load most of the
course content in one JSON request.
And, I guess it's fewer requests, and that's a bonus.
But our courses, I think, or the content is a lot bigger
than what we envisioned at the time of that decision.
And it ends up pulling more down.
We're actually in the middle of a rewrite that will go live
in January-ish.

And within the new system we're definitely pulling
things down incrementally, and the performance
is definitely improved.
So Evan please be patient.
We're working through that refactoring.
And we're gonna give you a whole bunch of other
oddities, I'm sure.
FRED SAUER: Sounds like fix is on the way already.
I hope so.
And this will be our last question.
This is from Kyle.
And this is for you Steve which is, why do you think you
and Alexis were so successful in the company?
Do you have any parting thoughts on that?
STEVE HUFFMAN: I could talk at length on that.

There are many paths to success.
And for Alexis and I, we were really close friends going in.
And we stayed close friends throughout the entire
And when you're starting a company, being able to lean on
your co-founder is supremely important.
And one of the things that I really loved about Reddit was
that Reddit was a company with a soul.
And Reddit's personality was our personality.
And Alexis and I, like, we're fun to hang out with.
And Reddit, it was a fun place to be.
And I think considering we submitted all the content for
the first couple of months, pretending we had traffic, we
were able to kind of set the tone to be
things that we liked.
And so I think it really came down to caring, letting our
company have a personality, and letting that personality
be our personality.
And so, we considered our users friends.
And if you've hung out on Reddit, like, there's a good
chance you've interacted with Alexis or I. And if you've met
us in person, we try to--
That was important to us because our technology wasn't
very good, right.
The website was kind of ugly.
It was pretty jank.
It was down a lot.
It was slow a lot.
I was still learning how to do all this stuff.
But I think we overcame a lot of that by having a real
community and making a place that people
really wanted to be.
FRED SAUER: Great, great.
I thought of maybe one more question.
IEIN VALDEZ: Yeah, yeah.
FRED SAUER: We can throw that out there.
So I'll start with you Chris.
And Steve, I'll give you a chance to
answer the same question.
So Chris would you ever go back to kind of managing your
own machines, be it virtual machines or real machines?
Or are you kind of sold on the kind of platform as a service
model and just worrying about application logic?
I'm biasing the question a little bit, but I do want an
honest answer.

So, I would say that AppEngine would definitely be on the top
of my short list in starting some other projects.
There are some things that you can't do in AppEngine, like
you can't execute arbitrary code really.
And so, like our program submissions here, when you
submit some programming snippet, that
doesn't run on AppEngine.
We actually send that out and run it at someplace else.
And so there are some projects that you might envision that
it's just not gonna work.
And you might end up creating some other hoops you have to
jump through when it would've been net easier to do it
someplace else.
And the Amazon Web Services has the Elastic Beanstalk,
which is kind of like the instance deployment kind of
system that AppEngine has.
And so those are some other competitors.
And you always have to be aware of what your other
options are.
But as far as starting something that actually has a
chance to stick with you throughout the entire life of
a project no matter how popular you get, even the fact
that there's a chance that it'll work is extremely
And Steve, would you ever go back to kind of managing your
own servers, playing sysadmin or--
STEVE HUFFMAN: I can tell you what I would never do is I
will hopefully never install a server in Iraq again.

Now, you know the difference between AppEngine and AWS--
Hipmunk uses AWS now, right--
is whether you are actually
administering the boxes yourself.
I basically echo exactly what Chris said is if I were
starting again or if I had the opportunity to really move
Hipmunk, we'd consider AppEngine for a lot of it.
But I know there are pieces where it wouldn't quite fit.
And what I wonder is will those pieces
exist at some point?
'Cause there's things like the way we run batch jobs and our
[? Fare Alert ?] product and we care about the IPs very
much and that sort of thing.
They're kind of little issues, issues that I'm sure Google
could actually solve.
But, my instinct is to say, I want to do
as little as possible.
So the more somebody does for me, the better.
And AppEngine does quite a lot.
But, we are not a company who is gonna be made or destroyed
on our skills on ops, right.
We're not an ops company.
We don't want to think about that.
We're a consumer web company.
And so as long as that's the case, we want to outsource the
ops to somebody else.
FRED SAUER: Focus on the value end.
FRED SAUER: Well, thank you very much both you for taking
your time and sharing your experiences with us.
I hope the viewers had a chance to enjoy it.
Thanks a lot.
FRED SAUER: Thank you.
STEVE HUFFMAN: Our pleasure.
Well done over there guys.
FRED SAUER: All right.
So, just to wrap it up.
If you followed the interview, liked it, and you haven't read
the blog posts yet that Steve and Chris contributed the last
couple days, do check those out.
And we'll see you back here on the Google Developers Live.
Thanks everyone.
IEIN VALDEZ: Thanks everyone.
FRED SAUER: Have good day.