Google IPv6 Implementors Conference 2010: Content Networks and Enterprise Networks

>>
Happy to have Tom Coffeen from Limelight here. >> COFFEEN: Tom Coffeen from Limelight. Is
it just me or did Vint Cerf bear a disturbing resemblance to Dr. Breen from Half-Life 2?
Anybody else notice that? Okay. I'll just--I'll go over briefly what Limelight has done with
v6 to this point. I was here last year and we had just turned up v6 in the core and that
was prior to us engaging in a proof of concept, a demonstration of v6 content with Netflix.
Since then, I had hoped to have a lot more progress to report but Limelight is in a relatively
awkward position of basically having to wait on the content developers and the content
providers to generate demand that compels us to then support v6. I've done as much as
I've been able to do as a sort of our official evangelist within the organization to get
people excited about v6 in the organization and recognize that the technical challenges
that we're going to face in adopting it are better approached earlier than later as we
learned with the deploying in the core and getting the v6 Netflix demo out. And I think
it's a common story for folks who've done it to this point, I mean, Facebook, great
example, you did--definitely, I detected the notice of a bit of shock that it was, you
know, as easy as it turned out to be. We had the same type of experience with much of what
we did initially, things that just worked right out of the box, things that required
very minimal amount of configuration change and--so, basically all the low hanging fruit
is, you know, has been grabbed up to this point. But going forward, we're really at
a point where until we actually get some of the business demand driving v6 adoption within
the organization, there's not a whole lot of resources that I can sort of get pushed
in the direction of getting additional v6 services up and running. What we have to this
point though, just to go over this slide quickly, IPv6 in the core, we've had that for a little
over a year. We'd like to think that we have minimal time to provisioning v6 at the CDN
edge, that it's something that we'll be able to offer any customers that come along and
say, you know, we definitely need this today. Again, it hasn't--the demand hasn't necessarily
been there, so it's tough to see where this will fall over, if anywhere, when tested.
We have v6 transit to--actually, this slide needs to be updated, we have v6 transit to
four providers, peering with critical networks. Obviously, as a content delivery network with
its own backbone, peering is pretty critical to us to get traffic offloaded as quickly
as possible. We've had quite a bit of luck with providers who've been willing to peer
v6 with us and I don't anticipate that's really going to be a problem going forward; the issues
aren't necessarily in the core as far as getting folks aware of the need to get v6 up and running.
We're currently using a parallel DNS architecture that allows us to serve Netflix over v6. We
did that as a requirement to keep our v4 production network stable, to make sure we didn't have
any issues with our DNS architecture that provides the secret sauce for our content
customers and that's something that ultimately, we need to synthesize into one DNS architecture
for reasons of performance and it's something that, again, until we actually have more demand
for v6 services from customers, it's unlikely to happen. We're delivering v6 traffic to
eyeballs today and providing v6 transit today to several customers and obviously, being
here represents close engagement with other industry IPv6 early adopters. And something
I certainly hope to take away from this is some ideas of where to go next in the absence
of actually having true demand from our customers. So, some things that remain to be done for
us and, again, until we have that pressure from the business side, it's a real--it's
sort of a really open question as to how quickly these will be addressed and in what order.
IPv6 functionality for all standard CDN products, the CDN product offering for Limelight is
pretty vast. There's a lot of services that are sort of either represent corner cases
for customers where we've built a custom service or perhaps, there's a portion of the service
that--well, in any case, again, it remains to be seen how much additional work there
will be to enable v6 for a lot of the product offerings that we have. For each silo within
our organization, we've got a sort of mixed bag in terms of buy off on v6 adoption and
in terms of status. If they bought off like actually where they are in terms of adopting
and deploying, most critical to this is probably on the finance and building end of things
where a lot of our back office systems, some of which are sort of home grown and hacked
together and others are, you know, more turnkey and appliance and software package type things.
It's really a big question as to how much additional effort is going to have to go into
those systems to allow us to fully bill and support v6 for the customers when they show
up. So, that's something that we're still struggling with a little bit. And then, something
that I'm trying to get a handle on is exactly what is driving v6 adoption where customers
are asking about it and if they're not asking about it, why they aren't asking about it?
Or in other words, we're trying to get our sales force engaged to the point where they
know how to articulately ask, you know, what's driving customer either interest in adoption
or lack of interest in adoption, and I think those data points will be helpful for us to
try to push things from the other direction in terms of encouraging customers to think
about what v6 adoption is going to entail for their own organizations. So, it's kind
of backwards in the process but, you know, I hope that there's an opportunity there to,
you know, to really sort of drive things from a different end and get more customers interested
in tackling the v6 problem upfront, which will in turn really put pressure on our own
adoption efforts to see where the weak spots are or things that we haven't actually taken
care of and gotten working yet, et cetera. And ultimately--okay, so--yeah, you know,
things like--that we've been talking about such as NAT in the core, you know, been talking
about over the last year for much longer than that, NAT in the core, broken geolocation,
broken DNS resolution, things that will negatively affect existing services. I'm hoping that
some of that will be sussed out by, you know, an effort to find out why customers are interested
in adoption when they've--when and if they are. And then ultimately, the goal is v6 functionality
for any Limelight Network product or equivalent v4 work around. So, you know, obviously we
just want things to work and v6 is one solution to certain challenges that we're going to
face collectively. You know, my--as I evangelize within the organization, I try to make that
point as frequently as possible that, you know, v6 is--what we have to do to make v6
work is one possible solution if we can, you know, if there are things we do to make v4,
you know, get the job done. And, you know, perhaps a Facebook success with LISP is an
example of that in terms of, you know, very small v6 footprint. Current projects, v6.limelightnetworks.com,
that's up and running. We're trying to get v6 SSL working for Netflix in particular.
We've got issues with the load balancer that we're currently using and that's something
that we're in the process of trying to tackle with the vendor. Synthesizing the v4/v6 DNS
architecture, as I mentioned, doing things like getting v6 enabled on corporate LAN segments
and then looking forward to, at some point, to actually white listing all of our space
for access to v6 services over Google. And geolocation for IPv6, we're testing the clover
database currently and have had some luck with that. It remains to be seen how quickly
we can get existing products that are relying on our clover database implementation to actually
use the v6 geolocation piece that they're offering. And then finally, you know, maybe
of some interest to folks here would be the amount of traffic that we're actually seeing
over v6 being that we're delivering it to clients. This is a quote that caught my eye
from Martin a little while back and I thought what was--what was rather amusing about the
quote was the 30 times ratio that he mentions without necessarily providing a base number
to apply that ratio to. So, in the spirit of that particular quote, here're some graphs.
Probably without the actual Y axis being provided there, what's of most interest is the fact
that the traffic is still very bursty, we don't really have sustained v6 traffic levels.
Those are all single sessions of people accessing Netflix over v6.
>> It is bits per second? >> COFFEEN: Yeah. It is--yeah, that's right.
Yeah. So, you can--you can interpolate from that. And I mean that's basically it. If anybody
has any questions, we'd be happy to attempt to answer them at this point.
>> Thank you. We have five minutes for questions or so.
>> I have a question. >> Go ahead.
>> Please. >> I was just going to say, that I really--I
don't really understand the interface between CDN and those customers. So, who has to deal
with, like, bugs analysis for v6 addresses? It's like have you just started serving, I
mean, like... >> COFFEEN: Yeah. That's...
>> They have to audit their systems as well as you audit in your systems?
>> COFFEEN: That's a service that we definitely provide and that's--you know, there's a big
question mark as to what amount of effort will be required to support log collection.
We know that we can do it today successfully but the question, as with many of the v6 services
that we've turned up, is to what degree will it scale? It's a question that's impossible
to answer just based on the lack of actual v6 traffic and the lack of v6 customers. So,
again, you know, once things start to ramp up, there are actual portions of our offering
that seem to work very well today that we're, you know, not necessarily 100% confident will
scale. So, I hope that answers your question. >> Yeah, that does. It makes sense. Thank
you. >> COFFEEN: So, that's it? All right. Thank
you. >> Thank you, Tom. For those that don't know,
Lorenzo is the tech-lead for IPv6 here. >> COLITTI: Sorry, is that working?
>> Yeah. >> COLITTI: All right.
>> You know, we've been at it for the last two and a half--almost three years, two and
a half years. >> COLITTI: Only? Feels like longer than that.
>> Yeah. September or August of 2007. >> COLITTI: Oh, yeah.
>> Yeah. >> COLITTI: Okay. So, I think a lot of you
will have seen and heard me, you know, speak before. So, I'll just touch on a few key points
of what I think are the most important issues that we're facing. Number one, I still--I
still see in many other networks, especially content networks, because access networks
seem to be now saying, "Oh, dear. No, this is really, really bad. We need to fix it,"
and they're coming to content providers and saying, "When can you give us your stuff?"
and we don't know. So, again, why is this? We know all this. We know all this. This is
a nice way to present it, right? We had--you know, we had 50% of the space free 10 years
ago and now we have--well, it said 10% now Takashi-san and his counters say 6%. So, this
is--it's the writing is on the wall, right? So, I don't think anyone disputes this anymore
but what are the implications? So, people are going to buy addresses or they're going
to do double NAT. And double NAT's expensive, it's hard to manage, it increases operational
cost but, you know, for a content provider, the major points are the ones that Jason already
made, right? So, what's the semantics of an IP address? Well, how do you geolocate it?
How do you block abuse if somebody ends up on some blacklist? Does he take out all his
neighbors? And if you don't want to take out all his neighbors, then how do you build your
infrastructure to detect and intelligently moderate those spikes so that it doesn't sort
of DOS your back end systems? Then, you know, port exhaustion, one that I think he didn't
mention was port exhaustion, HTTP intercept. So when the user runs out of ports, the--I
think the broadband form recommendations for that are to intercept HTTP connections and
put a helpful warning message with, "You have run out of ports. Please contact your SP.
You may have a virus." And while that's nice to the user, it's not nice for people that
want to deliver content or advertising or you name it. So, these are bad things and--so,
there are, you know, strong reasons for a content provider and these are the reasons
that drove us essentially to implement IPv6. So--and also, the lack of the v4 space and
the pressure on port exhaustion--if nobody does v6, the pressure on port space will go
up higher and higher, right? Because assume that no one's moved to v6, the transition
is somehow failed. This is a sort of extreme scenario but what happens then? So, the pressure
on port space goes higher and higher, we crank up the use of one IP address more and more,
more and more users are behind that IP address, what happens? Your timeouts--your net timeouts
have to get shorter and shorter. What happens on the content side, "Oh my edge.exe application
is not working, let me crank up the keepalives." And so basically, you end up in an arms race
where the application sort of stops working at some point because you--the connectivity
just isn't there anymore. So, this is--you know, when you--this is, you know, 10 years,
20 years away or whatever it is, in its net time it's probably more like 10 years, but
if we don't move to v6, this will become a problem. So, you can do NAT, but you can only
do it so far, right? So ultimately, we have to move to something else. If you look at
the population curve--I have a graph on my other computer of course, but the dog ate
it--but it--you know, there's this nice little curve of world population and there's this
nice little flat line that's about just over halfway; that's the number of IP addresses
in the whole--on the whole v4 internet, right? And the number of address is lower than the
number of people, so, ultimately, that will be a problem. So, okay, technical constraints,
v6 only deployments. Mr. Burn (ph) there is eager to do v6 only. There are various things
that can't be numbered out of v4 space anymore because there's just too many of them. The
RIRs are being reasonably strict about handing out v4 blocks, so it's not very easy, and
networks have been numbering that, they set up proxies out of private space for a while
now. So, if you want to get your content onto those set-up boxes then you need to be able
to speak v6. New applications. Yes. It's not clear to me how this will pan out. We--it
would be nice to think that v6 will restore end-to-end. For that, we need either people
not to firewall or we need a solid protocol to open holes in firewalls. Now, to me, a
protocol for opening holes in firewalls and no firewalls is effectively the same. I don't
understand the difference, but maybe somebody does. No, there is no killer application for
v6. Rather, there is one. Yes, it's called the internet. If we want it to keep working,
this is what we need to do. So, we can't enable v6 for www.google.com; you know that. If you're
at home, you do host IPv6 at google.com, you don't get an address. From here, you might.
You do, that's right, you do. Here we do. So--and this is--this is where I sort of say
this is what pretty much how we think it should work. We're, you know, we're pretty close
to it. So, you know, you have www.google.com. It's dual stack and you can--you know, you
can--you can ping it, you can connect to it, you can run TCPDUMPS and it's three milliseconds
away because it's in the same place as before us, right? Right? So, this is what we want
to reach, a situation where the content side is ready for v4, v6 in the same way, it provides
the same level of SLA. And of course, we're not there yet, especially, you know, there's
a lot of Wild West type situations in home networks where we just can't provide this
level of service but this is the goal. And this is what we need really for a successful
transition. So, I don't know if any of you have seen this before or you can do it on
your own laptops. Just connect to Googlegoesv6 and, you know, get a glimpse of the future,
if you will. Let's see if I can--I mean, the demigods, I'm sure, will be against me. If
I remember--if I remember my password right. Yeah, that's not going to work. Come on, okay.
So, all right, oh, dear, there's a lot of traffic already. Let's see. That's my favorite expression. Anyway, so
if you--wait, okay. So, I mean ideally, you should be able to--let's see--just do this
and just see the packets flowing right? And again, this is--this is seamless to the user,
it just works. So, this is what we need, right? The packets just flow, the application keeps
working and it'll work, you know, and five years from now, it'll work better on v6 than
on v4, but that's only because we've done the back end work, right, to make it happen.
YouTube is the same if we--I mean, if Eric had run the Vint video with a TCPDUMP behind
it, he would've seen the packets flow over v6. So, that's what we think is the way forward
and the only way forward really because there's just too much variety out there to have a
single model. We have to provide both services for, you know, years and maybe decades. Anyway,
to get back to my point, so, we can enable it. There is no technical restriction on our
side. Can you see this? Not really. There is no technical restriction on our side or
even, at this point, a scaling restriction to providing the service for everyone. What
is--what is holding us back is that we know that, you know, again, the numbers are one
in 2,000. One in 2,000 users won't be able to get to us anymore if we do this for the
whole internet and that is a high price to pay and it's unacceptable at this point, right?
So, that is why we started with white listing and many of you know how it works. Basically,
if your resolver has--is in our magic list, it will get v6 addresses for all our properties
just like you do here today. People like it but the problem is that some people turn it
on and then their users say "Oh, I can't get to Google anymore," and "Did you call your
SP?" "No, I didn't call my SP because it's only Google that's broken, so it must be Google's
problem." And so, this is a problem for us so, we--and this is the reason why we have
to be very, very careful in turning the service on because we don't want to provide a lower
SLA for our users and because our users might not know anything at all about what went wrong.
So, this is really the big challenge that we have at the moment. And so, you know, everything
else works, they won't call their SP, they'll just, you know, go somewhere else which is
not you want if you're a content provider. So, we'll talk about that tomorrow. We'll
talk about what our numbers are like, what the causes are, you know, what we know. We
don't know that much yet, but we do have some pretty solid data at this point on sort of,
you know, millions of data points a day, so we do have some knowledge, so we'll talk about
that tomorrow. Something from the network, some lessons learned just from our experience.
I mean, we've been at this for a while and we have pretty solid infrastructure at this
point. The implementations do mostly work, but they do have bugs because nobody has really
kicked the tires, right? You run it and there are some bugs on the next slide that you think,
"Well, how could I have found that?" So, yeah, don't expect something to work just because
it's supported; that's something that you know as network engineers. One thing that's
useful--that we found useful is if you find a bug in the lab, you report it, and you keep
going because you need to find the next 10 bugs. And what was useful for us is find the
10 bugs, report them all at the same time so they all get fixed in the next release.
You don't have to wait one bug per release and--because then you'll never qualify it.
And some people have been saying "Oh, I'm trying to qualify..." I won't say the version
number because it would give away the vendor but "I'm trying to qualify version X, and
this is my third try that's not working and every time, I find a new bug." So, don't that.
Try to find all the bugs at the same time and then get them fixed in the same release.
And one thing that you might want to consider is if you can work around the bug in some
way that will survive production use, trial it, because I guarantee that once you trial
it, you will find another bug and this bug will be very strange. So, here're some examples.
So, if a firewall filter has a 1-bit match in bits 32 to 64 and then just after that,
a 2-bit match in bits 64 to 96, then the second match will not work. How do you find this
in the lab? You get very lucky or you put into production and you see what happens.
So, in particular circumstances, the FIB and the RIB may get out sync due to race conditions
when pushing updates. Are you going to find that in the lab? There's a better chance of
that, yes. But maybe still because you need particular flapping scenarios. One of my favorites
is if you--if you loop a link for a backbone testing, if you're circuit vendor loops links,
you trigger DAD, duplicate arrest detection, you're interface goes down, ISIS points that
way because the shortest path is that way, and your traffic gets black holed because
it fails neighbor discovery. What happens? You can't shutdown the interface because that
doesn't fix the problem; you have to deconfigure the interface. Now, so, what do you do? Well,
luckily, we--in the release that we are running, you can disable duplicate arrest detection
so you can do that. But if you're running a release that doesn't allow you to do that,
you're stuck. So--oh, and if Linux gets a packet too big message on a receive-only interface
with no route pointing out of it, it ignores it. How are you going to find that? Well,
again, you might find it in the lab if you're lucky. But the race condition of the FIB updates,
we found that after months in production on a, you know, reasonable number of data centers.
So, it's unlikely enough that you might not see it in the lab at all and it is enough
to break your v6 routing and to break your service, so test and trial if you can. So,
yeah, this is, I think, you know, at this point, our strategy was--is well known. We
started with a band of volunteers, a merry band of volunteers, we used our 20% time,
and then somehow we managed to get something out there and to sort of get a snowball effect.
And I think--and you know, at the last step, management got involved. This might not work
for you; it did work for us because Google is a very engineering-driven company, but
your mileage may vary. But ultimately, management involvement is necessary. So you do need buy
in and I think these--the reasons that I was talking about earlier, those are the reasons
why we really need v6, to keep stiff--to keep stuff working, right? It's a business continuity
issue. Yes, don't go out and buy gear for v6, just fold it into your normal upgrade
cycles. Okay, we started in mid-2007. We had all our services running in late 2008, early
2009. We turned on the white listing program and then we turned on Maps, we turned on YouTube;
took a little longer to get those services done. YouTube in particular was a hardware
issue; particular vendor gauges didn't do v6 in hardware, it fell over to 100 megabits
and it's not acceptable. And there's a lot of work since then that has gone on behind
the scenes to make the implementation more solid. But one of the luxuries we had as the--as
an early adopter is that you don't need to be as careful when you're not throwing a 100%
of your traffic out. Just a few statistics, these are--I don't--I haven't looked at these
recently but I think they're still pretty much accurate. v6 is growing. It's very small
0.2 %, 0.3 %. It's all in France because the French is--the French thought of six already
first and they said, "Hey let's deploy it," and they have a few hundred thousand users
opted in and they all come in everyday, they clock in over v6 and it's fine. I think Grinter
(ph) will be talking about this tomorrow, "Please don't use 6to4. Please don't use Teredo.
Please invent a time machine and sort of remove it from your FC list or something." It is
a real problem especially 6to4, especially implementations. Let's try 6to4 before v6
and before v4. And this is our graph. As you can see, it's a little bit--this was the launch
date. That is lunch time, as you see, we turned it on and when we thought, "Oh, okay, we'll
go to lunch now because can't possibly break, can it?" That's not true, but anyway, this
is the graph. You can see the traffic, when you turn on code A's--since negative replies
aren't cacheable, the traffic just goes up straight. It [INDISTINCT]. Our graph is a
little bit more smooth, so there that--of course, we don't have axes either but there
is a smooth, aggravated traffic here, you can see that from the graph. And that's it.
As I said, we will be talking about white listing more tomorrow and then looking at
the challenges we face and what our data looks like and what we think the problems are. And
we don't have the answer at the moment and we hope that, you know, discussion will stimulate,
you know, answers for us as well. Thank you. >> Got time for questions?
>> COLITTI: Yes, yes. >> LEE: Mon Lee (ph), Hurricane Electric.
In a Purist world, we could all agree with your 6to4 statement but in a real world, we
can't. It exists, it's out there and as you and I have discussed, we've spent a lot of
time trying to at least correct what's out there. It can never be perfect. The "sucks
less" comment is a more negative way of saying it, but at least it's getting better. How
can you--how can you convince more and more service providers to at least do some form
of 6to4 work? Earlier, we have this discussion with John from Comcast earlier, but how can
you convince them to do more work in order to improve things and at least deal with their
own users when, in a way, you're trying to sort of say, "Let this problem go away by
not discussing it"? >> COLITTI: So the problem is that 6to4 is--that
so--and the question is why--for example--and one of the, you know, follow up questions
to that was why do you not run your own relays, right? And the problem is that with 6to4,
it's the last mile, it's the last link that's the problem. You get these packets coming
out of 6to4 and they just die. >> LEE: So that's why I said it's an access
providers issue. This is something that John understood at Comcast and other people have
slowly started to understand. So at least dealing with this traffic has improved things,
maybe in some cases, dramatically. There's an awful lot of traffic being relayed and
that we are and other people are doing it as well. But it's not a case as a content
play, it's not your responsibility to run a 6to4 relay.
>> COLITTI: Oh, it is. It would be. It would be.
>> LEE: Only to a very slight extent compared to access networks.
>> COLITTI: Oh not in relay, okay. But our own routers, yes.
>> LEE: So why not take a different approach and sort of--instead of trying to get it ignored,
why not go out and use your, if I can use it, pull bit to convince more and more access
providers to actually focus on this? >> COLITTI: Say you can convince the OS manufacturer
to disable it by default. >> LEE: That history, it doesn't work.
>> COLITTI: No, that's the only way--that's the only way we can get past this problem,
because there are--there are gateways out there in home networks that are announcing
RA's and that black hole packets and they've been sold and they're in the field and nobody
is going to upgrade them and nobody is going to replace them. And that's the one in 2000
users and that will not change. The only way to get around that is to--is to--is to, you
know--you can't make it work. You have to replace the CPE. If somebody has a vendor
Xbox in his home network that doesn't do V6--oh I'm sorry, that says, "Oh here, I'm a V6 router
for you and I'm going to black hole your packets real well," what do you do? You tell the users
somehow--you find them and you tell them please upgrade your box? How are they going to do
that? You tell them--the most you could say perhaps is here's a new box, but that's not
something we can fix, right? But the OS manufacturers could fix it.
>> [INDISTINCT] >> COLITTI: Right. That--so, the question
was on any given day, how much brokenness on v4 do you have? The answer--the easy answer
and the sort of a--the easy answer is we don't know, but that's not actually true because
we do see flakiness and our measurements are good enough at this point that we do see non-zero
brokenness over v4. But I can tell you that there's a nice measurable difference there,
and when you're trying to make--when you're trying to make--when you're trying to go to
a service owner, right, and say "We can do this v6 thing and it's about five times less
reliable than v4", what's the service owner going to say? So--and with that said, I agree
that, you know, 6to4 exists and we should run our own routers. It--various things make
it hard for us. I think we should do it, frankly, from the front ends instead of doing relay
traffic. But that's just arranging the deck chairs, right?
>> All right. >> COLITTI: Yes, that's true.
>> So about the percentage of the IPV6 users, for me, it's not surprising. It's still small,
because we are now trying to solve chicken and egg problems, but I'm curious about whether
other Google people understand or have the longer time view for the new--for the migration?
>> COLITTI: Do you mean does our management understand that we have to do this?
>> Well, other engineers not focusing on network-related issues. What do you think?
>> COLITTI: Awareness is spreading. It's--that's certainly a challenge, because typically,
application developers may not have specific knowledge of networking especially when you
start to become a large company, they focus on their applications and they have clear
boundaries between, you know, their applications and the rest of the world. And the packet
is coming in through magic and I send my packet here when it goes out through magic, right?
There's a lot that can be done especially if you have a sort of common source space
that everyone can access. There's a lot that can be done by prioting (ph) SHIM layers inside
your code or by protecting pieces of code that no--may not be v6 ready, Steiner's (ph)
going to talk about all of that. And so, you don't necessarily need awareness before you
provide--start providing services. You need awareness for the last, you know, 10% or even
1%. >> Oh, my question is about--with the--it's
like--just don't need to care about that, that's fine. But if they even try to dump
the idea of migration due to the smaller percentage of users, that's what I'm afraid about.
>> COLITTI: You make it clear that that's not acceptable, that this is not going to
go away. >> Okay.
>> COLITTI: And I think that message has come around at least at Google. It's clear to people
and that's why I say, ultimately, you need management buy in from this, because ultimately,
once management knows that this is not going to wait, just becomes a matter of time, right?
It will happen and so that's one of the ways to attack the problem. And the other way of
course is if you get a service serving v6 traffic and they–-and you pass them live
traffic, and they introduce a regression that breaks their service, they're going to find
out real quick and they're going to roll it back.
>> Yes, of course. >> COLITTI: So it has that aspect as well.
>> Okay. >> So on the issue about, you know, 6to4,
whether it's a good or a bad thing, I'd like to offer an alternative view that says that
it's actually a very good thing because it's causing a lot of angst and a lot of urgency
to get things fixed that might not otherwise have occurred, right? You know, to Martin's
point, you know, service providers can fix this. Your issue really is that an unmanaged
thing is getting in the way of your managed service and the managed services of the last
mile delivery. Well, as long as people are doing managed services, the 6to4 never comes
into play. It's the fact that you don't have the managed service that you want and this
automated thing is out there, it's creating a little bit of urgency that I think--you
know, so it's actually a good thing, because it's causing people to wake up and pay attention
in ways they might not otherwise have paid attention.
>> COLITTI: Well, it's not causing the users to wake up and pay attention and then the
users are the ones that own these boxes but sure, I suppose every cloud has a silver lining.
>> Yes, the point I was going to make, Tony, half of your point was my point that the best
way to fix the 6to4 problem is for the service provider to provide a v6 managed service,
whether it's, you know, 6RD dual stack or what have you, because then the hosts won't
try the 6to4 anymore. >> COLITTI: Not really.
>> Oh, come on. >> COLITTI: Not really, because my--the--I've
seen your network engineer at Google happen to wire one of these gateways and noticed,
because he's a network engineer, that he wasn't using his Hurricane Tunnel and he didn't know
why. And then you run SNAFU and you said, "Oh, but this box is sending me v6 through
advertisements, and my machine is picking them due to--because they're the most recent
ones." >> The most recent?
>> COLITTI: I don't know what, but he was using--he was using...
>> But if I get... >> COLITTI: ...He was using...
>> If an OS and I get a 6to4 and a global, the source selection policy is, I think, we'll
always choose the global. >> COFFEEN: In the--in the router advertisement
provides you with two pieces of information. One is the--the IP address of the default
gateway which is linked local and another is a prefix. Those--that information is not
necessarily tied together. You know that, right? It's the RA gives you a default lifetime...
>> Sure. >> COLITTI: And this is my link local address
and then it says, "These are your prefixes." So the stack simply says, "Well, I mean these
two pieces of current information are not necessarily correlated, I'll use the lowest
router ID," which is what most implementations do, right? Or the new--the most recent one.
And of course, he went into the control panel and somebody reported on analog "Here's how
you turn it off, you go to this hidden [INDISTINCT] then you go to," I forget what it was, "System.extension
and you click here or you unset Windows Vista ultimate compatibility and your 6to4 goes
away." But he did that and there's no such page, so he can't turn it off. So his only
recourse, and he knows what he's doing, is to replace that box.
>> Okay, the host should need to be fixed. >> COLITTI: The hosts.
>> I take it back. >> COLITTI: So.
>> BYRNE: Cameron Byrne, T-Mobile. So, the problem that we're trying to solve here is
that there's not enough Internet addresses. 6to4 doesn't solve that problem, it kind of
generate IPv6 packets or reachability to some degree. But the growth in I.P. addresses largely
comes from the mobile space and the mobile space is largely private address base and
Bogons that don't work with 6to4 anyways. So...
>> COLITTI: Did I say--did I say that the gateway needs a public address to generate
advertisements? >> BYRNE: We have seen...
>> COLITTI: We have implementations that do that. We have implementations that...
>> BYRNE: Yeah. Absolutely and that's a substantial source of brokenness in 6to4 but this is a--you
know, where's the solutions? What are the fixes and how they match up if I.P. addresses
are being consumed in mobile? Mobile is using predominantly non-routable addresses; non-routable
addresses don't work with 6to4. I blocked them into my network because they create a
lot of invalid half open sessions in the CGN that's already in place, so it doesn't work.
>> Yeah. So if you're saying, "Are mobile networks working?" then that's good, and we'll--you
know, our data will show it which means that we'll be able to turn it on, right?
>> RUDOWSKI: The only count that I'm making--unfortunately Mark (ph), this is John Rudowski of Comcast.
Mark (ph) walked away. I think, unfortunately, what'll happen with 6to4 even enabling need
of IPv6 or some other form of v6 isn't initially going to make the 6to4 traffic drop. As a
matter of fact, I think it'll get--it'll increase before it goes away. Some of the stuff that
I had seen a few months ago have given me very good reason to believe that, because
there are some things out there that are not--there--these traffic is not always destined to Google or
Facebook or whatever. It's going to some place else like a--like some other host in the network.
And native IPv6 connectivity will look attractive to some of those applications until you reach
the point where there are more native hosts or 6RD hosts in the network will that traffic
then start to drop. >> COLITTI: Oh, yeah. I wasn't suggesting
that the relays can go away and a far from it, if the relays go away, it's going to get
worse because the packet is going to get dropped. >> RUDOWSKI: I totally agree with you. That's
the bottom line, right? >> COLITTI: And yeah, as you said, I think
Martin (ph) was saying that half the traffic on the relay is 6to4 to Teredo, and it's,
you know, machines with public addresses talking to machines with private address because the
app happens to support v6. That's not a good state to be in but...
>> RUDOWSKI: Thank you. >> Haythum.
>> BABIKER: Babiker. >> Sorry, I was just going to say...
>> BABIKER: Yeah. >> Haythum is responsible for a huge percentage
of our corporate IPv6 deployment, if not nearly 100%, the reason that we have v6 to our desktops
and to the wireless. >> [INDISTINCT].
>> Yes. >> BABIKER: Thank you. So, part of the corporate
network engineering here at Google, maybe you are aware or not, Google run corporate
as similar as an enterprise network and content network as a broad backbone. They run independently
but they are part of the same group eventually. So, the deployment for IPv6 motivation, I
think, is clear for anyone. The main purpose for it in Google actually is to provide the
engineers the ability to connect to v6 backbone and be able to support application and test
and validate the features on the v6 network. Obviously, that also will come with a benefit
added feature, you will be innovation and early adoption of features and support of
a--be able to lead the way and discover the new tool on that new technology. So, what
is actually Google Enterprise Network? It's a--contain of distributive offices around
the globe. I don't know--those number may be not accurate but it seems there are more
than 19,000 users around 70 offices distributed in 33 countries and five continents. We, as
multi-vendors, we augment everywhere in the network and we will talk about the vendor
support and the issue we discover during the implementations. Initially, the project start
in 2008 by my colleague, Yeng Martinsville [ph], [INDISTINCT] who's not around to see
today. But in summary, it started as any major Google project starts, it was 20% deployment,
initial small lab deployment where we have a host, connect directly was manual GRE tunnel
to a single router. In--obviously, it's not scalable, very limited support, and that was
the earlier stage and then we move to dedicated lab. The lab was GRE tunnel to 6to4 routers--it
was 6to4 I mean, six connected to a four backbone, and move away to Dual Stacks the actual network
inside the office and route the GRE from the peering edge router. The second phase, when
the WAN provider was not supporting v6, that basically was interim phase and then 'til
we going to the phase three where we manage those dual stack all the way to the upstream
providers, the upstream provider here from our backbone network beside the MPLS provider
recently and several ISPs. So dividing the addressing plan, as recommended, /64 for each
subnet even for point to point link--that's what helps a mobile I.P. Also the address
allocation is not limited so why not--we /56 per each building. Imagine every building
in this campus has /56 and aggregate that /56 to /48, and advertise as /48 on the edge.
We have acquired from the registration--as the local registration, /32 now back and /40
in area. Certain protocol, HSRPv2 has worked like a charm. So as soon as you enable the
standby better until you'll to be able to have whole space--whole space redundancy.
OSPF version 3 has an IGB and MP-BGP, and for addressing, we're still using auto configuration
stateless because we discover the IPv6 have very limited support in multiple OS. Windows
XP, Mac, even Linux on the current release on the SEB, I think is--you need to upgrade
to future version, so it's still not stable. Routing policy is very simple, advertise,
accept default from your upstream provider, whereas for default, if you're going to need
the whole routing table and advertise office aggregate. So, yeah, what is the main challenge
we faced so far? I think the hardware vendor platform or hardware support and software
support is challenge. In general, certain vendors may support certain features over
v4 only or support IPv6 feature on the software on the box, so that will have a scalability
impact on your support. Service providers, when you talk to any service provider and
ask them about IPv6, certain response like, "A v6? We don't have it yet" or, "We can give
you v6, but the SLA will be very--it doesn't match the v4." So, either deployment will
take longer or you will not get the same service level agreement. I feel the ISP are not motivated,
okay. Not motivated because of the revenue generated from the v6 might not match what
they have when they go to enable a new circuit. Okay, that's for the Q&A.
>> We have couple of minutes for questions. Yeah.
>> Do you service Internet from each campus or do you do it from just a few hub campuses
or...? >> BABIKER: We service Internet from each
office individually. >> Individually?
>> BABIKER: Yeah. >> So you have a separate relationship with
different ISPs at each one? >> BABIKER: Yes, yes and MBLS provider are
our transit ISPs. Yeah. >> And then, between your campuses, you're
just leasing private lines from carriers or...? >> BABIKER: Inside the campus network, we're
on a--inside the campus... >> No, between campuses.
>> BABIKER: Between campuses, we run MBLS and MBVM...
>> But I mean, what are you purchasing from carriers to interconnect your campuses?
>> BABIKER: Okay, we have three vendors one of them is MBLS, PBN, and Transit ISP.
>> So, you're using I.P. from your carriers or private lines?
>> BABIKER: We've--we use the private lines. >> Okay.
>> BABIKER: Private lines, yeah. >> Okay.
>> All right. Oh, wait. >> Quick question. How do you manage private
extension addresses? Do you allow the use of private privacy extensions?
>> BABIKER: Yeah, we use privacy extensions, yeah.
>> So, do you have any problems authorizing the--a host that using some certain addresses,
IPv6 addresses? >> BABIKER: Yes, we had a problem with the
private extensions in certain hosts, but we managed to fix it by changing the configuration
on the host. I took... >> There was another observation--another
problem as well. We we're--where you're allowing privacy addresses and then when you use Kerberos.
So trying to authenticate the Kerberos, it would put all of the addresses that it has
in its ticket including everything that's expired or is in--is in deprecated state and
only valid for–-that would never be used for any connection anyways, so it would send
off--you know, and even if you pack it, it was way too big and have 16 privacy addresses
in it, none of which were being used. >> BABIKER: Also, it cause another problem
on the association table on the wireless, whereas a private extension being counted
on the user table on--then our service run of the license is on the wireless providers.
So, yeah, that's all. Thank you. >> Oh, one last question?
>> I have one last quick question. Do you have any statistics about traffic volume?
Percentage or say a... >> BABIKER: Not with me here, but...
>> Okay. >> BABIKER: I can share it with you if you
want. >> Yeah, I 'm just curious, because it's--I
may just have this register only a small fraction of people are using it.
>> BABIKER: It's not as--I think it's less than 2% or 3%. We're still--I think the internet
connectivity is still over v4, but... >> Uh-huh.
>> BABIKER: The v6 to the external is quite substantial enough.
>> All right. >> All right, you want the last question?
Is it quick? >> You might've just answered that. I was
trying to get an idea internally because I assume you have a lot of v6 enabled hosts,
you have--the campus v6 enabled. >> BABIKER: Yeah.
>> Your users internally probably uses Google services a lot. What percentage which is,
you know, v6 enabled, you're probably the ideal case for campus that's got a lot of
v6 running on it. >> BABIKER: Yes, but the stats, actually I
don't have it with me here, but we have substantial traffic going on v6, as Lorenzo show.
>> Okay. >> BABIKER: So, you will have enabled the
quad A and the A and you will see a lot of traffic going to the internal--the external
Google services and internal, but I don't have the number exactly on my head.
>> Do you imagine anytime in your foreseeable horizon having one of your sites be v6 only
or not? >> BABIKER: It's going to be hard for two
reasons. You still going to have someone who's going to run over v4 and you still have to
knock to them or access them in certain way. >> Which should proxy the whole Internet,
right? >> BABIKER: Yes, that's one problem. And the
second thing is actually trying to get everything to talk over v6 even inside the code write
network is challenging. >> Okay.
>> BABIKER: I don't want to mention vendor's name. There's a lot--there are a lot of them
who are--a lot of vendors who don't have even a plan to support v6 yet.
>> It's a shame then. >> BABIKER: Well, no, no, no sorry. We're
not in a business of defamation, we're trying to just address our--we are trying to address
a problem. >> Right.
>> BABIKER: So, if I need to support all the applications to my end-user and this is the
key aspect of my role, I have to run v4 'til all the vendors in the world or all the vendor
I use at least are on v6 natively. Once that done, I can turn off v4. I will be happy to
do that because I think that's the way forward anyway. Does that answer your question?
>> Almost. I could go a little bit further but I think--are we running out of time?
>> Oh, yeah, we need to get to--get to Ron. >> Okay.
>> BABIKER: Thank you. >> But thank you very much.
>> So we were lucky enough last year to have Ron from DREN come and give us a talk about
the Navy's IPv6 deployment, and he's been great enough to travel at the last possible
opportunity to come back and do it again. Let's see here. So, Mr. Broersma.
>> BROERSMA: Okay. So, a little bit of introduction. We've had a project for quite a few years
now that aggressively deploy IPv6 in our research engineering networks inside the Department
of Defense, and we're doing a case study here of one of the major enterprise customers of
that Wide Area Network, SPAWAR, which is a navy IT organization supporting most of the
rest. The Navy exists at probably about a dozen locations around the world, dozen large
campuses, so that's kind of the context of what we're looking at. And this initiative,
what I'm talking about today is probably a little bit different than some of the other
ones or other presentations because it's real production stuff. It's not just a test bed.
It isn't just an ISP view of the world or just a campus view or a system or application
view; it's kind of all of the above. It's an end-to-end study. The systems and the users
that are the customers in this environment are pretty much all autonomous, a lot of separate
projects. It's not like it's one big enterprise environment all controlled by active directory;
it's a lot of little independent things. It's very heterogeneous, there's about every operating
system under the sun that you can imagine in a research environment, and it's not just
a few systems, it's everything on the network where we're trying to turn on IPv6. So, the
goals were push the envelope with IPv6 deployment. See what's possible, see what's missing, see
what's broken and try to get it fixed. Dual stack everywhere is our approach and, where
we can, do IPv6 only. So places where don't have to talk to the Internet, we're actually
doing IPv6 only or trying to. And we want to share lessons learned, but mostly, we want
to learn from this experience because we feel we need to figure out all the issues on our
research networks before we deploy this on the DoD operational networks because there,
if something is broken, lives are at stake. So that is one of our motivators. So here's
some of our progress and kind of in the order of how we attacked this over the years since
about 1999 and then really aggressively starting about 2003. Wide Area Network is dual stack
everywhere. All of our peering is dual stack wherever we can. We do unicast and multicast.
All of our LANs and wireless LANs are all fully v6 enabled. We turn it on every router
interface with an address. We ended up renumbering our IPv4 addresses. It took us about three
years to do it because it gives us some better congruence and makes it--some of the operational
issues a lot easier, reducing complexity because you add complexity when you turn on v6. So,
you know, we're looking at the O&M tail here. Infrastructure services, all are recursive
DNS, NTP, SNTP all the standard things, we enabled soon by dual stacking all of those,
then also the support services, the triple A services, authentication, directory services,
Kerberos, et cetera, all dual stack across the infrastructure. Then the public facing
services, how we--you know, everything that faces the outside world; the authoritative
DNS, mail exchangers, www, NTP, et cetera. Security stack was the really hard one. How
do we get this deployed into our secure enclaves? Firewall was a challenge because it took a
long time to find a really robust firewall that did dual stack. For a long time, we had
duplicate firewalls at a lot of sites. The intrusion detection system was hard. We had
an IDS that was written inside DOD so, of course, we had to do the upgrade to the code
to make it dual stack capable. We run a lot of Snort, so we did all the code upgrades
to Snort, gave it back to the owner of that code. IPS, the product we were using was TippingPoint.
It took us four years and about two or three generations of product hardware to get something
that truly did dual stack natively in hardware at line rate and so, we're pretty pleased
with how that's coming. So, all of those pieces were very hard. It was a lot of work over
a lot of years. Then the security services, just a couple examples. We used Windows WSUS
for updating all of our Windows system. We have that all running over v6 now, all of
our clients calling in over v6. Also inside DoD, there's a mandated application that every
DoD system has to run, it's called the host based security system and it's actually McAfee
ePO is the product. And we recently got that all working over IPv6 which pretty much amazed
the McAfee people because they said nobody else was actually trying this and we had it
working pretty well. And then, over the last year, we've had a major push for all the servers,
desktop, laptops, basically everything on the network trying to get it dual stacked.
Had a large XP environment and so it takes things, like, you have to go out manually
visiting every machine to try to get it turned on. So that's kind of some of the background.
People like utilization slides. Here's one of the sites and compares of v4 traffic to
the v6 traffic and we're at about 10% now. A year ago, I would say we were more like
1%. A year before that, we were at 0.1%. And normally you'd think that, you know, the amount
of traffic, you sort of got--the ratios are going to be equivalent to the product of how
much your client-base, the percent of your client-base that's v6 enabled versus how much
of the Internet is v6 enabled. So when we're close to 100% internally, it's surprising
that we would have 10% of our traffic being over v6 to the outside world because that
would sort of imply that 10% of the Internet is v6 reachable, which it's not. The reason
is YouTube. This--we were at about 2% until YouTube got turned on and then we went right
up to 10%, so that was kind of interesting. So here's some of our perspective after doing
a lot of this. It's really not that hard. A lot of people think [PAUSE] make it of the
organization. From the CIO, the network operators, the network engineering, the people that run
the web servers, the IT shop, the security group, everywhere has to permeate the organization.
Too many places, there's a few that get it and others don't and you can't make progress
unless you have it be a corporate culture. You don't want to wait until it's a crisis.
You just roll it out gradually as part of normal tech refresh. That was one of our major
lessons learned. You don't have to like ask for a ton of money to do forklift upgrades.
If you haven't started yet, you're already behind because if you do it as part of the
tech refresh, it's like a five-year cycle. But we're going to be running out addresses,
you know, v4 addresses, here in a couple of years. And so, it's going to be a crisis for
some of you and you're going have to do those expensive forklift upgrades if you haven't
been working on it. I say, don't be afraid to break some glass. You know, I talked about
all the worries about SLAs and those few people that can't get to it. Our attitude is if you're
willing to break some glass, things get fixed a lot faster and that's been our experience.
And we don't buy from vendors unless they support v6 and if you do talk to your vendors,
beg for feature parity between v4 and v6 features because it's just really missing out there.
And we also--when we talk to vendors, especially when they come and say, "Hey, we got this
great product for you," the first thing we do is say, "Let's go to your website. Let's
see if we can reach you with v6." If we can't, it's, "Call me back when you can." It is a
great filter, let me tell you. And it's a good incentive and, you know, they go off
and scramble and actually get it fixed. So we always like to do these surveys too about,
you know, how many of the vendors out there really are doing v6, especially the ones that
claim they are. So, a couple of weeks ago was the Rocky Mountain IPv6 Summit, so we
did the thing where we looked at all the sponsors. Now, you would think the sponsors of a summit
would be the IPv6 of Angeles and have v6 enabled all their public content internally and it's
still pretty bad out there. And so, we've seen this thing that the vendors aren't eating
their own dog food, they just want us check the v6 box and sell you stuff. Here, there
are a few notable exceptions for, again, electric. Yay, got all those greens across the board;
there's a few others. So we like to see that, but it's still too red and I think that's
one thing we need to fix. So at last year's meeting here, we talked about a number of
things, so I won't repeat that, but you can certainly go to back last year's slides and
see what we talked about. One thing we did was trying to get all our DNS zone transfers
using IPv6 transport. Also, we moved our entire VTC network to IPv6 only. Our management LAN
seemed like a good candidate to make IPv6-only and we ran into a lot of challenges there
and are still working on it. The RADIUS, Kerberos, LDAP, all that stuff getting it to work over
v6, pretty much worked out of the box except we ran into things like the LDAP client issue
where the--one of the very important modules is not v6 ready. Challenges with mainstream
VPN clients, a big one for us was automating the populating of DNS with V6 resource records.
You don't want to do it manually. A lot of the other schemes don't work because DHCPv6
isn't there yet, but we wrote a tool so that if you already are having--have V4 populated
in DNS with NPTR records, there's a way to automatically populate it using dynamic updates
to populate with all the v6 records and so, problem goes away. Creating incentives for
customers, we have a lot of customers that--there's no way to mandate it and so we have to use
incentives to get them to turn on IPv6 in their projects and on their desktops and laptops
and so forth. So in the last year, we've had this effort to really, really try to dual
stack all of the systems out there that we don't control. We saw in January of last year
about 5% of our systems had v6 enabled, were dual-stacked. Before that, I mean, it was
only half of that, so we've made some progress, but still, this is way too slow for us. So
we had a major internal campaign to try to convince all the system administrators, all
the projects, everybody who had a computer, please turn on v6. And it took some education
as well because for all the Windows XP machines, you have to tell them what command to run.
You have to have the helpdesk frames for all the places it doesn't work because it'll get
calls. And so, what we had to do was lots of nagging, lots of emails. We set up contests
between different groups to see which one--which group is ahead of the other group. In some
of our servers internally, we removed the A records and so you could only get to it
with IPv6. And then, the real good one was when we blocked IPv4 to Google because--I
knew this would get a reaction. >> So that's the v4 brokenness we see in on
our measurements, is it? People blocking us. >> BROERSMA: No.
>> Don't do that. Don't do that. >> BROERSMA: This is a controlled environment.
So--but this has the canary in the coal mine effect for us. When there's something broke,
you find out instantly, and so, that's been very, very helpful to point those out and
it's a great incentive. So we're--here's one of those pages we put up for all our users
showing by organizational code, how you can see how your group is comparing to your neighbor's
group, and so this is what we worked on getting our numbers up over the last year. This is
just one of the sites. So why can't we get those last 5% or why was it challenging for
some of those? There were things like all the VMWare machines had to be upgraded, all
the ESX machines, get them to 4.0 and get it turned on, and trying to convince all the
VMWare operators to actually do this. It's, like, my stuff's going to break. You know,
these are the large servers that are providing services in our datacenter and so, they were
very nervous about that and so it took a lot of convincing all the Windows 2000 users.
We got it still about 1% of our machines are Windows 2000. Luckily, we can outlaw them
next year because there's a DoD directive that says, "If it's no longer supported by
the vendor," and for us, July 13 is a great date because Windows 2000 is no longer supported
and so we can actually kick them all off our network and then have the policy that allows
us to do that. Printers, it's just kind of too hard right now and so--but what we're
doing with all our HP printers are buying all the new jet direct cards that support
v6 and trying to get as many up on v6 as we can. And then there are just lots of appliances
out there, everything under the sun; widgets, devices and so forth, a lot of little things
that just don't support v6 that we're just sort of punting on that for now and we'll
get to them later. Lot of other little challenges we ran into, we are wondering like, you know,
once we dual-stacked all the machines, why were the people not doing email over v6? Why
was all Outlook traffic not doing v6? Well, the older versions just don't do it. You have
to upgrade to Office 2007 or later. A lot of groups have systems that are under configuration
control. It's part of some DoD project and has to look just like the operational networks
and aren't allowed to turn on v6 and so, there's just nothing we can do about those. There's
a lot of system administrators that just say, "I'm too busy. I can't turn it on right now."
And we have to convince them, "No, no. You just run, install IPv6. It's real simple."
And they just don't want to mess with it and so, lots of convincing work there. We've had
problems like all the rogue 6to4 relays sending RAs. The problem is all the people that have
ICS turned on in Windows because the minute you do that, turns those boxes into a relay
and they start doing RAs and suddenly, all the--you know, they suck all the traffic.
The work around temporarily is just set the router priority on your primary routers, bump
it to high and so, hopefully, they're our preferred because the Windows machines will
set it to medium. The long-term fix is RA guard. We really need to get that implemented
in various products. I see lots of hand waving here.
>> Is that in our priority [INDISTINCT]? >> BROERSMA: Yeah.
>> It's what it's there for. >> BROERSMA: Right. Yeah. And part...
>> The thing that [INDISTINCT]. >> Yeah. And for us, one of the router vendors
we were using actually didn't have a way to set that and so, we had to wait for a feature
request to come through and actually implement that. There's various DNS brokenness out there
in the world that we have to get around. There's also--we found out that BlackBerry stuff,
okay, doesn't work in a v6 environment and so, that broke some of our users. Oracle lack
of v6 support has been a long-term problem, so a lot of our back-end stuff can't be v6
enabled. And then NetApp appliances for doing all our NFS and SMB stuff, CIF stuff, try
to make that over v6, but every time we turned it on, mounts failed. Although I heard last
night that they turned it on again and I think they have it working this time, so. There's
problems like, you know, you'll notice Java never uses v6 and it's because by default
they haven't, you know, there's preference set default, so you have to go around to all
your Apps and change those to true, which is not very scalable. There's issues with
picking the wrong tunnel metrics, so we see some of our internal machines going to the--going
through ISATAP instead of Native when they have it. It has to do with some of the Windows
implementations not having the right metrics. Privacy addresses, somebody just asked me
about this just a minute ago, it's horrible. We hate it. It breaks our environment. There's
no central way for us to turn it off. You know, we've been wishing for a, like, you
know, another bit in the router advertisements that says, "Don't do privacy." There's got
to be some way to centrally turn this off because it's just killing us. We have to visit
every Windows machine and manually turn it off and it's a huge pain. Snow Leopard has
been killing us because it doesn't do address selection write in the DNS responses because
of how they switch to mdnsresponder and it only listens to the first answer. So I got
some references at the bottom if you want to go read the bug report. This is not fixed
yet and it's a killer for us. And Java on Mac OS, the other fix I pointed out earlier
for getting around Java, it doesn't even work on Mac OS X. Printers, you find you can't
put an IPv6 address in a Mac saying, "Go this IPv6 printer," and you put in a bug report
and they say, "That's the way it's supposed to work. You can only put names in there."
You have to learn it via Bonjour or DNS, so it's expected behavior, which seems a little
bit weird, but that's the answer. We'll go to that one. So we're still working on this
network management project, but just--basically, we're trying to do everything with network
management on a separate management LAN mostly out of band to all of our infrastructure devices
and we can--we're almost there. Almost everything works. The thing we're still waiting for is
we have one application that relies on CDP or FDP to learn to pollage (ph) the information
and that the tables supported there are only IPv4 addresses, and so we're waiting for LLDP
that supports v6 and then we'll be able to actually turn of v4. But you can get an example.
Here's one of our tools, we used InMon for a lot of the sFlow tracking and so forth and
you can see under the IPV address column, most of them are ULA addresses. What's cool
about this is the addresses are shorter than v4 and so, there's less typing for people
who actually have to type the address, so one of the wins of v6. Also notice, there's
a few things that still have to use IPv4 addresses like the Google search appliance which yet--doesn't
yet support v6, so a little plug there. So freeRADIUS 2, a lot of people think that you
can't make RADIUS do--do as--both v4 and v6 because of the things that are on the notes,
and if you search the internet a lot of people say it just doesn't work, you have to run
two instances. It's actually not true. You can just create two listen clauses and it
works. And so, I point that out because a lot--this has tripped up a lot of folks. Part
of the management LAN, we have to manage all the UPSs out of the network. This is really
a--one of the--one of the appliances we had an issue with, but we found that the new APC
Network Management cards actually support IPv6, so we're replacing all the management
cards and getting rid of all non-APC UPSs. And so, you know, we really put our, you know,
money where our mouth is and get rid of products that don't support v6 and are investing in
things that do. New approach to training, we've learned over many years, where we're
doing like, you know, the multi-day classes, everything you want to know about IPv6, didn't
help at all. People would go home. By the time they get around to it, they forgot everything
or it seems way too complex. We totally changed it around and said, "V6 is easy, get there
in five easy steps." And also, we pre-configure all the interfaces on the WAN customer interfaces
and we lay out some best practices because a lot of people were asking about, "How should
I do this? How do I do an address plan?" And so we just say, "Read my lips. Forget about
being conservative like IPv4. All subnets are /64. End of discussion." Don't encode
v4 subnet values into the bottom 64-bits because we saw a lot of people doing that. We just
said, "Don't do it and don't do NAT," which is another huge argument we have in the DoD
because a lot of people think NAT as a security thing, so it's a big argument. So this has
really helped because now it looks simple because even in the five easy steps, it's
here's a copy of the email you need to send at the NAT that says, "Please turn it on and
give me my prefix," so, a lot of help. So anyway, quick summary, our biggest issue right
now is lack of feature parity. There is just too many places where an important feature
is just not--the v6 support is just not there and it's not--these are the things that aren't
covered by RFCs. This is the vendor's secret sauce and that's why we went to that vendor
in the first place and if those features aren't v6 enabled, then it's a problem for us. We
think the highest priority for all organizations is to IPv6 enable their public-facing services
for a lot of the reasons you've heard here already and so we're trying to push that in
DoD and even make it a national agenda item. Dual-stacking networks works well as a transition
mechanism in our opinion and we're trying to turn off all the other tunneling and 6to4
and Teredo and all that stuff, we just absolutely block it in out network. And--but there's
still work to do before we can turn off v4. We've tried, we're looking for ways to turn
off v4 in our environments but there's still too many pieces that need v4. So, that's it.
>> Wow. Thank you. Amazing. And we have time for questions, please.
>> WALLACE: Ralph Wallace with Command-Control. >> BROERSMA: Yeah.
>> WALLACE: Ron, we've been talking for years. >> BROERSMA: Yeah.
>> WALLACE: Interesting point though with the DREN. As you know, we're currently under
your contract with SPAWAR to build a command--Navy command and control system on top of v6 integrator
with a solo framework. >> BROERSMA: Uh-huh.
>> WALLACE: And also do all the security engineering involved with that to make sure that we are
also in compliance with the certification and accreditation process.
>> BROERSMA: Uh-huh. >> WALLACE: You and I know it as Die Cap.
>> BROERSMA: Right. >> WALLACE: You're running production networks
now and you had to get your authority to operate through that authority at SPAWAR Pacific in
order to do that. Going through the workflow, what was your main challenges in order to
get that because that's--we're doing the same thing in Charleston.
>> BROERSMA: Yeah. >> WALLACE: Okay? For the IPv6 integration
lab that we're building for the command and control system we're designing because we're
in charge of the security of it too. So what was your challenges on the ATO (ph)?
>> BROERSMA: Yeah. In all the doctrine, the policy, and everything about how you do accreditation,
secure the networks and so forth, we just reread all the language in terms of--I know
they talk--meant IPv4, but how would you do with IPv6 and made sure there was an equivalent
for every single thing, demonstrated that to the DAA and got the DAA to sign off on
it. >> WALLACE: Thank you. It's nice to know that
we're doing the same thing. >> BROERSMA: Okay. Yeah.
>> HANKINS: Dif Hankins (ph), IOC. I want to make a quick note, you mentioned that there's
no DHCPv6 client from Mac OS X, but actually, if you download ISUDH (ph) before, you can
install that on a Mac OS X box and use that as a client.
>> BROERSMA: Yeah. >> HANKINS: And that's a nice first step.
What we'd like to also do is produce a--like in the old days, the first days of the Internet,
you had an installed disk with your ISP that would go and install whatever P2P software
they were using, that sort of thing. We want to sort of do the same thing with Mac OS X
there, so you can just download that and go and run. So there's progress being made there.
>> BROERSMA: Good, good. It'd be nice have it come with the vendor--I mean, we don't
want to have to visit the machines. It should just be turn it on and go, so--but I understand.
>> I don't work for Apple, but I've heard the people that work for Apple speak at microphones
saying, "We will never run DHCP..." >> BROERSMA: Yeah, I've heard that.
>> "...Because of various things." So, yeah. >> BROERSMA: Uh-huh.
>> LEBOVITZ: Gregory Lebovitz, Juniper Networks. Clearly, you had what we heard is one of the
key things, management support. >> BROERSMA: Right.
>> LEBOVITZ: I mean it's obvious, right? When you look at how much stuff you did and how
quickly you did it. >> BROERSMA: Uh-huh.
>> LEBOVITZ: So, beyond just the governmental mandate, which is, you know, a lot of people
could ignore that if they wanted to especially in the budget times we've had recently in
the U.S. >> BROERSMA: Uh-huh.
>> LEBOVITZ: What was the trick that you guys used to get this sort of total hierarchical
push down about the v6 mandate? And second, how big was the dedicated staff that was working
on just v6 projects? So, I'm not talking about people who were on a virtual team from the
network engineering team who had to help a little bit with v6 project but people who
were full-time dedicated staff to making all this transitions stuff work.
>> BROERSMA: Okay. I think the one real key thing was in 2003, when the DoD CIO put out
the directive to try to get there by 2008, DREN was selected as the official DoD pilot
and so, that was one of the good incentives. I think for the people working on it, it was
just something new and interesting because they were, you know, all engineer types who
wanted to make it happen. Because we have the DoD, you know, selected as the pilot,
we had to prove results and so, we had the management buy in from the highest levels
and that made it easy to get the other buy in elsewhere in the organization. So, top
down is a lot easier than bottom up. So, did it answer all the questions, you think?
>> LEBOVITZ: Staff? >> BROERSMA: Oh, staff? No dedicated staff.
This was as people had time and interest. There was not a single dedicated person to
that. It just--it's a lot of little bit of time over a long period rather than a huge
amount of time by just a few people. >> So I have two questions actually. Number
one is what do you do for HTTP proxies? Because the ones we use and the ones everyone uses
is don't support v6. So, what do you run as an HTTP proxy?
>> BROERSMA: Don't--well, we don't proxy the v6 code--I mean v6 traffic outbound. I mean,
we're not a content provider--I mean we have a few websites but--so, most of it's our customers
going out to the internet. So, we do have a proxy but it's only for v4 traffic and that's
mostly to do local caching and to block access to certain sites. v6 traffic, we just let
go through. >> Right. So, that--so, basically it's the
host that makes the decision? >> BROERSMA: Yes.
>> Yeah. Okay. That's the problem. The second, your statement to you--when you go to vendors,
you say we want feature parity. >> BROERSMA: Yes.
>> That's what I say as well and it's not working for me.
>> BROERSMA: Right. >> So, I go to vendors and I say, "Feature
parity, what do you want?" and I say, "Well, you know, I want, you know, every feature
you have for v4, it has to work for v6 in the same way." And they say, "No, but we can't
do that, it's like boiling the ocean. We need to prioritize." So, how do you fix that?
>> BROERSMA: Yeah. Yeah. Two things, we have the conversations with the vendors and they
say, "Well, how do we do that?" and I say, "Go--have someone go through your entire documentation
and everything in your documentation, make sure it says, 'Here's how you do this in v4
and here's how you do it in v6.'" And a number of them are going through that exercise and
coming up with the spread sheet and looking for the feature parity. The other thing is
we just have entered in a--into a cooperative research and development agreement with one
of the defense contractors where I said, "I want to figure out how to solve this problem
of feature parity." And so, we're going to sort of, you know, turn it into a real research
project, talk to vendors, see how we could approach that better because it is a big problem.
Because we've noticed that, I mean, we have in DoD there's a number of efforts to come
up with like, you know, what's the profile of all the RFCs that have to be complied with?
What are all the mandates? What's all the testing? There's organizations like G-DEC
at Fort Huachuca and other, you know, logo programs and other things. Those are necessary
but they're not sufficient; that's what we found. And so, what's--how did we get the
extra piece? And that's working with vendors and it's really looking at how best to do
that. >> KINNEY: Okay, Ferry Kinney (ph), Ericsson.
About feature parity, we get a lot of questions about that. So, the answer from our side is
which ones you want first? So... >> BROERSMA: Right.
>> KINNEY: So that's the kind of list because of course, feature parity means feature parity,
that means every feature should... >> BROERSMA: Right.
>> KINNEY: There should be parity between all of them, so...
>> BROERSMA: Right. >> KINNEY: But like, there's some implementation
issue here. >> BROERSMA: Uh-huh, absolutely.
>> KINNEY: Which require time and effort, so which ones do you want first? That's my
statement here. And then about being able to--your public services, Ericsson is hosted
on Akamai, so maybe we should visit our customer as a customer--them and say, "Hey, when are
you going to do v6?" >> BROERSMA: Exactly.
>> KINNEY: And a final question. I realize I'm talking to a Defense organization here,
but do you have any numbers on OpEx? Operation expense which you--are you going to save any
money on this and--or is it--are you going to save money when you turn off v4 or are
you already saving money on dual stack? >> BROERSMA: Okay, let me try to answer in
order. So in reverse order, OpEx, we don't have hard numbers on that. It is not double
the amount of work to support dual stack, but it is, you know, between, you know, one
and 2 X certainly because the additional complexity, but I don't have hard numbers on that yet.
The other question was the feature parity thing. Absolutely, prioritization is a key
and usually, when the vendors come up with that matrix and they say, "Here's all the
things that don't do v6 yet." That's the first thing they ask us, what's the priority? And
we'll prioritize because it's based on operational need. And then, what was your second question?
>> KINNEY: That was--I was--maybe if you could refer to are you--do you think you will spend
some OpEx when you turn off IPv4? Because you said--you kind of hinted that it was like
one time something more expensive to do dual stack, not two times...
>> BROERSMA: Right. >> KINNEY: So it's like is the v4 part the
fraction there or is it the one? So do you kind of expect to get lower OpEx?
>> BROERSMA: It's why we went through all the effort to do things like congruency in
the network and changing our v4 addresses so that the OpEx would be lower.
>> KINNEY: So it's not to get to sleep through the weekends or...?
>> BROERSMA: Not to get what? >> KINNEY: To sleep to the weekends, not being
able to support the customer there. No, its okay, I understand. Okay, thanks.
>> BROERSMA: Okay, good. >> Thank you very much, Ron. Fantastic.