Daniel Suarez: Kill Decision, Authors at Google

>>Male Presenter: Welcome, everyone. Welcome. I really appreciate everybody coming out to
see Daniel today. I've had the pleasure of spending some time with him today and it's
been a lot of fun. When I first read "Daemon," a friend of mine who recommended it and said,
"This is a book for you and you have to read this book."
And I'm a voracious reader and I'm a little picky. And when I read it, I just, I was blown
away at the writing and of how this technology can affect people's lives and change them.
And with the most recent book of "Kill Decision," we've had the gracious opportunity for Daniel
to be here today. So, with that, I present to you Daniel Suarez and "Kill Decision."
[applause]
>>Dan: It's very nice of you. I'm happy to see this is my second visit here and there's
more people here this time. So, that means it's getting better.
[laughter]
But I think a lot of people, well, some people, who read my books think that I have a very
dark vision of technology. I really don't. I really love technology. And because I write
thrillers, I have to go to the bad neighborhood of technology to make an exciting story.
I was talking about this earlier that if I wrote a thriller that started out in a really
happy place and it got happier, that would be like the worst thriller ever. So, I'm typically
taking ideas that I think are cool or that I think is really cool, and, of course, realistic,
and trying to do some devious things with them.
But I don't want people to think that that's a reflection of how I think technology must
go. As a matter of fact, I think it will be technology that, in many ways, helps us solve
the many serious problems we have. Oh, I have a microphone now, so I can probably tone it
down. But basically, I love tech. And I worked for almost 20 years designing big data systems.
And that's really where I come from in writing my stories, is a great love for technology.
That's why I take great care to try to make it accurate. And for those who've read my
stories, they know I do a lot of research and I try to get things right, even though
there might be some errata occasionally. But from a narrative point of view, you need conflict.
And that's why I introduce it.
[pause]
Now, with this book, "Kill Decision," what I was interested in was the question of what's
called "lethal autonomy." This is a military term for basically algorithms making a decision
about whether to kill. And again, this is science fiction in a way, but it's very much
in the discussion right now with the people who are building and deploying these systems.
I mean, we have predator drones and reaper drones. Human beings are still making the
decision there. But there is no technological impediment to algorithms making these decisions
right now. And in fact, right now in the world there are two systems that are deployed, that
are capable of making that decision.
One is a sniper station in the DMZ between North and South Korea. And another one is
a system on the border of the Gaza Strip. Now, both of these systems have that feature
turned off, but they have that capability. And that surprises some people to say, "Wow,
really? That already happened? That's already possible?"
And it very much is. And that's why I thought it was such an important thing to write about.
And again, with my books, I really try to take complex technological issues that affect
people, put them in a thrilling story so that more people can understand what the issues
are 'cause they're very, very rarely black and white.
They're usually quite nuanced. Usually within a new technology there's something that is
tremendously useful and compelling. I do think drones are that--autonomous drones. I think
there's so many terrific uses for civilian autonomous drones, whether it's search and
rescue or environmental monitoring, or delivering medical supplies into areas beset by banditry
on roads, or you name it.
There's a whole bunch of things. And plus, they're fun. I really like messing around
with drones myself. So, I like this technology. And then, of course, then I go off and write
a book about killer robots. So, but you have to make it thrilling.
So now, in this one, I was really concerned with that fundamental question of whether
it is not just ethically, but from a very practical point of view, a good idea to make
machines--even though we can do it--to make machines that would kill people, even if they're
enemies of ours. And in particular, and that's why I've called the book "Kill Decision" of
course, 'cause that's really the fundamental thing we're talking about here.
And what I was concerned about was the corrosive effect that might have, as expedient as it
might be, on representative government. Because I'm concerned about how it might focus and
centralize control of a very crucial thing in very few unseen hands. Because right now,
if you wanna have conflict with somebody, have a war with somebody, you need the buy-in
of other human beings.
And, of course, other human beings, especially free human beings, have opinions and they
do or don't act depending on whether they agree with you. And I don't think you have
that same thing when it comes to autonomous machines. In particular, in this story, I
don't have Terminator-like creatures running around.
They're not aware of what's going on. It's one of the reasons why I chose an insect-level
intelligence. I wanted to use something that was very technologically possible with the
technology we have now. And I wanted to show it is messy and inaccurate, but in the main,
effective if you don't worry too much about making a mess.
So, again, that's a very different type of deployment of military force than what we're
facing now. And it's so imminent that I really wanted to weave it into a story that people
would read right now, before we cross that Rubicon, basically, and make that choice.
'Cause I think it's gonna be very difficult to come back.
So, again, even though I like technology very much, there's certain aspects that I think
we have to be mindful. I like to think of it as trying to avoid ice bergs, even though
we're going to be climbing this ocean. We need to be mindful of the ice bergs. And that's
really what I try to do. I try to make it fun and interesting and exciting and all that.
One thing I was thinking about, though, Asimov's Three Laws of Robotics, I think it was 1942.
Man, we threw that out fast. That was like, we're just getting to the point when we're
building robots and it wasn't like ten minutes later, somebody said, "And let's give it a
gun." It's like, and that's with 70 years warning.
So, they go, "Wow. Man, we really messed that up." So, it's not quite out of the bag completely
yet. But again, I'm just trying to bring people into thinking about it in different ways.
[pause]
Now, one of the things I did in this book was look at intelligence. Probably people
in this room deal with this professionally, question of AI. I started looking at this,
again, to try to make it real in this book, "Kill Decision."
And I immediately started looking at social insects and how they exchange information.
And part of that made me realize how many different types of intelligence there are
in the world today, that human beings have a very different type of intelligence from
ants, but ants routinely solve very complex problems, like the traveling salesman problem.
And finding out how that happens, how they used what's called "stigmergic propagation"
to change their environment and basically record onto the real world by laying down
pheromones, record what an individual knows so that the other individual agents in that
system can find it and react to it, like the neurons in the brain reacting.
And it just made me more aware of how humanity really is just part of this fabric. When I
then did research on ravens, because I have use for ravens in this story, that's another
type of intelligence that is very impressive. And again, in the case of ravens, there's
a lot of evidence that ravens have been interacting with us as an intelligent species, not just
separate of us, that we're a special species with them.
And again, all of this informed what I was writing about when it came to machines because
what you realize is that intelligence, very capable intelligence, you really don't need
a general AI to accomplish that and to create agents that go out and interact with us and
make changes in the world. Now actually, I guess I didn't prepare a very long talk.
So, I was thinking we could open it up for questions. And I know some questions have
come along. I'm happy to answer questions both about this book and previous ones for
anybody who has them.
[pause]
>>Male Presenter: So, what we have, we have the Dory page if you wanna go to go slash
Daniel Suarez. We have lots of questions there. I'm sure we're gonna have a lot of questions
here. But one of the questions that I do wanna ask is Daniel had the opportunity to demo
Glass and Chauffeur today.
And so, most of you have read his first book. He predicts, basically, Glass and Chauffeur.
And based on your observations--
[laughter]
predicts.
>>Daniel: Yeah, I'm sure some people would argue--.
>>Male Presenter: I use that word, not "predicts" because of the book, but what are your thoughts
on that?
>>Daniel: I thought they were cool as hell. And nobody got hurt today, which was an added
bonus.
[laughter]
For those who haven't read the books, of course my automated vehicles harm people. And the
HUD Glass is there to basically give you the dirt on who you're looking at and what might
be their weakness and so forth.
And I'm sure that app is coming, but today was very--. That's why I thought it was funny
when they asked me, "Hey, what apps do you envision?" And I started going through in
my mental Rolodex, thinking, "Oh, can't do that one. No. Not. I'm a troubled person."
That's when I said, "You don't want to make anything I'm thinking of."
[laughter]
But no. I thought it was great as a form factor. What I loved most about it was how unobtrusive
it was. You're just doing whatever and then you encounter something you don't know, what
I typically do is look up and go, "Hmm." And there's the screen right there where you look
up. So, I thought that was pretty cool.
>>Male Presenter: And Chauffeur?
>>Daniel: Chauffeur, of course, Rick and I, we were hoping to get them to change lanes
and do donuts and all that other stuff that automated cars must do. They must. But apparently
it's capable of doing those things, but they wouldn't do it. We still enjoyed it though.
It was a better driver than I am.
>>Male Presenter: Excellent. We have a live question.
>>Male #1: Hi. My name is John Wiley.
>>Daniel: Hi, John.
>>Male #1: And I design Search. And you write books.
>>Daniel: I do.
>>Male #1: And I'm wondering if you could talk a little bit about the feedback loop
here that happens because you are creating stories about technology that are near term.
>>Daniel: Very near.
>>Male #1: And I'm reading your stories and I'm trying to make some of that technology
come true. Not the bad parts of it, but that's the thing. Technology doesn't judge, right?
It can be used for good and bad.
>>Daniel: Right.
>>Male #1: And so there's this continual back and forth, I think, between the lots of us
who read technological thrillers or science fiction. And we set out to create the future.
>>Daniel: Yeah.
>>Male #1: And then that influences the stories that y'all create 'cause it goes back and
forth.
>>Daniel: Exactly.
>>Male #1: So, I'm wondering--.
>>Daniel: Who pushes the frontier?
>>Male #1: So, I'd like to hear about it from your side of it. I'm out there trying to create
some of these things and so is everyone here. And you could tell us a little bit about your
perspective from the alpha ship.
>>Daniel: Absolutely. I would say that I have the easier job of the two in this relationship
because a lot of what I'll do, and again, I was having this conversation earlier about
how I arrived at using certain technologies at certain parts of my stories. And generally,
when I start out writing a story, I'll think there is at its heart some technology or trend
that I'm interested in.
And then, I immediately create these mental slots of must-haves to make that happen narratively,
whether it's some specific technology that can project sound into the middle of the air.
And then I proceed to do this voracious search to find what's out there. And then, I'll find
something that's either close or dead on and then I'll try to assemble existing technologies
in ways and make that happen.
So, that's how I use what you guys do to try to push it just a little further. 'Cause typically,
what I'm doing is taking stuff that is very real and again, trying to combine it in new
ways that maybe people hadn't thought of and just push it back a little further. And then,
of course, then you guys go and invent new stuff hopefully, if it's compelling.
And again, it's not just me. I'm just one ant doing this and there's lots of other writers.
You read it and it gives you ideas. And then it continues that cycle. And so, it's just
constantly edging further back. But I think in many ways, we do both have the same job
in that technology must tell a story to people, to have a narrative where they can see how
it relates to them and what compels them to use it.
So in that sense, I think we have a very similar job. You just have the harder one where you
actually have to do it. I think my, the hardest part of my job is restraining myself so I
just don't go too far. So mostly when I write, the first draft will go too far and I'll always
reign it back 'cause I want it to be as near term as possible so people can relate to it.
And I must admit, people like you can relate to it because I'm a technologist and it's
important to me to get stuff right. So, where I'm exceeding the limits a bit and pushing
it back, I want us to know that. I want you to know that I know what the foundation is.
I know where we are and I'm pushing it back a little.
And I've gotten some really useful feedback from engineers and people, usually in a great
way. And it helps my work a lot. That's why I've started giving galleys early on to engineers
who I know will really tell me what they think. So helpful for me. So, I think it's definitely
a positive feedback loop. So I hope that answers your question. Good.
>>Male Presenter: All right. So we're gonna go through a couple Dory questions. If you
do have any questions here in the building, of course, just go up to the mic and we'll
let you speak. So, the first Dory question is, "I'd like to hear your experience about
getting published, how epublishing has affected your relationship with agents, editors and
publishers."
>>Daniel: Yeah, that's a great question. In some ways, I'm a poster child for self-publishing
because I did the thing that everybody told me wasn't possible. And it's not because I
thought, "Oh, I'm gonna do it." I just wanted to do it. I wanted to write a book. Now, I
wrote "Daemon" from 2002 to 2004.
And it took a little while to get it out there 'cause I went right down Main Street. I tried
to get an agent for about a year. And I read all sorts of books on it. It's like I'd write
customized career letters and try to identify the literary agent who covers the area that
you write in. I tried that for about a year and the response that I got from a couple
of agents was my book was too technical.
People aren't interested that much in tech. And I totally did not agree with that. As
a matter of fact, I think technology and some literacy with technology is a new form of
literacy today. I mean, most young people, they grok technology. They get it. They use
it and I just didn't agree with that. And, of course, publishing is hidebound industry.
I'm told that all the time by people in publishing in New York 'cause "well, we're kind of a
traditional industry." And everybody knows it. Everybody's got technology, but how they
integrate it into these hundred year old systems they have, that's the indigestion I think
they're having. But then, when I went to self-publish, I took a look at the various vanity presses
'cause there was a bit of a stigma at the time to the extent that there still is a bit.
I think a lot of that's gone away. But if you self-published, that's like printing a
book. It's like you printed something. I then started taking a look at it because I was
a logistics guy, right? I designed logistics systems and I started thinking, "Well, vanity
press gotta get rid of all these middle men." And I'm a technological guy.
I'm gonna type set this book myself. I'm gonna make sure that all the kerning is right. I
mean, you know, I'm an anal retentive guy. Design the cover in Photoshop. Found Lightning
Source, which was basically the back end, well, basically this is a company that's owned
by, I'm trying to remember the name of it.
But they're a distributor that all the main publishers use. Ingram. Thank you very much.
Ingram is used by all the mainstream publishers to do their back list titles that don't sell
a lot. And I thought, "Hey, I could do that." You have to be a company to use them, but
I have a company. So basically, I just made this happen.
I willed it into existence. And I got it out there. And I got it to people like Rick and
other people whose work I was interested in. And it just took off. We started to get an
audience--mostly technologists--people like you who really help me. Thank God for that.
I thank you all. But interestingly enough, here we are a few years later.
And I'm like a dinosaur now. That's like, "Wow. You really printed on paper? What were
you thinking?" And then you see it's like 60-page novellas. It's like people are churning
them out. It's like it just grew like crazy. So now, I'm like the old school. I did it
the old school way. And I was thinking at the time, when I finally--.
And by the way, how I finally got an agent was Wired Magazine printed an article about
my book. That blew up and suddenly every agent who wouldn't talk to me was calling me saying,
"Oh, you're awesome. You're great." And I finally found--. Actually, what I kept asking
every agent who called me, 'cause I was making money on every book sold.
And of course we were selling it online. I didn't have to do anything. Fulfillment was
taken care of. Very scalable. And I'd say, "Well, what can you do for me now? I don't
need an agent now. I'm selling books." And the one agent called me, who eventually became
my agent at the time, Sagalyn, they said, "Yeah, we can get you into foreign countries
and we can get translations and we can get audiobooks done and have you reach these other
audiences all around the world."
And I thought, "Well, that's a pretty damn good answer." But not many people were saying
that. So that's why eventually took the deal with Dutton and Penguin. And my experience
with them, as traditional as they are, is they were terrific. They were really great
to work with. They had some good suggestions.
And for those who took a look at the self-published version versus the paperback release, was
only one major difference. And that was the first chapter. And I had a different first
chapter, a more technical one, in the self-published version. [laughter] And the only suggestion
they had 'cause I was like adamant, "I'm not gonna change a damn thing."
That's it. And they said, "Well, here's the thing. If you put a different chapter in as
the first one that was, consider it taking the barbed wire fence from around your book
so that people who are not technical can get into the story. You can be as technical as
you want later." I actually thought that was a pretty good suggestion.
So, that's why I took that suggestion. And as it turns out, it worked out well. I kinda
like it better now actually. But, so it wasn't bad. It really wasn't. But again, it continues
to change. So, what would I do now if I was trying to break into publishing? I would absolutely
self-publish. I would e-publish. I'd use social media to try to build my audience from around
the world, where ever they are.
It's a little different since 2004. But part of that is that now there's a lot more noise
in some ways. So, when I was doing this, if I self-published, sure there was some stigma,
but if it was any good, people typically found the time to read it. Now there's a flood of
material coming out. So in many ways, it's like all these websites coming out when the
web took off.
It's like you can get out there. You can get your blog to try and get eyeballs. So, I don't
know. I think it's a mix. It's just different. I don't think it's any easier or any harder.
It's just constantly changing. So, anyway.
>>Male Presenter: Awesome. All right. Another live question.
>>Male #2: So, you talked a little bit about how you try to make your novels as realistic
as possible. Can you talk about like where you do the other thing where you deliberately
know that something's deliberately a technical problem, but you decide to deliberately ignore
it just to make the story better?
>>Daniel: Yeah. I love this question actually, 'cause--. I was telling my wife one time that
typically, if I wanna put a character in a bad situation, I'll put them in an impossible
situation, like something people couldn't possible survive from. And then, I'll sit
and noodle and try to think how to get them out of it.
And if it's impossible, I'll take away one of the variables and I'll keep trying to back
end into it. Sometimes, you get to a point in the story where you do, for the sake of
the story, have to make some concession. I try to make that as rare as possible. I'm
trying to think of an example where I did that on purpose.
'Cause, of course, there's fictitious elements to--. I think the perfect example is the Razorbacks
in "Daemon." They were talking about a Kawasaki Ninja motorcycle with twin Katanas running
around killing people. I had written one chapter in the sequel and it was, took place at like
two in the morning at some Missouri Texaco station.
Somebody's standing there and then these motorcycles come in and start using credit card fobs to
refuel themselves. And I thought, "That's completely possible." But it seemed a little
unlikely. So, I took it out.
[laughter]
And my editor thought, "Dude, come on." Plus, it didn't really serve the story, but I thought
visually it was cool. But not terribly possible. I mean, first of all, a bunch of robots having
credit cards, gas credit cards, I thought that was funny.
[laughter]
It didn't really serve the story. I hope that answered your question.
>>Male Presenter: Roy Merritt as well during the siege of the compound. The first book,
you know? I mean, him going to room to room to room to room.
>>Daniel: Yep.
>>Male Presenter: That's another example of just--.
>>Daniel: Well, that's a perfect example of putting somebody in an impossible situation.
>>Male Presenter: Yeah.
>>Daniel: And yeah. And certain iterations, he did not go, survive as far. That's where
my gaming skills come in handy 'cause I treat him like a gaming character. And he died many
times in that hallway.
[laughter]
But that ain't gonna work. I had to change it.
>>Male Presenter: We have another Dory question. Any news on the Daemon movie?
>>Daniel: Ah, the Daemon movie. They've done a number of scripts, Paramount has. I haven't
seen anything recently. So, is that really good? Is that really bad? I don't know. Hollywood's
like a black box to me.
And it's like I hear stuff happening. I don't know what it is. So, but I think this is very
common. I've had this conversation with other novelists and I think there's a couple of
novelists who have pull in Hollywood. JK Rowling, Dan Brown. I think I've just done the list
right there.
[laughter]
So, it's like--. I love it. You have meetings with people in Hollywood. It's like, "Oh,
you're really popular. Yeah, that's great. We'll call you when we need you." You try
to think, "Hey, what are you doing with the technology?" But they're spending a lot of
money on what they're doing.
>>Male Presenter: Yeah.
>>Daniel: Just gotta be great.
>>Male Presenter: So, the next question is, "You mention you're an avid gamer."
>>Daniel: Yeah.
>>Male Presenter: And I don't know if you wanna keep your anonymity online of what you
play or characters that you play online, but the question did come up. "What games do you
play?"
>>Daniel: Way too much Skyrim lately. I have an 81st level. Of course, I maxed out on that
thing. But interestingly enough, that is a game that is not a massively multi-player
online game. And that's only because if I do something like Call of Duty, some munchkin
just whacks me instantly. I swear to God.
I can't survive more than 20 seconds. I feel like I'm this old guy who has old reflexes.
I don't know whether they're aim bots or what, but I, and the constant stream of abuse doesn't
exactly help.
[laughter]
It's like I'm on the bus and I'm 12 years old again. Yeah, you. Jesus, you know? You're
killing me. Is that not enough for you? But so, Skyrim was cool for me because of Bethesda.
Here, they created this really rich world and I just found myself wandering around occasionally
and instead of killing things, I would like look around at the view. I mean, that's a
hell of a job. And it's like I almost, like, "Oh, do I have a camera? Could I take a picture?"
I mean, it's an awesome game. I really love it.
>>Male #3: Any MMO? [inaudible]
>>Daniel: Oh, OK. That's good to know. That's good to know. See, here's the thing. I never
felt compelled to do that before. So that's why I don't know that, that F12. That's good
to know. Although, I'm playing it on the Xbox.
>>Male Presenter: Is there MMOs you play?
>>Daniel: Not lately, man. Not lately. I'm trying to think. I've dabbled in some--[indistinct
] Online, stuff like that. But part of it is that I really wanna play a game that is
not so open-ended now. And it's because I have deadlines. And I can get really into
games.
>>Male Presenter: Yeah.
>>Daniel: So, if I play a game that I know is gonna be 80 hours, generally enough--.
But a lot of games, I like Sandbox Games because you can go in and out of them without feeling
compelled. And, of course, there's this thing now in Massively Multi-player games where
I--just guessing--nobody get mad at me.
I'll bet there are psychologists working there thinking, "How can we get that Pavlovian reflex
going where you need that sword?" And this whole idea that you're gonna get this downloadable
content, this DLC content, that you can a blue sword for nine dollars. It's like, "Oh,
my God. Really?" It's a virtual object.
You're not gonna go questing before you go buy it. That whole dynamic freaks me out.
I don't think it's, I don't think it's gonna improve story or experience. And again, you
look at something like Skyrim--very compelling, self-contained, really well envisioned. And
with the Radient Quest system, there's not one story.
They went for the swarming theory of story. So, if you're going on a quest, you don't
like, just screw that. I'm gonna find one I like and apparently they can keep generating
them. I thought it was a really compelling model.
>>Male Presenter: Excellent.
>>Daniel: So, that's what I do with games. Yes. Hey.
>>Female #1: Hey, Dan. So, have you read Neil Stephenson's "Reamde?" And what did you think
about it?
>>Daniel: I haven't yet. I haven't. It's sitting in my stack. I've read most of his books.
The whole Baroque trilogy, "Diamond Age," and of course the "Cryptonomicon," "Snow Crash."
>>Female #1: He went along with what you're just talking about, about the [inaudible ].
>>Daniel: Oh, really? So, he and I share the same opinion? That's cool. No? [laughs] No,
I'm looking forward to reading it. And I don't know how he does that. He punches out like
a thousand page book like every year and a half. It's like, yeah.
>>Male #4: Long.
>>Daniel: That's pretty amazing. Yeah, that's pretty amazing. But I will read it.
>>Male Presenter: All right. So, this is a Google Glass/Chauffeur question. You predicted,
or you've talked, you had Google, or the idea of the Glass and Chauffeur.
>>Daniel: I had an idea.
>>Male Presenter: Yeah.
>>Daniel: Yeah.
>>Male Presenter: What other futurist, technological predictions do you think about?
>>Daniel: Well, obviously the one that this book is based on--
>>Male Presenter: Sure.
>>Daniel: concerns me a lot. Let's think. Aside from these obvious ones that crawl around
the world. No, I can't talk about that because I'm working on that now.
>>Male Presenter: Oh, the next book?
>>Daniel: Yeah. I'm working on a forth book now.
>>Male Presenter: Is it "Daemon?"
>>Daniel: No, no. It's not.
>>Male Presenter: It's totally different.
>>Daniel: It's totally different.
>>Male Presenter: OK.
>>Daniel: Yeah. I'm trying to think of something that I can actually talk about without ruining
the next book here. Let me, can I ponder that one?
>>Male Presenter: Yeah. Absolutely. We can come back to it. That's not a problem.
>>Daniel: All right. What's that?
>>Female #2: [indistinct ].
>>Daniel: Oh, oh. I'm sorry.
>>Male Presenter: I'm sorry.
>>Daniel: It's very interesting.
>>Male Presenter: Go ahead. I'm sorry.
>>Male #5: Hi. There's an organization called Code Pink, which I think you might have heard.
Code Pink.
>>Daniel: I'm sorry.
>>Male #5: It's an organization called Code Pink.
>>Daniel: Oh, Code Pink. Yes, absolutely.
>>Male #5: So, one of the things they do is, for those of you who don't know, is they got
on and tell people how the negative impacts of automation, like drones systems that are
coming out. So, in your opinion, what do you think that, what are they doing right and
what are they doing wrong in terms of communication of these impacts of technology? Is there any
advice?
>>Daniel: Code Pink specifically, or the people who are building?
>>Male #5: Similar organizations.
>>Daniel: OK. I would say being concerned about it, they're doing right. I'm not saying
they're Luddites, but having a Luddite reaction against this I think is really just not gonna
help. Drones are gonna happen.
Automation's gonna happen. There's a confluence of things here. There's the processing power,
memory, just the material. The price point has gone down so much and given it so much
capability, it will be used. It's just ingesting that as a society and especially developing
a framework of law around it what I think we need to do.
So, for instance, you could picket murder, but murder's illegal. We already agree to
that. We set up institutions to help contain and control that. I think that's what we need
when it comes to automation. We need to basically create an extension, an appendix, of the legal
system that we have for humans for automation.
And that would, I think, help everyone, including manufacturers, because if you're a manufacturer
in automation, sorry some feedback. If you're manufacturing automated systems, you're gonna
want to know what the legal framework is so that you can release your product and not
have to recall it or be liable.
So basically, laying out that legal framework is, I think, the very next thing we have to
do. I value the fact that they might protest and they might care in that it increases the
conversation. But I think that's what we need to have and then compel our leaders to act
about it. So, that's what I would say to that question.
>>Male #6: Hi, Joseph Smarr.
>>Daniel: Hi, Joseph.
>>Male #6: I'm sure that, besides geeks like us, you're books have made a few lightbulbs
and warning bells go off in other people's brains. I'm just curious if you have any amusing
anecdotes to share about being called up by government or military people or companies
of other sorts--just places where they've wanted to learn more, talk to you, and any
strange rabbit holes you found yourself tumbling down as a result of being so intellectually
provocative?
>>Daniel: Yes. Actually, I have. I love this question. The problem is trying to answer
it 'cause--.
[laughter]
Well, no. I wouldn't have to kill you. I'm not saying. I'm not saying. No, it's just
one of these things where there was a point in time where I was giving a speech at a certain
place, thinking, "What the hell? How the hell did I wind up here?"
[laughter]
I mean, seriously. It was one of those things where if two years earlier you had said, "Guess
what you're gonna be doing?" You're like--.
[laughter]
And the other thing I thought was, "If I'm the expert on this, we are all screwed."
[laughter]
What is it really? 'Cause when people are showing up to your talk in armored columns,
it's like wow. Come on. That's just crazy. That's nuts. But that happens 'cause they
have to move around town, too, is what I found out.
So, I was heartened by the fact, 'cause I go to DEFCON regularly, and I notice people
like Jim Christy a few years back, instead of arresting hackers constantly started thinking,
"Hey, let's hire them." And what I've noticed from the people I've talked to in the defense
establishment is quite a few of them are actually pretty cool and open-minded.
And also smart. Which made me feel better. Like, they, the automatic response is, "Let's
not build something to destroy something." They said they wanna understand and they're
open-minded. And so, that much I can tell you, that the conversations I've had with
people who've approached me of that ilk have been pretty positive by and large. So, I hope
that answers your question.
>>Male Presenter: So, I'm gonna open it up to the VC. Is there any questions on VC right
now?
[pause]
>>Daniel: Doesn't sound like it.
>>Male Presenter: No, it sounds like not. So, we'll go back to, we'll go back to a Dory
question. And this is not, so they're not all softball questions.
>>Daniel: All right.
>>Male Presenter: Some of the criticisms of your book, that your book has received in
the past and also this book, is due to this technical parts of the book. The Washington
Post recently had a review specifically calling this out at "Kill Decision." How do you respond?
>>Daniel: The Washington Post said what about "Kill Decision?"
>>Male Presenter: It was technical and how it was a little too technical and that it
would've been nice if some of the technical explanations would've been in there.
>>Daniel: Yeah, yeah. I know. I guess part of it is that being a technical person, I've
tried to write it at a level that I think people, most people, can get to it. And the
only thing I can say in defense of myself in that regard is that I, very often, get
email from a 50-year old housewife or a grandmother or somebody who's reading it because they're
son or grandson is reading it.
And they were concerned about what they're reading. And then they get sucked into it
and they say, "I had no idea. This is really cool. And now I understand better what my
son is doing or whatever." So to the extent that I might challenge some readers who might
not know about that, I don't know. I think that's good.
And I definitely, definitely do not wanna dumb things down for people. I make is accessible,
but I just don't see the point of dumbing it down.
[applause]
Just saying, "And then cool stuff happens."
[laughter]
It doesn't work for me.
>>Male Presenter: I think one of the things you've said previously is that you have to,
the people are still using SQL injection, then there's something wrong.
>>Daniel: Yeah. Yes, exactly. You're still susceptible to those attacks. By the way,
I did have a friend tell me one time, this is early on, he said, "You're helping the
terrorists." Really? Pretty dumb terrorists. These are pretty old attacks. And I think
you have that vulnerability.
>>Male Presenter: All right. So another Dory question and then we'll get to a live question.
Actually, let's do the live question first.
>>Daniel: Yeah, let's do the live. He takes precedence.
>>Male #7: So, my question, so you spent a lot of time writing about drones and stuff.
>>Daniel: Yes.
>>Male #7: Did you go out and buy your own RC aircraft at all?
>>Daniel: Well, I started doing that. I joined DIY Drones and what I realized, looking at
my deadlines, especially with what people were doing, I thought, "This is like one of
those things. I'm gonna spend three months messing around and it's gonna be fun, but
I'm gonna be at the entry level at that point.
I'm gonna know how to program it to do way points and all that stuff." I am now gonna
get more involved in it because now I have more time. But what I quickly started doing
was reaching out to people who were doing really, really interesting things. And you
probably saw this, but there were some guys who were at BlackHat and they all sat at DEFCON.
They built this flying hacking platform that was a GSM tower that could hijack phone calls.
I'm, it was basically a gunnery drone, a marine gunnery drone from the '70s or something.
And they bought it cheap. I think they built the whole thing for two thousand bucks. They
had a 22-pound pay load.
Maybe it was a 16-pound pay load. They found the smallest computers they could. They basically
put a laptop up in the air and they were sucking up phone calls. And even if you encrypt, and
this is civilian encryption, even in you have civilian encryption on your phone, the encryption
is supposedly happening at the cell tower.
So, it would go to them in the clear. And they were demonstrating all this and half
the audience were Feds. [laughter] But that's the only thing. I don't know how many of you
have been to these. I love to spot the Fed contest. That's hilarious. You get a t-shirt
if you spot a Fed, it's fun.
[laughter]
But going to those people who have spent several years, I found much more rewarding because
I got where I needed to be much quicker. But now I'm gonna go back and start making that
journey. I won't say where I live, but where I live it's a little problematic to have a
drone and fly it around. It's one of those things. So, I'll have to go out in the desert
and mess around with it, but I will. Yeah.
>>Male Presenter: Did you have a question? I'm sorry. OK. So, we'll go back to the Dory
question. All three of your books make, would make amazing graphic novels. Other authors
have moved into this medium, most notably Stephen King and Dark Tower series. Any plans
for this in the future?
>>Daniel: Well, this in some ways folds back into the question of are there any differences
between self-publishing and publishing with a major company because, of course, now that
the rights are owned by major companies, I would have to phone in those companies to
see whether they're interested. So that's the trade-off.
So, I got access to markets I might not have otherwise. Now, of course, if and when a film
comes out--I'm sure that will happen--but would I consider that for future things? I
would. I would. But only I think as a companion. I would very often like to do a novel and
then have the graphic novel as a companion.
A while back, I can't remember where I read this, but I thought it was quite compelling,
but the idea of reading a book. Again, I experienced this as a child when I read "Lord of the Rings"
at a pretty young age, was the idea of imagining something and making it very personal. I think
that's really cool. And again, I don't want to denigrate graphic novels 'cause I enjoy
them, too.
But I wouldn't want to just do one or the other. I wanna do both. But anyway, the idea
of imagining something, I think, is kind of cool. It makes it a very personal thing.
>>Male Presenter: Excellent. Is Barack Obama and George W Bush's use of drones criminal?
>>Daniel: Wow, man. Now we're talking.
[laughter]
Wow. Yeah, so OK. Let's see. This is being videotaped.
[laughter]
All right. I'm gonna try to say, this is why we need a settled body of law on this topic,
so that we can determine that. Because, honestly, in talking to some very powerful people at
one point, I remember thinking, "Gosh, these guys are just really normal." And then I tried
to think about what pressures people might be under and this is how law is such an advantage
because I don't think we have law designed specifically for the use of remotely piloted
weapons right now.
We don't. There is an organization. I'm gonna try to remember their name correctly. I think
it's the International Committee, what is it? Remote control. I have it. It's like the
International Committee for the Control of Robotic Warfare, or something like that. I
think it's called ICRACK, whatever that stands for.
You can Google it. But, I know. It's an unfortunate acronym. But what they're trying to do is
lay down a set of principles that I think are quite reasonable. And under that, that
statement of principles, would it be criminal? Well, the remotely piloted weapons, I don't
think so. But I would encourage us as a society to determine that.
And I think it should be something that we determine, not some lawyer in a cubicle or
a hole somewhere. I think it needs to be a public discussion because really, the implications
it has to representative government are very dire. Again, the idea of the application of
lethal force, especially in a military context, representing the nation state should be something
that needs buy in.
For instance, if you were to tell me that two branches of government needed to approve,
you know what I'm saying? You just need more than one person making these decisions. And
then there's the context. And there's crossing borders. All of these are legal questions
that I don't think are settled.
Because they're not settled, I think it's hard to say that it's illegal, but we damn
well better find out soon and determine that. So, that would be my answer.
>>Male #8: So actually, I have an immediate follow-up to that point.
>>Daniel: Sure.
>>Male #8: How would you contrast that to the thrillers of the '60s and the '70s, where
the President has immediate launch decision for thermonuclear warheads and there's nobody
else in the loop?
>>Daniel: I would--.
>>Male #8: Can you contrast that, thanks?
>>Daniel: I would contrast it in this, first of all, it's a miracle we're still here is
all I can say. I think about it. I grew up during the Cold War and I remember thinking,
you see these films about drop and roll and stuff like that. You just ignore it after
a while, that threat. But, that said, when it comes to one person making that decision,
I think that was driven by the fact that, what did we have? Thirty minutes notice if
somebody had launched--.
>>Male #9: [ inaudible ].
>>Daniel: Yeah. And I think that's what was driving it. And what were the consequences,
total, complete nuclear annihilation now? In this case, we're talking about autonomous
systems that might be assassinating people.
I don't think the consequences for mankind, in general, of those things getting out of
hand are anywhere near as dire, even if somebody's trying to get at somebody for some reason
as some sort of preemptive attack. I just don't, I mean, those were very special circumstances--complete
annihilation of the species.
And that's not what we're facing here. So I think this does lend itself to having a
legal process by which somebody has to approve and there has to be a written record of who
authorized what. We cannot have autonomous systems making these decisions. And of course
now, just having one or two people, it can't only be one branch of government either 'cause
again, the idea of separation of powers.
That is key. It's a key element. You can't just cut that out. And who decided to cut
it out? All those things. So, you create this impression of absolute urgency. We must do
it now. I'm not convinced that that's the case here.
Now, if you tell me you have a situation of absolute urgency when it comes to the Soviets
launching missiles against us, well that sounds pretty urgent. Thank God we all survived it.
But that's not the case here. I would say this is very different, very different case.
>>Male #10: My role at Google is that I automate our defenses to being attacked.
[laughter]
>>Daniel: I would imagine you mean that in a a cyber context.
>>Male #10: We don't really have time to, yeah. it's a related context. And we don't
really want to waste the time of having a human get involved.
>>Daniel: Very. Yeah.
>>Male #10: So, machines are a lot faster. And so, you can imagine in the case of like,
nuclear launch against the US, like do you really want to wake up the President in the
middle of the night and say, "Hey, you have five minutes to decide whether to end the
world?"
>>Daniel: Yeah.
>>Male #10: Or do you want to actually send that decision over to a computer because maybe
the computer can have the algorithm thought out in advance of, "Is this actually a real
event? What is the right thing to do?"
>>Daniel: Have you not seen "Forbes?" The--. You're talking about thermonuclear war, or
are you talking about some other type of attack?
>>Male #10: Well, just in general. Like, I realize your books are about the risks of
automating technology, but what do you think about the risks of trusting humans?
>>Daniel: I'm much more comfortable with the idea of algorithms informing humans. 'Cause
here's the thing. Automating things, again, things change. Conditions change. And anybody
who's designed systems knows that inertia starts to gather around systems.
And that portions of any system of any considerable complexity become black boxes. And people
start to trust what comes out of them. And sometimes, those people leave and other people
come in and use it in a different context. That's where automation starts to concern
me, is its ability to persist past the point from which it was originally designed.
And I would take some of the algorithms used on Wall Street as an example there. I mean,
here you've got quants who might have been looking at a statistical model in a different
context, trying to learn about risk and then people from the sales force say, "Hey, can
I borrow that to try," 'cause it boils risk down to a number.
And then it just spins out of control and people start using it really where it should
not have been used. But I don't know if that answers your question. You want to bring it
back to nuclear annihilation, what was it?
>>Male #10: No, no. That's fine. Basically it's similar to like, Chauffeur. Chauffeur
doesn't have to be a perfect driver, but if it's better than a human driver, that's pretty
impressive.
>>Daniel: It is. It is. You'd want a failsafe, obviously, and you'll have it. They have that
red button. And there's a driver in the seat right now. But when it comes to cyber war
and things like that, I agree 'cause all of that's happening at a very rapid pace.
But again, I would imagine as it starts to escalate, human would be notified and they
would be able to either pull the plug. So, it's not like after the explosion goes off
in the real world. That's a very different case. You start to roll things back. So, I
hope that answers your question.
>>Male #10: Yeah, thanks.
>>Male Presenter: All right. How much of David Brin's "Transparent Society" inspire or inform
the social structure you used by the Dark Net, particularly in "Freedom TM?"
>>Daniel: Yeah. Actually, I enjoyed that book to some extent I would say. The idea of transparency,
the idea that some, if someone touches my data then I'll know it, that intrigued me
that it's a fish bowl. We can see each other. 'Cause I mean, our data is consumed many times
a day by various entities.
Yeah, his book compelled me a bit in that degree. The whole question of privacy and
anonymity, I struggle with it sometimes, too, because there's a part of me that thinks we
need two internets. A big part of me actually thinks we need two internets. One for anonymity
and one for critical infrastructure, in which you don't have any anonymity.
And so that's the one that you run your financial system on, that you operate your dam sluices
from and your power grid. But if you want to have free speech and play games and stuff,
I think anonymity's OK. But when it comes to transparency, I do agree with him that
it needs to be a two-way street. So, quite a bit I'd say. Quite a bit. Thank you so much
for coming, guys. I really appreciate it.
[applause]