2012 Innovation Expo: Universal Orlando

   
  Announcer: Please welcome Bob Hartline, chief sustaining engineer for Universal Orlando.
  Bob Hartline: Hi, good afternoon, everybody. Thanks for inviting me.
  It's my pleasure to be here. My name is Bob Hartline, I'm the chief sustaining controls engineer,
  I'm a technical guy. I talk about control systems and how we make things move,
  and lights flash and sound come out of things, and that's what I think in the parks.
  What I'm going to talk about today, how this works, oh, they left it, fancy.
  Innovation through partnerships, in other words, how to succeed in innovation partnering with outside
  vendors or commercial off the shelf products.
  I have learned being here today that, you know, if we were talking about spacecraft, apparently commercial
  off the shelf spacecraft are available so I guess I'm done.
  Let's buy one! Anyway, we're going to talk about a few things, the first part is why is this a good idea,
  and I know that C.O.T.S., commercial off the shelf has been the topic for many years and
  everybody has the general reasons why they do that and pretty well still hold today.
  You have reduced development cost because you don't have to recreate the wheel.
  The diversity and third party certification as well, we're talking about there is a lot of different
  products out there, and now technology has advanced in very recent years to where now we can buy systems
  where somebody else is doing a lot of legwork for us, more than normal, to the point that we're getting a
  third party like a TUV or somebody line that to tell us that this component is safety rated.
  So when you're building safety rated systems and we all were in that business, we can take a lot of that
  workload off of ourselves, because somebody else is now doing that.
  That's important to me, because in my business, we run theme parks, we do attractions.
  It's a little different, our systems, most of our systems are life safety rated, a little difference between
  us and maybe you guys is you have trained people on your machine, and you've got training people operating
  in the console, and you put them on the machine and they go somewhere.
  We get Ma and Pa smith from Kansas that have saved up for bringing their family for that once in a
  generation trip to a theme park for the experience and they get on the ride and at best, I hope they really
  read the sign that said keep their hands, arms and legs in the vehicle at all times and that they remember
  that when they get on because they don't know anything else.
  They don't know what to expect, so we have to put them on this machine and you know, we can't rely on them
  to help us operate the machine, so one of our things we had to deal with is what can they do for us or to
  mess up our machine, what can our machine do to mess up them and we
  have to do functional safety testing for everything.
  We have trained operators at your launch control system.
  I have high school kids. And they're trained.
  They know what to do and they know what's going on,
  but it just puts more of a reliance on the system to do the work.
  So the way we operate our systems basically is, I'll pop these up here and run down them a little faster.
  The way we do our systems is we have to buy off the shelf equipment just like we used to say nobody built a
  rocket but I guess apparently again, people are, but when we do theme parks, the mechanism that we put you
  on to move you as a machine that generally exists somewhere else in the world, somewhere else in industry,
  and what we do is we take two or three or four different pieces of that machine or two or three or four or
  five pieces of that different machine, stick them together and we make a new machine, and then we stick you
  on the end of it, and then we send you around and we do this where we have to do it sometimes we do it as
  often as every 15 seconds, we cycle equipment so we've got equipment
  that will cycle in some cases a million times a year.
  But when we go to do that, we buy this stuff from other people because we don't have the, really the funding
  and stuff to invent this stuff from scratch nor do we normally have a time frame.
  Usually we're in a rush, somebody comes with an idea to build an attraction based on intellectual property
  right we take off and run and get it done and that's definitely about a three-year turnaround from concept
  to the day we get it open, but we have to go buy these parts from manufacturers.
  When we do that, we go long and hard and deep into the guts of their company to find out how they work and
  what they do, and what we learn with things like this is manufacturers have better availability to parts, so
  when you have, you don't think about that normally but when you have something like a tsunami hit japan and
  the only manufacturing facility, now you're down to one that makes a thin film capacitor or something like
  that, well the guy that's buying, you know, 4 million of them a year is going to beat the pants off of you
  when he goes over and places his order compared to you wanting a couple
  dozen for something or whatever the component may be, so they have that.
  Inventory and training and maintenance, when you go off the shelf with industrial products that are out
  there today, inventory for standard products is held by distributors as well as you can do assignment, you
  have factory stock so you're not custom building a bunch of these things and putting them on your shelf and
  dealing with that cost, and as far as the training goes, and maintenance cost, same way you have factory
  training available on the items and that we've learned over the time is as things have gotten better and
  technology is improved, things are now we can take more things off the
  shelf and do more things with them because things have more connectivity.
  I can take this box and talk to that box and talk to this sensor, exchange an awful lot of data, and I can
  get all the information I ever wanted to know, and more, and I didn't have to invent a thing, so from the
  process of the one box that takes off and does the operation over here, all the way over to the display
  screen over here that gives me, that's all off the shelf now.
  People are doing this in other industries, not just we do, but pharmaceutical industries,
  petrochemical, all that stuff they need all that same information.
  Manufacturing is now wanting to trace all the raw products,
  I want you to produce in an hour what their yields are, they want all that same information.
  The information is there now, it's free and available and you can get to it and use it.
  We have smart devices now which plug and play.
  We just run one cable to something and we get up online, it tells us what it is and we can control it.
  Another thing for security, that's become a big issue, if anybody knows the word stucksnet from a couple
  years ago for industrial control systems and automation systems, it was malware that has got into the
  control system, designed for one control system, we won't talk about what it was for but it was actually a
  cool thing, you missed it, which is all good but it brought to the attention of the hacker community, which
  after now for the last couple years is spending a lot of time researching that.
  We have learned that the hacker community is pretty diverse, but that's been brought to the attention that
  these machines that control things, that make things happen, make things move, do not normally have any kind
  of protection on them or really don't have a methodology protection because
  they were designed to turn things off and on and control machinery.
  So hackers have now decided that this is a fun thing to do, it's a fun thing to take over a water treatment plant,
  it's a fun thing to take over the power grid, it's a fun thing to take over control of a roller coaster,
  it's a fun thing now to go down the street and change the readings on your power meter, because you have a
  wireless power meter on your house and get an electric bill for $15,000 because somebody thought that was funny.
  These guys are hacking and doing everything, so we've learned that with the commercial off the shelf
  equipment because of the common protocols we can use that technology to protect
  itself a lot simpler than you could do if you have some proprietary protocols.
  We've actually done a system throughout the system that we're starting up now that I can say I had a part of
  that we actually have a patent going forth with my name on it that detects changes into security systems --
  into the security of any kind of automated control system.
  So we're getting there, and making great head ways, things changed dramatically in the last three to five
  years as far as the technology and what you can do with what you can buy off the shelf.
  Another thing is technology life cycle, you know, I'm sure you guys fight the
  same battles with obsolescence that we do.
  Over the years we used to say things were good for 10 to 15, maybe 20 years on the outside.
  Life cycle now in technology from manufacturers is basically three to six years and in the next ten years
  they expect it to three years is about all you'll get out of something, all to expect they'll support it.
  Microsoft obviously and windows operating systems, you know, they always, the new answer is just go buy the
  new one, but sometimes if you got windows 2000 machine and want to put windows 8
  on it you have to buy the computer to go with it.
  They don't care about that. They'll tell you to buy a new one.
  Technology life cycles are changing quickly and again with newer technologies, manufacturers are starting to
  move forward with backwards and forwards compatibility so you don't have to do this complete hardware
  replacement so you can stay in one platform and migrate without starting all over again.
  Again, we'll just go with what I just said, same thing. I'm not going to repeat myself. So where to start.
  You have to determine your functionality requirements, obviously what you're going to build, whether it's
  safety and reliability or always number one in my industry and I'm sure the same thing for you, we have to,
  we run Central Florida theme parks last year according to the
  internet because we don't release official numbers.
  It says we moved about 60 million people through Orlando theme parks last year, and that's an awful lot of
  people coming to ride rides and attractions and take that once in a lifetime experience and they all want
  the different experience where they're over at Disney or seeing us or at SeaWorld.
  So the safety requirements that you're going to set forth operational requirements, maintainability,
  you have to know what your baseline is for that.
  The requirements for functional safety and certification are not going to change whether you do it yourself
  or whether you're going to go off the shelf or partially off the shelf and to go examine similar
  applications and like industries, whatever your process may be, there's a very good opportunity, some piece
  of it is done in a similar fashion somewhere else,
  whether it be petrochemical or pharmaceutical or even the theme park industries.
  Jeff from Disney and I were commenting and touring some of the back areas here that
  the back areas here look an awful lot like our back areas.
  Lot of similar equipment, a lot of types of things, we do in some cases we do a controlled explosions,
  by dumping natural lRP gas inside of buildings, do it every 30 seconds, we dump liquid nitrogen, we dump carbon dioxide, we dump high pressure steam,
  we dump large volumes of water and monitor oxygen content in the buildings to make sure people still come out breathing on
  the other side, we have to do biodynamic testing and monitoring all the time.
  Our systems are much more powerful than you expect.
  I mean, we typically run them, we don't run them at full blast so they have the capability of, you know,
  moving some body organs around if we shake you around too much so we have to make sure we do that same thing
  right every time so a lot of similarities, but we do this all with off the shelf products.
  So there's a fair amount of similarities.
  I will say this, and this I cannot stress enough, success in getting this done is purely in the details.
  It's all about dotting the Is and crossing the Ts, and taking the time to do it right.
  When you start dealing with outside manufacturers, will you find out they are all not created equal.
  I'll say that when we deal with technology, and vendors that sell us technology, I classify them as, in two
  different categories, there are manufacturers that make a technical product,
  and they are out to strive to make the best technical product.
  There are manufacturers that are out there to strictly go for price point and volume.
  How many of these can I make? Okay.
  And that being said, not all parts are created equal, not all sales engineers
  and support people and designers are the same as well.
  When we deal with sales engineers I find in my 30 years of doing this that I deal with the same small group of people.
  It doesn't change a whole lot, so anybody outside that group that really just wants to sell me something so
  they can say they sold something to me or NASA, it's a mark on their card or report to the main office,
  I call those guys peddlers.
  So either a sales engineer and you have as much interest in making this project successful as I do,
  or we don't deal with you.
  You're not going to do this on price point.
  It's not going to be the cheapest part, again people market stuff to see how
  many they can sell for how cheap they can sell.
  When you compare components like an electrical component, you look at the specifications and the details,
  one guy's got 30 volt isolation, the next guy is $10 more but got 600 volt isolation.
  So all to the details of getting it right.
  Rarely will you ever be successful with one manufacturer across the entire product line,
  despite what they tell you.
  You will get a manufacturer and they'll tell you they sell you everything from
  one end to the other and it's the best made ever.
  They all do some things better than others.
  Some guys are better at one product, another guy is better at another product and you will figure out how to
  match them up, despite the fact they tell you they won't work together.
  Lot of things we come up and do, we come up and do it at the manufacturer; you're going to do what with it?
  It's never been done before or they tell you, you can't do that?
  We say why can't we do that? They say it has never been done.
  When you deal with these guys and we set up a mutual nondisclosure agreements so we can share our needs and
  they can share their deep secrets, we want to know what's coming out next year.
  We can influence a product, if you made this one little change which in the product development stage means
  absolutely nothing to them, you make this one little change we could do this, and when we do this,
  now it's great, and I'm one up on the guy down the street.
  We have friendly competition with the guys down the street but we always one up each other.
  I have my secret projects, he's got his secret projects, so anyway, do that, audit processes and
  certifications of manufacturers to make sure that they're doing what they're telling you they're going to
  do, because not everybody is straight up on that either.
  I always tell you, it's listen to the words they use when they tell you, because it's the words they don't
  say is what they're not telling you the rest of the story, when they leave something out of a sentence.
  When they ask a question and you don't get a direct answer it's always a true tale sign of what they're up to.
  I'm not a trusting guy for inventors. I beat them all to death.
  I appreciate you guys having us here, and thank you. [ APPLAUSE ]