How To Recruit, Motivate, and Energize Superior Test...

Uploaded by GoogleTechTalks on 08.10.2007

>> FELDSTEIN: Is the audience in here mostly test managers or QA managers, individual contributors?
>> It's a mixture. >> Feldstein: Mixture, okay, good. So one
thing is that whenever I go to these conferences, I always say, hey the cool thing about these
conferences is you get to see your photography blown up, I never get a screen big at home
so just indulge me for a minute there. I promised Harry I wouldn't do those standard jokes but
I couldn't help myself. So I'll just introduce myself, talk about some common quality issues,
talk about what I'm calling a classical test engineers, talk about what--my opinion of
what an ideal test engineer is, developers' perception of testers, some organizational
support requirements, eight reasons why I think--why test engineering is better and
this started from a conversation that Harry and I had maybe year and a half ago and evolved
into this talk actually. So I think my whole speaking career I could blame on Harry, and
some alternate--alternative approaches that I've come across. Some of the few retention
tips, some common issues I ran into, you know, just wrap up conclusion and then Q and A.
However, I'm happy to take questions and challenges at any time, you know, this is just my opinion,
my way of doing things. Also I have to say I'm here with my colleague Nitin Akarte and
so we'll have two, you know, his approach will be a little bit different my mine. I'm
sure your approaches will be different so feel free to chime in whenever you like. So
who the heck am I? I've been programming for 24 years professionally since 1981. I've been
in software test for six years and test automation for a little over five years now and I've
been married for three years so my wife likes to hear that, that's how she lets me travel
to these things that I'm advertising her. So my current position is managing a group
of 41 engineers located in San Jose, Netanya, Italy--Netanya, Israel, Bangalore, India.
And what I'm working on is it's a network management operating system what--I work for
Cisco. So Cisco makes routers and switches. Our customers buy anywhere between few to
a few thousand of routers and switches, anytime you have more than a few--that many of anything
you need software to keep up with it and so we--the department that Nitin and I are from
write this manage--this software to manage the network and manage these devices on the
network. And I'm--and this--with my specific responsibility is around a new network management
operating system. It's basically a job application and distributed job application talks to a
bunch of devices at one end and it has an API and a GUI on the top end and then I like
Nitin to just spend a minute. And also I should say that six years ago, Nitin hired me at
Cisco so he gets the blame every time I make a mistake. I go, I didn't hire me. So here.
>> AKARTE: [INDISTINCT] >> FELDSTEIN: Yeah, just the same thing, introduceů
>> AKARTE: Yeah, this is on right? Hey. I joined Cisco about 11 years back and more
or less same background but I don't know what language Jeff was programming but I started
with assembly language. So that really pushed apart from the current [INDISTINCT] because
one of my experiences that when you develop assembly language and you will start develop
CC++ and looked the code and you realize that enormous wastage of resources while writing
the code in assembly--in C and CC++. And one of the reason I'm in test engineering is that
after doing development for so long, this was the area which can really give you the
bird eye view of the product, the customer, the technology and everything in one shot.
So that's one of my little short introduction. >> FELDSTEIN: Okay. Thanks. So I had a little
different approach to test basically I was--I was--my introduction to test was completely
underdressed. I spent from 1981 to 1999 as a developer, avoiding test like the plague.
And I was successful for that amount of time. In 1999--I mean 1997, next thing to say that
say that's '81--yeah, '81 to '97--in '99. In 1997, I was living in Santa Barbara working
for Cisco, they shot down the office and asked me to move up to the Bay Area. I wanted to
stay in Santa Barbara so I left--I left Cisco. In 1999, I get a call from Nitin who says,
you know, come back to Cisco it's going to be really cool, you can stay in Santa Barbara,
life will be great, I really like to have you here. And then he said, will you be a
tester? And I said, well, okay. I liked the company at that time. I'll come back and test.
So it wasn't something I wanted to do. It wasn't something that I picked out but I wound
up doing it. So I'm sure all you guys are familiar with this common quality problems,
bugs in the software found too late, regression bugs are introduced while we're moving along,
in other words the software itself regresses. We may have some insufficient performance
poor scaling, poor response to stress in other words what happens if a system's design for
a thousand usage and you put 2, 000 on it, that's what I call stress testing and poor
long term stability, we call that soak testing. In other words, we try this--in our case it's
a commercial application. It's not something we have control over, the way a lot of your
software is it's more like the--more like the Google map software that's given out to,
you know, sold to thousands of people and we can't control it anymore. It's all--it's
out. And so what we try to do is simulate two years worth of usage or one year worth
of usage in three of four weeks and we call that soak testing. I might have missed a few
but does anybody disagree that these are the problems? Have you guys all seen this?
>> [INDISTINCT] >> FELDSTEIN: Okay, good. So what is a classic
tester in my case? It's basically they are--they do everything manually, they come form the
program space instead of--or the--they come from the problem background or the problem
domain instead of the business. It's--and the business instead of programming or engineering.
I think they're often experts in the product and they may understand and use the product
really well or understand the prompt space. And because of that they can get very good
at designing tests and writing test cases because they know--they've used the product,
they where it's going to break and they know where to look for those errors, they know
where those cracks are going to be or the opportunities for error. I think they can
be really creative about--around ways of killing the project. I've seen that at the--you know,
people who are working for me and working at Cisco as well as test conferences people
are really good at breaking the software. And they weren't necessarily software engineers
or even computer scientists, you know, when I first started testing at Cisco, one of the
guys who worked for me was a marine biologist and he somehow backed into a networking career
and then he became a tester. There's a lot of that going on. And then the next sort of
level in the hierarchy of my mind in terms of automation, are people that are writing
scripts. So some people without a computer science background have figured out that,
you know, they--you do the thing--you run the test the first time and you do a terrible
job because you don't really understand the application, run the test the second you're
doing a really good job because you understand it, it's all new you really get--glad you
got handle on it. Third time, you're already getting bored, you know, and missing--and
missing bugs again. At least that's what I was doing when I was coding for Nitin. Luckily,
I got moved out of the way before he noticed. And then the next level is, you know, you--somebody
might look at that the third and fourth time and go, I'm tired of doing this, what I'm
going to do now is duplicate what the human does inside a machine. So I'm going to write
some simply scripts to do the most repetitive test. Not a lot of software engineering, not
a lot of innovative test approach basically just--just doing what the user said. A typical
example of this maybe scripting a GUI scripting tool like a Rational Functional Tester or
Win--Mercury Interactive, WinRunner, Segway has one something like that. In other words
somebody that's basically coding in a high level language like Pearl or one of these
scripting tools but none necessarily doing any advance forms of testing. So in my opinion
this is where when I give this talk and you hear people going, "Blah, blah, blah," so
didn't agree a lot. In my opinion, ideal test engineer is going to be a software engineer.
Why? Because I think that software engineers are good at making sure that a product is
designed for testability right from the beginning. It's hard to understand conversations with
software developers unless we're one our selves. I think it also ensures that--in our case,
we have developers doing a level of testing before it's handed off to test by union integration
test and if you have another developer sitting on the test side they can ensure either automatically
or through looking through reports or at least integrating the unit test inside the functional
test they can see--ensure that that's done correctly. Another thing that you can do is--and
this is a big point that I use when I'm trying to recruit people is that you will--the--a
software engineer can design test systems at least as complicated as the application
itself. One example of this is the space shuttle that is still running on the same program
it runs five computers that they built in the '60s and they vote on the right answer,
the navigation for the space shuttle and it runs on--like it still runs on a magnetic
tape in a loop and the software to simulate--to test that and simulate the environment the
space shuttle has to be in is, you know, huge. I don't know how big it is, but it's got a
lot of people working on it. It's way more complex than the actual navigation software
itself. And I think, just simulators in general, flight simulators same thing, way more complicated
than the software that's actually flying the plane because you have to do a lot more work.
So, there are systems like that where the testing is actually where the unique stuff
lies, not so much in the application. And also I say you know, space shuttle worked
twice out of a hundred and fifty times roughly, so that's not too bad. I mean worked a hundred
forty eight times out of the hundred fifty. They only had two failures, so they're doing
okay. Does anyone--is this working? So, basically, also I think that test engineers can come
up with ways of simulating a customer environment automatically. I know in the case of my network
management application and where Harry used to work and he worked on this MOM-- the MOM
application. You know, you need a lot of computers of--you need a lot of users to--either a lot
of computers or a lot of devices to get a lot--get a lot of activity running in your
application. And if you ever tried to keep 40 PCs running at once you'll know that, you
know, I'm sure you guys have tried it. It's nearly impossible you do that for two, three
days and one of the--you know, the--either software crashes or power supplies are going
out of sum. So, you need a way of stimulating that, plus with you guys using you know, thousands,
hundreds and thousands. There's no way you're going to fill up a lab big enough for that.
So, you have to have a system that's simulating customer environment. I claim it's easier
to do that as a software engineer than not. Does anybody disagree with any of this who
think that I have it wrong? I'm happy to hear challenges because I'd like to learn as well.
But this is my approach, finally. >> Do you have to use this? Okay. So what
percentage of test automation would you say is actually at least as sophisticated as the
system under test? >> FELDSTEIN: That's a good question. So,
what percentage of the automation that I'm building or what percentage out there in the
world? >> Out there in the world.
>> I don't know about out there in the world. I only really know about what I'm--part of
the disadvantage of six years of testing it's all in one company. I know that in the conferences,
you know where we met--it's--I'm a little bit under a rock and so, I would say that
it's still a minority even at--even at our level, but in my group it's most of it. In
other words, we designed a test system right from the beginning that's probably as complicated
as the application itself. Why, because we have functional tests built right from the
beginning, and we wanted to reuse those functional tests in a model based scenario. So we've
basically built all that in a framework to handle all that--all at the same time as well
as a, you know, when you're doing--everybody familiar with Model-Based Testing yet? Okay,
good. And so, in a case where Model Based-Testing is you know--at least I spent most of my time
messing around with the Finite state machine, keeping that running, every time the application
changed, I had to change my Finite state machine. So, we wrote a little infrastructure to do
that automatically for the programmer. So, in my case it's pretty automated, but my boss
knew that I--because of this I didn't have any patience for people that have--that aren't
software engineers. I don't even know how to talk to them that well, I guess. And so,
he gave me basically the guys that were from the beginning are software engineers. And
they wrote tools in the past and now they're doing testing. In Nitin's case, people are
more heterogeneous and he's been having--he's been having more people learn to become software
engineers as time goes on. So what percentage, I don't really know. In my--in my group, it's
about--it's over two-thirds of my test team of software engineers. Ideally, I like to
keep it because our problem space is complicated around 85 to 90%. And the other 90% are what
we call you know, networking guru. Cisco is--you can get what's called the CCIE certification
at Cisco. That's pretty hard to get, it's like getting a master's degree in Computer
Science called CCIE. And we want to have a few of those around and they don't necessarily
have programming skills but other than that, the people have programming skills. And the
one-third that don't work for me is because we hire--we newly acquired a company that's
the basis for our--for the software that I'm writing and in there they didn't--they weren't
so careful about hiring software engineers for test. Most of their testing was manual,
and anything that was automated at the company we purchased was pretty much built by the
developers themselves. Yeah. >> You're mentioning that test engineers are
able to build these sophisticated systemsů >> FELDSTEIN: Yup.
>> ůthat are also testing sophisticated system. >> FELDSTEIN: Right, how do you test the system?
>> How do you--not as usually how do you test the system. How do you keep them motivated
to stay in QA rather than going--jumping to actually building the system thatů
>> FELDSTEIN: Boy, I'm glad you asked that question. Could you hold onto it for--until
I get to the eight reasons thing why I--why I think testing is better? And I would say
the short answer is that I really do think testing is better now. And we'll see why when
we get there. That's a really good question. Yeah.
>> So in terms of putting test under these systems [INDISTINCT]
>> FELDSTEIN: Yup. >> ůas well as the simulations space.
>> FELDSTEIN: Exactly. >> So how much influence does the [INDISTINCT]
on the application itself has influence on what you testů
>> Yeah, that's a good question >> ů[INDISTINCT] approach testing the software
that I wrote. >> FELDSTEIN: Yup, that's a really good question.
And inů >> Can you repeat the question?
>> FELDSTEIN: Oh, yeah. So the question is testers have to be expert in both the application
that's under test and the simulation space or the problem space and how much influence
does the developer have on what the tester's going to test. And the answer is I think quite
a bit because the way--the philosophy from my SVP like our boss's boss is that developer,
in test for every application sit in the same room. And actually, the way he laid out my
group is that there's a--there's a set of cubes on the outside and a set of cubes on
the inside, and test is on the inside, and all of the developer individual contributors
are on the outside there. So right from the very beginning they're trading--they're trading
information. The testers go to all the same training as the developers do. And there's
a lot of hey, here's my API, you know, you should do this or what about this, is this
a--a unit test that I should do or is this a integration test or functional test or performance
test that you're doing. And the other influence that they have is there's two places where
we--where we do formal documents. One is called the Master Test Plan which we do before the
project is committed and developers review that, so everybody reviews everything and
then--and then later on we write detail test plans before we do that and developers are
reviewing that. So they do have a lot of influence on what's happening. I should say that it's--in
our group in Cisco, it's what's called the team environment. Everything is run by something
called the Program Team and it's made up of a Program Manager who's a--I call--I call
her in my case, it's a woman Diane Wingfield. I call her the, you know, the Microsoft Project
expert right, the one with the project in those weird diagrams, I forget the names of
them. And then there's someone from--there's marketing--Product Manager he's marketing
guy. There's development, there's test, there's TAC which is our Technical Assistance Center
Support. And each group sitting around the table has an equal say in what happens next.
And everybody reads everybody else's documents. In other words everybody reads the PRD, everybody
reads--everybody reads the functional spec, everybody reads our specs. So, you know, when
things are running well everybody's got a lot of influence on all of the areas. Does
that answer your question? >> Yes. [INDISTINCT] The other thing is in
the other direction. Now how much influence as a testing organization has on the design
requirements and PRD definition of the software itself, now because you know the domain space.
>> FELDSTEIN: Exactly. So, one of the areas is that I've talked about, is that, right
here on the first bullet is that a tester, their job is to ensure--since they're working
right from the very beginning is ensure that the test--that the--that the design the developers
are coming up with our testable. What does that mean in my case? That means making sure
that the GUI does nothing but--the GUI is nothing but a presentation layer of the information.
It doesn't do any logic or business logic and that all the business logic takes place
behind the scenes through a published API. The other thing that we do is, I have my team
look at the PRD which is the Product Requirements Document. That's how all product start at
Cisco. And it's basically, the document--the software on the test has to meet these requirements
since basically a list of requirements. And so, I tell them look at that for the following
things. One is clear, do you understand what the guy wrote? Second, is it testable? In
other words, you see that requirement, can you think of a set of test case to verify
that the system is meeting that requirement. If not, the guy's got to work some more on
the--the marketing guy has got to work with some more on the requirement. The third thing
is, does this make sense from an overall system point of view? In other words, step back a
minute and take these--a lot of times, there's a lot of individual requirements that may
conflict with each other, that may be incomplete, that are written by two different product
managers even though it's one product and they sound like they're two different products.
So, those are the things that I tell my guys to do at the--at the PRD level. And basically,
you can't execute a project until test signs off on marketing, a development signs off
on marketing, you know, development and marketing sign off on test and then everybody signs
off on development too. Everybody's got to sign off on each other's documents there.
And basically, each person has an equal say of what's happening. Make sense? And also,
I'm talking about the ideals, right? Sometimes we get into fights and the ideal breaks down
a little bit, but that's when we're out recruiting, that's what we say. And so, what is developer's
perception of test engineering? This is what I've seen before. It was when I first came
to Cisco, it was certainly like this, testers are idiots, you know. I remember getting that
from some of those people we used to work with, I can't remember their names anymore,
luckily. And I guess they were partially right, but I'm still here. And testing is boring,
manual, and repetitive. I have to be honest, I thought it was like that from 1980 until
2000 pretty much, until after the first year of testing. And testing is not creative. One
of the things I really liked about my job in programming was that one, is I didn't have
to deal with people. And the second thing was I thought it was very creative. Creative
ways of coming up with solutions to a problem, and I really felt good, when I--at the end--at
the end of the week or the end of the month when I went home and I really wrote some cool
thing, you know? I liked it. It was fun. No innovation takes place in testing. Testing
is nothing but verifying that the application works. Therefore, I can't be doing any innovation.
So, testing is a necessary evil that stands between me and the world seeing my beautiful
product. If only test would do the right thing, I'd get that thing out there. Testing isn't
a career. You know, testing you may do it for six months if your boss tells you to,
hopefully get a bonus at the end, and then move on to the real world which is development.
And testing--test engineers are pencil-pushing quality geeks or tense--pencil-pushing process
geeks. In other words, they said, did you tie your right shoe and then your left shoe
this morning because that's what the--that's what the standard says you got to do. Even
though left shoe, right shoe works just as good, you have to do it this way. So, did
I miss any? >> [INDISTINCT]
>> So, there's two sets of things that I was telling folks in New York about this and I
don't know how--and there, everybody was different companies. So, everybody had different view
points, but I think there's some of this in recruiting and keeping people in test that's
organizational in nature. In other words, that you have to convince your boss to do,
and your boss may have to convince her boss. And basically up to the SVP level, maybe you're
already doing a bunch of this. First and foremost, and I feel like those suffrage women in the
1900s, you got to be paid equally. And that's how it is at Cisco is there's no difference
between a software engineer and a test engineer in terms of--I hope I'm not getting you in
any trouble here. Software engineer and test engineer in terms of great salaries or pay
scales. In other words, we're in the same pay scales, the same ranges, and you actually
get to pick your title. And it could say, QA engineer, it could say test engineer, it
could say software engineer. And there's one, two, three, four and it's all the same. I
think this is really important for a person--for a test engineer to feel equal to the--to the
developers. And for the developers to take on--to get the right people and for developers
also to give the respect that's demanded. Also, I think there needs to be a career track
inside the companies. So, at Cisco, you know, my boss has 320 people working for him. He's
the director, which is a pretty big deal there. It's--there's other directors in--actually
he's got one director working for him and the other--rest are managers. There's other--others
at the director level at Cisco. Cisco is basically a hardware company. And we're a little weird
that we're software developers inside a hardware company. But on the hardware side, there's
VPs of what they call quality assurance, which basically is testing hardware. And quality
assurance, I think that paradigm makes a little more sense with hardware. And so there's a
VP there. So, the bottom line is that people come into Cisco and see that there's a--that
there's a--that there's a track that they can go on. They can make a career out of this
thing. It's subtle but I think it's important. It's also important like I was saying before
that testers and tests in general have an equal say in the project. They shouldn't be
viewed as a service organization to development. And a lot of times, developers will say, "Just
do this, just do that." You know, "Can you recreate this bug for me or can you, you know,
set up this lab environment, because I don't know how to do it, the network is kind of
complicated in my case." And it's important that that doesn't happen, that hierarchy,
that development is automatically better than test. And in my opinion, I don't know how
it is here too. So, I'm sorry if I'm stepping on toes. But just my opinion is that, the
test organization should be independent of development. I like that at Cisco, that all
the way up to the director level--basically, the development engineering groups are a completely
separate part of our business unit than the test is. So, the VP who runs the whole thing
has marketing reporting to him, development, you know, several development organizations
and marketing organizations, some operations organization, and independent--he's got test
sitting right there talking to him everyday. And what--the reason I think that's important
is, because when we used to do it the other way at Cisco when Nitin and I both worked
for development managers, it was constantly the development project was getting later
and later and later and it got down, you know, we had a three month test schedule, development
was two months late and we were asked to test in a month. And although development managers
didn't say, I promise don't do that. There's always this implicit pressure to take away
that kind of importance from test. And also to be honest, its people, you know, that development
manager is not thinking day to day about that person's career staying in test and what they're
doing in test. So, the bottom line is, this is the ideal--I'm sure there's other things
that'll work, but it's--I think it's one thing to keep in mind that a lot times test being
independent of development is going to help a little.
>> So, being independent, you mean like the reporting structure developmentů
>> Yes. >> ůplus they sit with them. They work with
them, but they're reporting to a differentů >> FELDSTEIN: Exactly. By independent, I mean--I
mean independent reporting structure. The way we started six years ago was a development
manager would have twenty people working for him. He maybe have fifteen of those be test--be
developers and five be testers or he may have a bigger group than that and he'd have one
development manager and one test manager. And then that was--it was basically the development--the
first line or second line development manager had both. And it--and we found that we were
able to achieve better quality by having a test engineering organization. Better quality
and higher productivity, because there was more reuse among the team's test technology.
So also, I think test engineers--in my opinion, less is probably a strong word. I think it
helps to be co-located with the development engineers. I've also seen it work pretty well
where development was in San Jose and test was in India, but that's under very specific
circumstances where the testers are very--there's a lot of travel. Back in those days, when
we did that kind of stuff, money was no object at Cisco. And we did a lot--a ton of travel.
We needed a guy here for three months from India. We just bring him here, even though
it costs double his salary to have him here than there. And so, we had a few leads--basically
a few test leads here, and most of them in India, and that worked. But it's a lot better
to have--if you're going to have a group in a--in a, you know, that's remote. I claim
the way to do, is have some development and have some test, have a critical mass if you
can, and have them work on a project together, it increases the cooperation. Viewed as development
peers by upper management so, I think that's just basically another way of repeating equal
pay for equal work, blah, blah, blah. And also, I think it's another important thing
is that--I'm not a big fan of people that say they're quality assurance groups. And
the reason is that, I claim everybody's--assuring quality is everybody's job. I think marketing,
development, and test all have a significant role to play in the quality of the product.
And I never really ask my boss's boss if that's--if he believes in that too. But I think he does,
because he keeps calling us test engineering. And we have a few people that are called quality,
but they tend to be--what I made fun of before as the pencil-pushing process geeks. They're
the guys we go to. Our products have to be what's called TL9000 compatible. It's a compliant,
it's a--like an ISO 9000, specifically for the telecommunications industry. And like
everything in telecommunications, it's been around a hundred years, and it's really complex.
I don't really understand it. So, I have to go to one of these guys and say, "what am
I suppose to do now?" and "Is this right? Is this compliant with this?" And those tend
to be the--that was the help that I get from the quality guys. And does it really increase
the quality of the product? I don't know, but it's certainly not hurting it, I guess.
And they're--and they are the guys basically that are--that's the guys that we're calling
quality. Nobody else on the team is called quality assurance because it's basically everybody's
job. So what about the software engineer, testing software, I think I put this picture
up before they accused him of being a drug addict. Anybody know who this guy is? This
is Lance Armstrong, he's won the Tour de France seven times and now they claim that the first
six times he was on an illegal substance so whatever.
>> Cancer drug? >> FELDSTEIN: Pardon?
>> Cancer drug? >> FELDSTEIN: Yeah, actually EPO which is
the cancer drug which one you're not on cancer, you're not allowed to take anymore so whatever.
The weird thing is there's now a Tour of California which is sponsored by Amgen in Southern California.
Amgen's the inventors of EPO so it's, you know, I think maybe it's the only people that
could sponsor bicycle--bicycling races now. So anyway, what does the software engineer
do, that's testing software? I claim we can design this sophisticated software to perform
not only the functional test but also develops a complex software to stress the system. And
I--and I claim you need the level of sophistication. So then it's getting to my questions of--here
is where I have this eight points of why I think testing is better and I really do. One
is, as I said before, you still develop highly engineered software. It's still sophisticated
and advanced and it's just the purpose of it that is different. Instead of writing an
application that people are using I'm going to be writing, you know--that customers are
using, I'm going to be writing an application that insures that the application on the test
meets its requirements and will meet the customer demands and won't break down in an adverse
environment. And I'll use these things on the phone, I'll use these things in interviews,
when people think they're coming to talk to a--as a development job, I'll use it phone
screening because we'll just ask for software developers and then start saying, "Oh, I've
got a job that's almost like development. It's testing but it's in a little different
way." I'll use it when I'm asking when development--developers are--inside Cisco that are unhappy with their
management for some reason. They come talk to me and I'll say "Oh, look. This is what
we do on testing." So I'll use these points pretty much anywhere. The second thing is
you get to decide what to build. I don't know how it is at Google. At Cisco, the developers
don't decide what to build, it's the product manager and they go--and they--the product
manager decides what to--what to--what the product's going to do and yeah, you have some
influence to tell the guy it would be really great if it did this or that, but in the end
he--that's his domain and he's going to tell you what to build. In test, he doesn't care--the
product manager doesn't care. If the thing's working that's fine, if it's not working that's
fine. I think you basically--for those of us individual contributors, we want to add
some new thing. We basically have to show our boss that it's going to increase productivity
or increase quality and if we get our boss to agree to that, it's--we're going to build
something cool. That's how I brought model-based testing into Cisco. I think it's how you brought
it into Microsoft as well. How--basically I did it--and what was called--what Harry
recommended as a micro project, I did that in my case where I spent a weekend doing it
just myself and then I got an intern or two to do it and then it started taking off from
there. So basically I did something interesting by just telling my boss here's a little experiment,
it's going to be really cheap, let's move on. Developers in Cisco, they don't get a
chance to do that, hardly at all. In general at Cisco the ratio is three developers for
every tester. So in any given project the test team is about third the size of the development
team. That means that the teams are a little bit smaller I think it allows us an opportunity
to stand out a little more being testers because of that. Because of the smaller team, we're
a lot--we're less likely to get lost than the developers. In my case there's--I have
a--out of those 41 developers, 38 are testing up--out of 38 testers--excuse me, 41 people
reporting to me, 38 of those are testing the product I'm working on right now and there's
about 120 developers. I don't know those 120 developers very well at all, I know their
managers and that's pretty much who I deal with. On--the opposite is true on the testers.
Most of those development managers know who my testers are because it's not that big a
group and so there's a little bit more visibility there. The creativity, again it's about the
product manager telling you what to build. It's I think test engineering can be very
creative about what they're doing. I think model based testing is one of those examples
that--in our case simulating our customer environment with complicated networks is another
interesting place where people can develop software. And then also we occasionally run
across these people, I'm sure Google is filled with them. We probably used to have lots at
Cisco, it's getting less and less now, where we had folks that just--we call them patent
machines. You know, you talk to the guy for five minutes and he's got eight patent--eight
patentable ideas just coming out like this. And in testing, you know, there's a lot of
those--you get one of those guys in testing which we have had in our past, and it's just
amazing where you just can't keep up with all the ideas the guy's having on how to test
a product. So, also--this is probably the best maybe I should have put this in the beginning.
You get to break your buddy's code. You know what could be more satisfying than your friend
going home all pious that he wrote this really cool thing, you run your automation overnight,
he comes in the next day and the thing, you know, you hand him the cup of coffee and said
"Oh, yeah, by the way your application crashed last night. Here's the--here's the live file."
Does anybody hate that part of the job? Okay. Innovation, I--I'm not a very innovative guy
like I'm not one of these ideas--guys that come up with ideas all the time. I'm basically
a good plagiarizer. I plaiged all my management philosophy, all the stuff you like from Nitin,
anything you didn't like I made up myself, I plagiarized a ton of the advanced kind of
testing I did from Harry, that's basically me. But in the world of testing I'm able to
get four patents on testing. In--they're are all for patent pending and that saying a lot
for a guy that's not that innovative. I just don't think I am and I think that if I can
do that then others can do that as well. There's must be a--it's a lot more wide open space
I think because software development has been around a long--very long time, complicated
testing specially on the kinds of applications, kind of systems we're working on I think is
relatively new and so I think the cracks in what we've done are a lot wider than before.
Another thing which I think is cool is you get to have the customer interaction and the
business knowledge that developers don't have. In other words at Cisco we learned to never
send developers to the customers side because what they wind up doing is patching something
there. Nobody knows what's there, it's a little different here where you have control over
it all but in our case we have these commercial applications and the developer is going to
go fix something and it will never get back into the main line source code and we're done.
Another thing is that developers, it's like if you go to a cardiologist with a stomachache
he's going to check out your heart, you know, if you have a headache he's going to check
out your heart, you go to a brain surgeon with, you know, your foot hurting, he's going
to go, "Well there's something wrong with your brain." Because he's, you know, those
are the developers. The tester I think tends to have a better--a higher overview approach
of trying to figure what is the customer doing and understanding that customer. So they are
a little bit more natural of a person to send to the customer, to talk with the customers,
interact with the customers and to me it's a good break from the actual day to day, you
know, grind of doing a work and so I think testers just get out more, is the way to say
that. Number eight, did I mention you get to break your buddy's code? This is the part
we're supposed to laugh again. So, it's really only seven. So let's see, some alternate approaches.
Any questions so far? Okay. Some of the alternate approaches that I could feel like, one is
interns. I'm a big believer of interns, anybody here an intern, still in college? Here part-time?
Good. I'm going talk how--about how we exploit you. So, I would say that the people that
work for me now that I'm a--like least likely to give up. Nitin and I are supposed to be
a team and we work for one guy and, you know, we had this opportunity recently where I could
give some people up to Nitin, and you know, I just can't bring myself to bring--give my
interns up. I don't know, they feel like children to me. I have--they give--they quit, you know,
they left McDonald's at three bucks an hour to come work for me for ten bucks an hour,
super motivated, you know, they'll do anything you say it's just--it's just amazing. They're
really happy to get their job. It's pretty cheap because we're not--we don't--I think
at Cisco we pay them time off but still it's a very--it's still very, very, cheap to have
them. They're just so jazzed to get a good job that they, you know, highly motivated
and a lot of high energy and, you know, they're willing to do the testing is the bottom-line
because they're like, "It's Cisco, it's Google I'll test, I don't care. It's way better than
flipping burgers." And what we tried to do when we can, you know, since 2000 we haven't
done too much of this at Cisco, but we try to convert people to full-time when they graduate
and then by that time we've had people in test for two or three years and they see all
of these things I'm talking about and they're a lot less likely to leave or a lot more likely
to hang around because they've got that experience. At first they couldn't leave and it's kind
of like me, I couldn't if I wanted to work for Cisco in Santa Barbara, and after a while
they like it better. It's not every one but it's most people. And that's why I said they
might enjoy developing this automation and I think there's basically two interns that
are still working at Cisco that started with me I think five years ago and they're really
great, they're, you know, they're still very highly motivated and they're shooting right
to the top. Another guy that I don't know how he talked me in to coming--his wife works
here now, this is when he was single. He used to work for me and he works for Nitin now
and figured, "Yeah, I'll take the test job." He was right out of school and it was a similar
thing, get right out of school where--he wanted to get in Cisco really badly and now he's
pretty much one of our most creative testers that we have. You know who I'm talking about?
And so what's the second alternate approach I found? One is--and this I haven't done but
some of Nitin's team has done where they go, "Just come to us test first, develop later,
stay with me for 18 months that's all I'm asking. I guarantee you can move over to development
if you like." It turns out that very few people wind up wanting to move. And the one case
we have--where I have the most data on this is one team in RTP [INDISTINCT] and he said
that he was surprised that at the end of the two years that out of the--I think it was
about 10 to 15 people that did it once switched to development. And that's a pretty good rate
and then the thing is that he didn't even mind if they were switching to development
because then you've got a tester friendly developer there. A guy that's been in the
trenches of testing who is now doing development but it turns out when we use this approach
people don't want to move that much because they also see the advantages. Retention, so
I think it's really important that the leader or the manager enjoy testing. I think you
have to have people that like it better than development otherwise if they are just faking
it--well--I remember Casey Kasem, he was this guy who used to do the top 40 stuff when I
was a kid. He said, "The most important thing in show business is to be sincere and once
you face--once you can fake that sincerity you have it made." So, same thing here and
I think though the leader really has to enjoy their work and another thing is, I think it's
time--I think it's important to make time for training no matter what--no matter what
you're doing there. It's probably true in any job but it's really important in a test
job because we have to understand the customer space a lot better than the developers do
and we need to--we need to do that as well as maintain our software engineering skills
which in the java world--how many people are testing java applications here? Anybody? Yes.
So, the java world has the [INDISTINCT] you know, of basically lots of people contributing
all the time and there is always a data structure coming out because it's an interpreter. It's
not really a language. There's always some new data structure, some new approach and
you got to keep up with that and, you know, G2E, blah, blah, blah, websource, there's
always new things happening. It's good--it's also a place where if sometimes people want
to learn the business skills, in other words learn business itself, Nitin had a guy that
three years ago decided he wanted to get an MBA, you know, and so he got that and his
masters thesis was basically his job at Cisco on testing inside using it for a masters thing.
So, that was another way to keep this super sharp guy one of our top ten performers interested
in the job by, you know, continuing the training. And also management and leadership skills,
people want to move up to management, people want to take technical lead skills and there
is, you know, should make time for that training as well. Any questions? Challenges? Few more--another
thing that helps keep testers around is--like we were talking about before, involve them
very early in the test cycle. So, at Cisco basically as soon as the idea comes up that,
"Hey, from now on we're going to use IBM web services for our--for our web application",
developers go off and learn that so do testers basically every class developers go, the testers
go. As a matter of fact, when we went to purchase this company I was--I'm involved with, it
used to be called Sheer Networks, on the very first flight I went to do the technical diligence,
I went over there as a tester and I was the test guy. And so that shows, you know, that
means that the organization cares enough about looking at their test organization when they
want to buy as their development. So, it's a way of keeping the testers interested. I
went over there, I couldn't figure everything out, so I had to send some of my guys from
India to help dig out some information and then the next thing I hear was from development
individual contributors complaining to their manager about how come test keeps going to
Israel to evaluate this company and development hasn't. So, key--involving testers early helped
them a lot and certainly it pissed off developers, which is probably even better than helping
testers. I'm teasing. If development is researching and training--so, the second bullet is what
I've said before basically simultaneous training. What are some of the common issues? I don't
get respect the respect I deserve from the developers. Those--managers, have you ever
heard that before? Who hasn't heard it before? You have never heard that? Oh, you've wonderful,
wonderful developers here. So, I used to get that a lot. And I go, you know, part of it
is you have to earn it, right? You have to show the guy that you're--the guy or girl
that you are as good as him or better than him or as smart as him or whatever so I think
that's really important to do. Another thing I get sometimes is I'm not--I'm not invited
to the development meetings. Well, my answer to is, you know, again what did you do? Did
you talk to the manager? Did he say you can't come to those meetings? Did you talk to his
boss? Did he say you can't come? Most of the time--or maybe you didn't even talk to the
guy running the meetings. Most of the time they're like, "Sure, come to the meeting.
We'd like to have you sit in maybe you'll have some good input every now and then."
So, a lot of times it's basically around making own life a little bit more interesting. I
think that's the kind of coaching I wind up doing. Development wants me to perform the
XXX boring task. I used to get that a lot. I think though, sometimes there is a just
a little bit of that every, you know, a bunch of development, a bunch of testing is just
boring, you know, sometimes that's what it is, it's mundane, but at the same time, you
know, if we teach developers and if we create tools and create ways of automating that boring
task, we're not going to have to do it. So, that's one way of looking at that. And then
also we need the organizational support to make sure that test isn't viewed as a service
to the developers. So, just a recap is I claim the best test engineer--the best software
test engineers are going to be software engineers. Organizationally, if it's not the case already
elevate the level of test engineering to the software engineer. I think test engineering
should have its own career path. And I think its okay and this is something I didn't talk
about, its okay to move back and forth. It's really helpful to have developers be testers,
testers be developers. I think Harry is mentioning at Microsoft that's one of the--that's a normal
thing in our case in Cisco it's not normal to move back and forth but when we find people
either way we encourage that. And that's all. Any questions? Did I finish on time? Look
at that. How do we get questions from Kirkland, is there a way to do that?
>> Yeah, I don't think they were able to record. So they're not watching us.
>> FELDSTEIN: Oh, they're not watching it. >> There was issues with [INDISTINCT]
>> FELDSTEIN: Oh, okay. Good. I'm glad were not the only ones that have problems with
that. Yeah. >> Question.
>> FELDSTEIN: Yeah. >> You were talking about some people being
your top testers? >> FELDSTEIN: Yeah.
>> I was just wondering how you judge that. >> FELDSTEIN: So, the one who rides me the
most. So, that's a good question. So, we have at Cisco, Cisco follows--this is no secret,
that Cisco follows a general electric model for readings and writings. So everybody in
Cisco gets a rating, one to five ratings, and then everybody in, you know, in a view
in a grade level--within in a grade level gets a ranked as well. And there's criteria
to follow at each level so we will have things like--and standardized across I don't know
if it's standardized across all of Cisco but it's certainly standardized among the 350
testers we have in an 118 developers there. So, I think the top 10% wind up being the
people that can out develop the developers. They can understand the customer space better
than the customers do. They can design test cases and test systems like there is no tomorrow
they're bringing new and innovative ways of finding some bugs. They're a lot of times
it's a matter of attitude where there is like they never say no, it's always can do, you
know, it's like, you know, for being, I'm having this big problem here, you know, this--we
have a bug and it's crashing once every three weeks and it's not--it's not happening often
enough to reproduce it. Can you go after that thing? And he'll say sure, you know, and he
goes after it. So, I think it's also people that get along well everyone around them,
they tend to do that. I think they tend to do all of these things right in our situation.
Am I missing any Nitin? >> And then for people who really bring them
multi-talented skill. For example in Cisco we have a very--slightly different area where
a company--Google we are focusing on the application software, right? We also have a focus in the
network aspect. With a network, eventually we manage by this piece of software. So along
with the software skills set, we also need a lot of other skills set. Like network configuration
technology used in the market and all that. So we basically take the--all our aggregate
skill set and then [INDISTINCT] performance. >> FELDSTEIN: And in the end we're supposed
to see results. You asked about the department. >> So hardware testingů
>> FELDSTEIN: Yeah. >> ůhas always kind of had better reputationů
>> FELDSTEIN Yeah. >> ůthan software testing. Do you think that
is part of why it's easier to sell test engineering ads to communications companies?
>> FELDSTEIN: That's a good question. I don't--I don't know because, you know, I'm not basically
a hardware company. I'm a 100% software. I'm a web application. So my interaction with
the hardware to be used is very, very little. And we've had--just recently we basically
had a move over into our organization from some--from a test--from a director of test
who came over to software and he brought a couple of the senior people there and that's
really our first interaction in the last six years with the people from testing IOS software
in a hardware world. And, you know, I just--I know that they are also--have a long history
of software development over there and probably because it's the only way to test the software
when it's IOS software is, you know, it's basically an operating system, you have to
automate that testing and so they have a high degree of skill there. I think in general,
I think this is going to work for Google also if it hasn't already--it's the--one of the
things we do in a--I meant put this in here because this is specific to all organizations,
especially when you're overseas. I'm sure it works here but even more so in India overseas,
people care about the prestige of the company they're working for. And you know Cisco has
a great reputation in India. When Cisco calls or a recruiter calls Cisco they--people jump
at that. I'm sure Microsoft had done positive, Google does too. And I would use that as an
advantage and say--you know and we've seen that before. People who can work in emphasis
or Wepro or Arnolds tractor company or whatever, but working at one of these big American companies,
HP, Intel's were really big over there, Microsoft is getting big, Google, Cisco, all of these
guys have a lot of prestige there. It's important for people--you know and I learned this from
one on my first trip to India, they--the Indian manager Sam Gandhi said, "You know what this
engineer he's here to please his mother." And it's important--and if he's--if he--I'm
not kidding this is what he said. And he said, "If his mother doesn't hear that he's got
promoted in a year she's going to what to know why what had happened." Who has going
to the answer? The same thing is if the engineer's mom knows the name of the company she's working
for--he's working for or she's working for it makes it that much easier." So I would
definitely, when we go overseas to recruit, I definitely would use that, you know, if
you haven't sincerely thought of it, but that prestige you'll see is a big deal. Yeah.
>> [INDISTINCT] how do you--I mean I'm speaking with how do you handle software engineers
overseas? [INDISTINCT] >> FELDSTEIN: Right.
>> QA is different. I mean you have QA to proceed and you will defend them.
>> FELDSTEIN Yeah. >> How do you--how do you voice this message
across to software engineers. [INDISTINCT] >> FELDSTEIN: So the one thingů
>> QA is that meanů >> FELDSTEIN: ůis that telling that--that's
a good question. Try hard to keep your [INDISTINCT] >> That's it.
>> FELDSTEIN: Okay. So the--I think that those seven points is one thing. There's several
aspects, one is that when we hire people right out of--we hire--we use to hire them--when
we hire them out of school we don't wait until they graduate. We go in as--we get them as
juniors and we don't tell them specifically what they're going to do and then I think
that was--you have talked about this before. And I--well, euphemistically line them up
in the big line and go developer, developer, developer, test, developer, developer, developer,
test and all of the sectors that go out one, you know. And so, then here and it's my job
to cheer them up a little, pick them up, up the floor and go, "Don't worry. Let's clean
it off," you know and so that's one aspect. And then also what I do is when I'm searching
for resumes, so I just go for the software development resumes. And sometimes they have
a little test experience its fine. But I go for the software development resumes, I basically
go to through those seven steps using the fact that it's Cisco to leverage when I'm
there and you know that up there. And I've noticed too overseas especially what people
like are people they can relate to, you know. A developer will take a chance on test if
they like who the leader is, if they like the company that they're working for, if just
seems like it's a good place they interview. Another thing is at Cisco you have to interview
10 people--you have to interview with 10 people in order to get a job there. You know that's
good for the company but it's also good when--to give people the person coming on board a sense
of, "Yeah, I like these people," so and, you know, I'll take this chance that maybe--that
maybe development is better. Am I successful with--I mean testers, but am I successful
every time? No, not even close. And especially in India right now, especially in Bangalore,
it's very hard environment, very hard to hire people so we go through--and a lot of junk
resumes. Yeah, a lot of junk, so we go through a ton of junk resumes. We get pretty good--we
get the pretty good candidates and I would say even half of the candidates that we want
to talk to won't come in for an interview when they find out it's a test job. So it's
absolutely taking more work than development in the beginning but it's--but it's doable
from these things. Do you have anything to add to that?
>> Yeah, the two basic things is, first of all we stop calling QA because that appears
>> Specially the place where you got, you say QA they don't believe that with the engineer.
And second update you'll--where you then say, well to the fact that you join the engineering--the
whole engineering you have very narrow focus of what you can be doing in a big cart. You
join testing and then you get the board ideal plus you are in-house testing, you get to
know the costumer and the enlisted problems. That is--that has some success variants.
>> FELDSTEIN: Plus I'll come home and you dinner with your family. That's all throughout
that one. And when's your wedding, I'll come to that?
>> What do you call it instead of QA? >> FELDSTEIN: Basically test engineering.
>> Yeah. >> FELDSTEIN: We're testers, we're test engineers
and then have developers and developer engineers. Yeah.
>> You know originally the awesome generous really occur. Is that a proof developers records?
>> FELDSTEIN: Yeah. >> How about your input on [INDISTINCT] on
something you interview with? >> FELDSTEIN: Yeah. So once people are interested
then we have to do the other side, right? I've done--what I mostly talking about is
the cell side and you're asking what about the bi-side or maybe it's the other way around.
And so the bi is a similar--is a similar--in my experience a similar interview process
to the software developers. What are software development skills in our case too because
it's networking and keeps networking in schools now. It contacts people. It put something
on their resume and I want to make sure that they know something about what's on their
resume and so we'll ask questions about that as well. You know one of the guys actually
maybe this is what you're hitting on, one of the--one of the feedback that I got from--in
New York last week was that, "Hey, you're dismissing the skills of a test engineer which
are different from a software engineer. In other words software engineers build off,
test engineers break stuff and it's a different skill set. And there maybe some truth to that
but I don't ask for that in the interviews and I--and I'm claiming that it's not really
a different skill set, it's just a different orientation to the same problem. In other
words one persons looking at and breaking it, one person is looking at fixing it and
they--and they both do that from similar backgrounds they're going to come up with the entire fully
carts, >> Now, [INDISTINCT]
>> FELDSTEIN: Yes. >> But I'm not sure you guys already know
this. >> FELDSTEIN: Yeah. That--in our environment--you're
saying that you do that here. Yeah, in our environment they do that too. And one of the
things I ensured was when we first started this, Nitin was the guy who introduced, developers
have to write and you need to test. It's the first task organization to do that. And what
they did the first time was creating from [INDISTINCT] with this memory is they sent
us a bunch of paperwork about how--about how their unit test work. In my group where we
took that one step further we basically like I said plagiarized everything he did and now
we just integrate their unit tests into our functional test. So they'll complete--we automatically
kick off the unit test and those complete well enough we have certain criteria for that.
We automatically kick off our functional test that completes, we'll start off the model
base test. Do you have questions? I'm glad there's such a minority of people sleeping
right now. Anything else? Okay. Well, thank you.