GTAC 2009 - Automatic Workarounds for Web Applications


Uploaded by GoogleTechTalks on 05.11.2009

Transcript:
>>
All right. Thanks for coming back in time. We've got our next talk which is from Mauro
and Alessandra about automatic workarounds. They are from the University of Lugano and
thank you very much. >> PEZZE: Thank you. So, I'm Mauro and I'm
from University of Lugano. And I present a sort of a work in progress in Lugano about
automatic workarounds. And as most of the academic work will be more like a look ahead
rather than what we can do in looking today. And I'd like to give you really a different
perspective in what we are doing. This is the picture that I see as a metaphor of what
is going on today. Whatever you have, if when you have a problem in your system whether
it's also during development, or whether it's during the maintenance, whether it's during
the operation, you have to go in a testing environment. So, if your aircraft does not
leave because the pilot sees a straight light, he would actually call for maintenance. And
I've been actually personally stuck in there for a few hours just waiting for the aircraft
to be repaired before leaving. And the question is really this, you know, when you actually
go through testing and then through maintenance that whatever is automatic, is manual, is
tool driven, that to a tool system--that to a tool system, by expert, by non-expert, you
have to start operating and you cannot operate, why is this. That's actually the main problem
that we see. And what we're trying to do in this new project is really to move to a different,
completely different perspective which is actually represented metaphorically with this
guy and which means basically, you don't really have to do anything, you know. What you want
to do is you don't, but you can probably. I don't think you can get everything out,
I guess, get the human out of the roof or get rid of testing. But what you can do and
what we hope we can do is basically have someone or the system itself trying to repair itself.
So, instead of actually--you have to stop and wait, you would likely need to prefer
to have the system working for you. And maybe continue working and that's actually a very
big savings in terms of time. Now, let me give you the perspective from the most Web
oriented perspective. You know, if you have--if you are a user of a Web application, I got
there and if failure occurs and what happens is that you can do a match. You basically,
have to report and that would be some bad report systems. Actually, you have to report
as a user to your application developers--inspection developers may or may not be reachable. Eventually,
you will actually report to the bad report systems. And someone probably in this room
will look at the box and with their all priority maybe low priority for them, they would like
to fix the bug, would be annoyed by this new error, a bad report. It will ship the bug
out and it may take hours, days, weeks, I mean, as a user I'd be waiting for weeks sometimes
even more just for the bug to be fixed, probably, because it is not in the priority list of
the Google developer. And that's actually something that you can simply stay and that
you can do a match about that. And that, what we really want to do is we want to move to
a completely different scenario which is represented in this way. This is the scenario that I'm
trying to explain how to reach which is--whenever the failure occurs you don't really look for
your Google developer to actually try to fix the tool not always at least. But you'll try
to--you'll notify the failure to the application and that's one of the first things or process
we do which actually was very well in Web application, which is you re-the test some
of the tests--their application failed which may not be easy in all the cases. But then,
it's easy because you actually look at the application and you see the application does
not react as you expect. So, you get something that you don't want and you notify that failure
to the system instead of going through this partially manual, partially automated, partially
asynchronous loop which in days, weeks, months, will actually return to you. The system basically
comes back to you immediately and says, "Oh, look, is this better now?" And that's the
idea--on the line of what we are doing here, you're trying to have the system proposing
a different solution and see, whether we can actually fix the solution, that's fantastic.
And how do you do that? So the next question is how do we really exploit the workarounds?
Well, oh, how do we I--how do we really--we really actually try to solve the problem?
And what we try to do is we try to emulate what people do in manuals. Okay, just go back
for a moment and think about what you do when your application does not work, okay? And
this is an example. Actually, there's two examples of the box here from Flickr which
is fixed now, so you don't find it, you don't find it anymore but it used to be in Flickr.
So, you have your favorite photos there and your favorite pictures are on your site and
they are private to you. Now, what you really want to do, you're really--for example, you
want to share it with your family and then if then if you used now to work. So, that's
transition from in my own pictures to--my family picture did not work and that's specifically
because of Flick. So, what you do is you basically look at--can I do something about this before
notifying to my favorite developers, hoping he'll actually react to me? And this is a
Flickr developer not on Google developer, by the way. And you go back and say, "Oh,
I can maybe try to use and make it completely public." And then, suddenly, immediately,
go to your family and it did work in that case. Okay, that's exactly the kind of thing
that people would do, you know. If you interacted in application, the application phase, the
first thing you'd do if you are a smart user, usually is actually looking at the application
and say, "Yeah, I can probably, I can obtain the same effect but with the different--in
a different way." Okay, and that's what we call a workaround. And that's what we call
or what we look for. We look for what we call intrinsic redundancy. What is intrinsic redundancy?
Intrinsic redundancy for us is something which is in the code, which can used in the code
and thus produces the same or similar effect. So, the tricky thing here is can we automatically
look for the redundancy in our code, so that we can actually then have the system look
into the redundancy of the code. And use that redundancy to automatically found workarounds
without requiring the user to be smart or even without requiring the user to be able
to access that redundancy because that's another important problem, you know. You have the
examples and it's entirely simple, you can play with the interface, you know. And I'm
sure I know probably many here have some experience of finding--to try to get the same effect
that they actually didn't get, within almost of this approach by finding some workaround.
And to do that manually as a user you really need to sit down and be able to find the workaround
through the interface, and in many cases and in other cases, in many cases of redundancies
inside the system. And so, in many cases, what you want to do is you want to actually
open the system, look in the system and use that redundancy and that you cannot do that
and why the system can't do it? So, as the conclusion for this is you really want to
have a--we allowed the fact that your family is equal to anyone in your family. So, moving
from only you to your family is equivalent to moving from anyone to your family and then
the kind of information that you are trying to extract, okay? Let me tell you how we do
that. We do that in two steps. Step number one, step number one is that we look for what
we call equivalent sequences. We have an example there. Equivalent sequences are basically
something that you can identify and that it will tell you how, let's assume that they
are there for the moment. So, assume for the moment that we can actually look at the system,
for example, the codes, for example, in the specifications and assume that we can actually
start some information like, oh, whenever you do, your family, you go from--you only
you to your family, you can actually get the same effect by going through anyone and then
your family. Or whenever you actually like to do an operation, you can do another operation
instead. Or you can do the same operation and then another operation and get the same
effect, okay? That's the kind of information we'd like to extract and then tell you how
we do that. At the moment, we actually assumed that we have these equivalent sequences. And
what we do is something like that, we try to have--whenever the user does find a problem,
what he does is simply signals the problem to the system. And the system looks up in
this repository of equivalent sequences which is the way it would work. That's to find the
good one and returns to you and your page. And hopefully, that's correct. And if not
you can repeat it again, and again, and again. Of course, you know, up to the time where
the user says, "Okay, I give up, I don't want to wait and listen to you anymore." So, there
is obviously a trade off between how much the operation is important for the user, and
how much time the user does not want to spend on the operation? So, if you want to look
at the page, it doesn't show [INDISTINCT] okay, forget about that. But if you really
want to look at your account--bank account on Saturday, you really want to know whether,
you know, you want to--what's your balance and you have time to deal until Monday to
get a real answer from the bank, you may actually like to try several times to get the information.
So, you know, depending on the important information, you can go through that loop once, twice,
many times. So, first part is how do I actually create equivalent sequences? The second part
is how do I actually select the equivalent sequence that solved the problem which is
the workaround, what they call the workaround. Let me try to start telling you work. How
do I--do we extract equivalent sequences? Honestly, what we did we started with a long
set of experiments. So, one thing that we did and you've seen it in the strength of--under
the top, we actually mind the Google bug repository, and to see whether there were workarounds
there and you have to understand what they were, how they work, and why they work, whether
they're automated or not. And we eventually, identified three major classes of workarounds.
We'll put it in sequences. Class number one is what we call Function in Null Operations.
These are operations that affect anything but functionality. They may affect timing,
they may affect scheduling, they may affect anything else but not the functionality. And
we have a simple example there which is actually one of the favorite workaround in Google application
we found which is Set Timeout. Set Timeout does change the application because in the
Set Timeout we have a different timing at different response time. It does not change
the functionality, we should not change the functionality, okay? Now, we've noticed that
in many cases that's accepting a Set Timeout may solve synchronization problem, may solve
risk condition problem, may solve many different, many little problems that often arise in many
Web applications. So, what you do is--the many cases, although, the Set Timeout should
not change at all the semantics of the application, when the semantics is wrong, whether in the
application phase, inserting the Set Timeout sometimes helps you in solving the problems.
And that's one of the possible classes and we have several others that we'll show you
later. The second class of equivalent sequences is what we call Invariant Operations. There
are operations whose functional behavior is important that they actually complement each
other. So, for example, you may--if you have some shown item on your webpage, you can hide
and can show it again. Hiding and showing something which is available it doesn't change
anything, we should not change anything. And if another kind of thing that may be really,
really helpful in terms of solving little problems like in utilization problem, like
state problem, so what you do, you may do is basically, yeah, that's actually interplay,
hide and show, and see whether they actually work together or not. And that's the second
class which we call Invariant Operations. The third class is what we call Alternative
Operations. And that's what we've seen before, by the way. And that's you see many times,
in many different application which is in many, many cases, you can really get the same
effect through different steps, okay? For example, in this case and that's again from
Flickr, so these are actual examples from flicks. So, we actually start and working
on Flickr first. You can set tags of your favorite pictures and you can add tags. And
you can set them in multiple set of tags. I hope most of you are familiar with Flickr
and not just the--and application. And you set the toggle, you can set several tags to
your picture and then, you have multiple parameters. But you can also set the first tag and then
add the other tags. The effect is exactly the same. You know, you actually eventually,
have two tags, one and two attached to your picture. But the difference is that you actually
use different codes. So if one of the code is 40 and actually brings you to failure,
you might try with the other one and hopefully, it doesn't mean that it will you the same
failure, okay? Now, how do we--work with that, let me give you some more examples and try
to explain you to better what they are. And use some complete examples, and I think of
something good, some of you are familiar with these Google Maps, okay? These are by the
way, you see up there issue number 519, this is where issues--issue that you find it in
the--about the repository of Google Map and they're actually what's important is that
it will set information for us. And that's one for example, you know, that's--up there,
you see a piece of code that is suppose to center the map and display any for Window.
And it doesn't work, I don't know whether you see but the info Window, the picture is
very small but the info Window is over the border so you don't see the info Window on
the top part of the picture. And the workaround that you find in the actual bug repository
that that should really do that to actually solve it is SetTimeout before you put it in
center and this so what I think is a synchronization problem between the two things. And, you know,
I think that simple SetTimeout would actually give you the correct display with this one,
okay? That's the kind of one example of what you call Functioning Null Operations. Another
example, that's another one issue 1305, there is a function and in Google Map again that
allows you to draw a Polyline, that was faulty in some old version. These are old solved
problems, so it's history more than just a reality. So, don't try now because they don't--they
won't work now. Let's say, a Polyline and in that version, you could actually draw the
vertices and you can see that, you know, the beginning of the Polyline and then the vertices
and you can thus see the Polyline. And what was--what this suggested in the part of repository
is again, or try to do something which should not do anything but actually help in solving
the problem and that something is basically delete and insert the last words. Okay, somehow
and, you know, to me it sounds magic because I'm not a Google map developer, so I look
at that and I say "Wow." So, some of you may actually sound normal or reasonable by simply
setting something that is exactly what we call an Invariant Operation. An Invariant
Operation which means you undo and do again and you get exactly the same result. Doing
that before drawing the polyline would actually make the polyline work, okay? So, the problem
of their class of workaround is this one, now again this side of the picture does not
help but you may see up there a little red spot in the top picture and that's a marker
and that you don't see, I'm sure this is--the bottom down there that says "Delete marker"
that you can actually delete them out, you can remove the marker basically. Now, we have
a marker that doesn't even work anymore, so, you remove the marker that decode, that's
the issue and then press the button, the marker stays there. And again, you look in the repository
and you find out that you can actually substitute the show and the hide operation with different
sets of operations, so you step into the show with another varlet show and then you offset
and hide with the removal varlet. And they do actually work and it's exactly the same.
It's exactly the same. So, when you press the button the second time, the marker correctly
disappears on the picture. And by the way, if you remember, I started with an example
which is basically playing with the interfaces. This is hidden to the user, you know, you
favorably develop or it can actually help you in removing that the problem by inserting
or suggesting that workaround to the application developer. But as a user, you click the button
the marker stays there or what, change application, if it's may not be that important but maybe
important then you can do anything than waiting for the application to work. So, let me actually,
now tell you go to the second step for a moment. So, right now, what I tried to show you is
that there are existing basic redundancies, some intrinsic redundancy it's all for some
given classes and can be easily exploited actually, can easily be exploited manually
in applications to solve problems. Now, let's go for, a step further and see how do we actually
extract to select the actual workarounds. So, think about that. You know, I can spend
a lot of time and I'll show you why, how to correct the equivalent sequences now. I have
well let's say 10,000 equivalent sequences in my repository and my user says, oh, fail;
and I then I have to pick up one. I mean, I don't know how many users would actually
try 10,000 times fail, fail, fail to pick up the good one. Okay, so you really do want
to have something and we are experimenting with this kind of thinking which we call a
priority scheme, a priority scheme, which is very simple at the moment. We mentioned
two things, one is the success rate and the other one is the number of successes. So,
the success rate is obviously the ratio between how many times you succeed compared to how
many times you actually use that specific workaround, that specific equivalent sequence.
So, you pick up SetTimeout 10 times and it was effective only five times, so that's 50%
of the times and the success is actually how many times it actually succeeded. So, let's
say I have two operations, I actually have example here, they have two operations, I
have a SetTimeout before a set center and that's a classic workaround I would say in
Google maps. We used it 14 times and out of 14 times, we got 7 successes and other workaround
is you actually--after the other operation, then another kind of workaround, we use it
only two times and we succeeded once. So, we have the same number of success rate puff
but we have a different number of successes. So, our strategies so far in this something
that we're working on is, you actually look at the success rate first and you see how
much, how well it did help. So, you prefer to get someone with a success rate a 100%
of the time rather than 10% of the time, 50% rather than 40% and in the same class, you
actually, prefer something that has been successful many times because it's likely to be successful
more often. And then of course, you know these priorities have changed dynamically, and so,
in this case, we're actually to pick up the first of the two rules if both can apply to
your faulty page. Okay, that's actually downloaded automatically by the system. So, we have somehow
where you're dealing to this equivalent sequence repository with all the possible alternatives
that can actually represent the workaround and the very simple dynamic mechanism that
assigns priorities and tells you the more you succeed basically, the more likely you
will succeed in the future. And that one more thing is of course, pages maybe used by more
than one user, so actually that's why we look for many workarounds. So, let's assume that
we have a workaround, let's assume that our pool user had developed 700 times through
different fail, fail, fail before actually getting its success. So, what we do then is
basically we store the valid workaround once the user says "Yes, that's one, that's succeeded."
So, that the next time the next user will try and will see the same failure would actually
at the same time, have a privilege workaround. We could actually attach the workaround to
the user of the page. We don't do that yet because we prefer the user to select workarounds
because they're not really sure about how safe the mechanism so, we prefer to give you
the plain application and you pick your fail then you say, well once you've failed you
prefer to actually try your workaround but then do nothing and that's another part of
but we're actually working on it now. Okay, now let me go through the experimental part
which is sort of prelude to the demonstration. We did run experiments, of course, because
that's nice, that's exciting, I would say. It's a new idea about does it work, so we
build the very simple prototype which is basically a proxy. Okay, you see proxy okay and the
proxy filters the communication--the user that you have on the left and the server that
you have on the right. And in the normal case, the user adds something good and the proxy
transmitted the request and, you know, the proxy doesn't do anything but inserting the
little marker the page that allows the user to seek another and fold. So, the proxy needs
only to know which is the page, and which is the failure, you know, which page the failure
belongs to. So, in talking back recording the flow of pages, however, when the user
gets a failure, he can seek another failure and then you actually activate what we call
the automatic workaround module that search in the equivalent set of sequences that are
available and discard the one that are not applicable to that specific page, find the
one that are applicable. Try to understand in which one is the most likely successful
one, apply that and return to the new page of the user who may be happy and accept it
and that's an important information because they store that information and say "Look,
that's a valid workaround for that specific page" or may not be happy and try again or
simply give up, okay? And how do we generate the equivalent sequences? Well, we generate
them in many a couple of different ways. Here, I showed you one way which is the one we're
using now, and then there is another way which is the one they're working now. The one we're
using now is simply and we showed the experiment that we rely on the ability of the developers.
So, the first time, the users seeks the workaround and would ship it to Google and we hope that
it actually solves it; the first time the users sees it as failure we ship it to Google.
We hope Google actually corrects it and that's populating the equivalent sequence repository
and in a short time, we have something to try before shipping the failure to the Google
designer and hopefully, in a short time, we get to the repository which is good enough
for us to solve some workarounds. The other situation is to take advantage of the knowledge
of their billing gap right now which is the understanding that there are few classes of
very frequent workarounds that work very well can be generalized, can be extracted from
the information from the code, and so, we look at the specification, we look at the
code and I'm sure you can tell me who are and what are the workaround because, you know
I'm as Nicolas gave this morning said that I'm a professor, and so, I don't try them
anymore, you do actually, so. I noticed that for example, setTimeout specified as being
having no functional operations so you can say "Okay, that's actually category number
one." You may get a bit more actually, we'll show you later. You can get some other operation
that will belong to category number one. So, operation that by construction, by definition,
by specification should not affect the behavior, the functional behavior of the system and
so can be in server. You can look at the specifications and say "Oh, I know that." Adding and removing
it doesn't make any difference, so it's a set of class of workarounds or equivalent
sequences and, you know, even you may even get information about, oh, adding multiple
tags or one tag at a time eventually gets the same results and then the third category.
So, in the future, we may actually and we are trying to investigate now whether we can
populate that repository with the set of workarounds of equivalent sequences from the beginning.
Okay, right now, we don't have data about that we do have data about the first experiment,
so that's actually the report of the first experiment. I can actually illustrate that.
It's fairly simple report, so, I'll tell you what we did. We took our favorite batch of
repository probably not because it actually contains a lot of very well structured and
useful information but because it's particularly faulty, and which is the Google Map one. We
looked at the issue that was solved as a workaround, so if you look at the repository, you would
find issue number 33, 49, 61 to 71, blah, blah, blah. These are all issues that we identified
as solve with a workaround and we simulated this game. The first time that we started
with an empty equivalent sequence repository and the first time we had failure, the failure
is shipped to Google and Google corrects it and we actually paved the correction which
is in that case a setTimeout before we open it for window which is now a functional operation
and we put it in our equivalence sequence repository. Stop, then the second time, the
user signals a failure that's actually they enter 49 in the repository, we see whether
the equivalent sequence we have, it's only one right now which is setTimeout before by
windows applies and if it applies, they actually try to apply it to solve the problem, it doesn't,
so we shipped it back or we simulate the shipping through the Google designer and the Google
designer comes back with another kind of workaround which is actually an alternative sequence
of operations. And before I go ahead, you know, the third time, back in entered number
61, again, we tried to understand whether we have solve the problem or not, again it
doesn't. It's a setTimeout again actually but this is not the same setTimeout. So, look
and that's why we work now. SetTimeout works in many cases but you don't want to simply
set the setTimeout at any possible place in the line of code because many actually get
millions of different things. These are actually privilege locations because they're all locations
before instruction that are potentially interfering with other instructions because of some raised
conditions, it is some ordering conditions. So, you put the setTimeout before opening
a window, before it sets time to the window, not in general and, you know, what experiment
we went there, it was sort of very frustrating because it's not the first, the second, the
third, the fourth, the fifth and none of them was actually the user, oh the sixth was, the
seventh was and I wonder why after the fifth most of the times you actually succeeded.
So, what you see in that picture is two things actually. The first thing you see is down
this red and green line, okay? That red and green line is the summary of the 24 problems
we actually analyzed in that order in the Google that repository and we try to used
exactly this approach. Start with an empty equivalent set, try to populate with the first
equivalent sequence and try to use that for the second one, fail, populate again, third
one, fail then populate again and you noticed that after one, two, three, four, five, then
the success rate is really, really high. Okay, so we really have that, if we have this system
in place, we could have had the most users happily served by the system itself without
having you guys involved in the repairing, okay? And that's' because from a given point
on the setTimeout which at set center is fantastic. It solves a lot of things. The setTimeout
before the opening for windows and stores a lot of things and so on and you see what's
we actually have recorded the experiment is not that big actually, it's only 24 problems
so far but you see we have the workarounds we've been using the 24 cases. So, this is
a little bit small amount, okay? That gives you a sort of that feeling that it's not that
you have one problem, one workaround, another problem, another workaround, another problem
another workaround. No, you have one problem, one work around, oh, another problem, the
same workarounds work again or the same workarounds work again. So, that's why they are really
thinking that in the near future we can actually give some partially automated rules that can
say, "Okay look, pick up all the operations that share some common features with the opening
for Windows with center." The features obviously the possibility of interfering because of
a different timing and generate automatically potential use for workaround which is for
the setTimeout before. Okay, pick up and so we really think that this can actually be
automatically measured if you used in that way, even if you can not automatically generate
it, we can simply leave with the same situation we have now for the first part of the first
slide for the system and then soon get to the point that the number of that reports
goes from, like, 100 to 10, at least for this specific kind of blocks. But we have another
thing at down here which I try to explain to you what it is because the other question
is how much do we annoy the user? Of course, you know, wear the shoes of the user right
now, you try to get something done and you failed. Okay, being able to say or fail and
then you get another reply which is correct, or base again fail and get another reply which
is correct. I'm sure you don't mind--do it two or three times not for every page of course
you know, but let's say once in a while. You would mind doing it 20 times. So, we also
measure the efficiency of our priority scheme, which is fairly simple and of course it was
on a small set of workarounds but to think that they can be applied in many different
parts of the JavaScript in general, you know, the single workaround can be applied many
different times. And we found out that we have--we measure the average number of attempts
for each workaround before solving the problem. So what it means, the highest priority workaround
that did work for the first time, okay? And in many cases, it's actually the results and
the worst case we had so far is 2.8. So, we think it's most at three clicks. We are offering
that in average, we solve the problem, so that's an average number of applications per
workaround. Right, it's, you know, whether it comes too easy or not. It's a good measure
actually. You know, we can actually--we have--I think the worst case was in clicks if I remember,
which is annoying but not that annoying compared to waiting for three months or someone you
don't know solving the problem and then suddenly shipping the solution. Okay, so, applicability.
So, so far, I tried to convince you that they have a good idea, that's easy, and that it
can work, that's a bit more difficult. How often can it work because you say, "Oh yeah,
great." If you can solve two problems per year, oh come on, you don't really, you don't
really, you don't really help us too much. Yeah, I do what,
you know. So, we try to start this simple measure which is we took a game as a Google
Maps repository and we use this simple search mechanics which is look for pack a match,
look for workarounds. And we found the word workarounds in 63 different angles. So, I
don't have enough students to actually have them looking at each single entry with single
fault but, you know, I could have people looking at that 63. So, we look at that 63 and only
43 actually are workarounds because, you know, the word workarounds a person thinks like,
"Oh, this is a problem, it may be solve with the workaround. And it's not the workaround."
You know, there are some cases where it's actually workaround. So, out of--the Google
Map repository, we've--we've been able to identify 43 actual problems that have been
solved with the workaround. And then we took them and we try to solve them with our automatically
generated equivalent sequences. And we found that 14 out of them can be solved. So, now,
if there is anyone strong statistic has stopped here because, you know, for the number such
small so you can't generalize too much, but at least in this experiment we can claim that
we actually can find automatically 33, 1/3 of the workarounds that you guys generates
manually right now, not a big number in absolute terms 14 but if you generalize well enough
and you think how many times you actually use that for work solving the workarounds,
it can be a big amount of saving in terms of--for the other work. Now, at this point,
I want to convince you even more by having Alessandra showing you a short demo about
what we did as an experiment. So Alessandra would show you both two different experiments
in how and of the effectiveness of the workaround and effectiveness of this notification of
this access to the actual system. >> GORLA: Thanks, Mauro. Can you hear me?
Okay, so, I'm going to talk about a couple of issues in Google Maps. So, is there any
Google API Developer in here? Oop, much better like this so that you don't hate me. Okay,
now, let's see what happens when the users are not using the automatic workaround approach.
So here we see a Google Map, pretty simple that can be dragged. And as you can see there
is a marker in here that is centered in Zurich where we are now. And what we should see is
an info window that appears as soon as the user clicks on the marker, but trust me, I'm
clicking and nothing happens. So, what can the user do now? Nothing. The user can just
hope that the person who developed this page somehow realizes how to try to ponder workaround
and avoid this problem or wait until the developer goes into the Google Maps issue page and brief
for the issue to the Google Map Developers and wait until the developers fix the problem.
And this is--this is actually a known issue. This is the--the issue number 1118 and as
you can see it's been recorded in here. And it's now fixed, but, yeah, again, until the
developers don't fix the problem then the user keeps experiencing this. And all the
applications who are using that particular function that fails will also have failures
and also note so with other pages. But now let's see what happens when the user instead
is using the automatic workaround approach. So now, I'm going to start the proxy. Okay.
And now, I already configured my browsers to go through this proxy. So now, as soon
as I reload my page, the request will go through the proxy. So, now let's see, it works. So,
as you can see, I, again, have exactly the same page but I have now this small button
on the top left corner that says, "Find workarounds automatically." So, when the user sees that
the page has a problem as we see here because, again, the duplication is not working, because
when we click on the marker, nothing happens, the user can now say that, there is a problem
here. And we see that the page--the map is reloaded. So, and we see that now the application
is working. So, whenever you click on the marker, I successfully can see the info window.
Now, let me try to tell you a bit of what we did. And by doing that, I need to show
you the original code of the Google Maps API that we--you saw in the page. As you can see
it's pretty easy. Yeah, shoot, that fits so everybody can read it better. So basically
in that map, we just create a map then we set the center to the Google Zurich office
then we add a marker, and then, we associate to the marker an info window, okay? And the
issue that we are talking about actually involves exactly this operation here. So basically
the problem is that for annulled version of the Google Maps API, this method was just
not working. So what--what indeed was working was to--and this is now I'm showing you the
new code that has been loaded after we try to find the workaround. As you can see, it's
exactly the same but hidden here, the bindInfoWindowHtml has been substituted by bind operation that
is equivalent, that is associate to a marker the function that open an InfoWindowHtml and
whenever the click event occurs. Okay, so, this was easy because we manage to do it the
very first time but we'll see that it doesn't always happen. So now let's go on to the second
case study. Again, we have another--another Google Map. In this case, I cannot drag the
map, so it's fixed. I have instead, a fixed marker and if I click on it, I see again where
we are, that is the Google office. And then I also have another marker that can be moved,
okay? By clicking on this marker, I should again send them an info window, but, again,
I don't see it. So, this is another issue that is known in Google Maps API. It's the
issue number 33. And it's pretty annoying because it's one of the oldest but it's still
open, although, there is a workaround. So, in this case, the user tries to grasp for
a workaround and we see that it doesn't work. We'll see--now what is the first workaround
that has been tried and maybe I should first show you the original code, it's this one.
As you can see that in the original page there is nothing but the creation of the map then
the creation of the marker that can be dragged. And if you notice this, we associate to the
marker the info window by using the "addListener" operation. And then, we do the same for the
other marker. So, if you remember the previous case, we use exactly and now I'm switching
to the new JavaScript code that has been executed now that still doesn't work. As you see, the
first workaround that has been tried is to substitute the "addListener" with the bindInfoWindowHtml.
Why? Because it worked before so it can work also the other way, but in this case, it didn't.
So, now we go on. Try to ask for another workaround and, I mean, we broke the application even
more because now we don't even see the markers. Now, again, I want you to see the code of--the
new code that has been loaded basically in this case is to add a "setTimeout" here while
disabling the dragging. And let me go, yes, so but still it doesn't work as you can see.
Now, we go again and finally, I can tell you already that this is finally successful, okay?
And I can see that, I can still click on the fix marker and still the application is working
and the map is still not draggable. So, now, I'll show you the valid workaround that is
basically doing "setTimeout" when you call the "openInfoWindowHtml" function. And so,
what is the point in here? Well, in the first case study, we successfully could find the
workaround that the very first time this time it took us quite a lot of attempts before
finding the valid workaround. So, as Mauro said, so actually, since we found the workaround
I should actually save to the system that I found it. And so now that I found the workaround,
what happens if another user is asking for the same page? Well, I'll simulate this by
opening another browser and by calling again the same page. As you can see in here that
the user still experience the problem because by clicking on the marker, nothing happens.
But as soon as it finds, he asked for a workaround then the workaround that was accessible for
the other user is the first one that has tried. So, this, yeah, that's pretty much conclude
the demo. So, I'll give the work back to Mauro such that he can conclude.
>> Okay. So, a couple of words for concluding and I think you'll ask me questions hopefully.
I want you to sort of start from the beginning and tell you what we really tried to explain
here. First of all, a completely new way of looking at the problem. You're not trying
to solve the testing probably by itself, although we have a little edge on that. But we're trying
to actually look at different perspective, which does not substitute but complement this.
We cannot get rid of testing, we do not--we do need the good testing. What we're trying
to do is we're trying to get rid of some of the annoyance that comes the way after this.
You know, when you actually get system despite good testing, despite good maintenance still
breaking into a real work and the other thing is this is not a magic door, so it doesn't
solve every problem. You know, in fact we actually focus only on the problem that can
be solved through automatic workarounds and through workarounds and some of them cannot
be solved automatically though. So, in general, the goal is not to or--actually the goal could
be to solve all problems regardless what they are but, you know, let's say our, let's--
shorter term, goal will be actually to solve some of the problems and we know, if we really
believe you could get rid of 10-15%, 20% of the user reports we save a lot of money. We
save a lot of effort and we put that effort in many other different things. In the other
hand, we noticed is that and you've seen that the reason--sort of set of information that
accumulates and may be useful for developers itself. So, in some cases, and we mentioned
like the some boxes which is still open and we actually did-- we didn't show that we don't
have time but we are starting this new experiment which is get to the repository of workarounds
we have now, apply it to open problems, and try to see where it works and it did worked
actually indeed in a couple of record cases. Of course, you know, it's not that frequent,
you know, because, first of all, because it takes a lot of time for each experiment and
again, I don't have that many students to--be able to run too many experiment in part but,
you know, it is actually interesting to see that sometimes you get an open problem that
probably simply do not attract the attention of anyone in Google yet. And it can be solved.
Actually, maybe not. It not the solution it's basically like itself of a patch that you
put on the problem to let the user work better without actually interrupting their flow of
operation. As I mentioned before, at the beginning, it's an open project there, we have--we're
putting a lot of effort in it. We get to the first of gooder sides out of it, we believe
it is applicable more extensive within just where the application and they're actually
applying it to other kinds of applications, we have some problems although with our classic
application because there are two elements that actually I didn't mentioned specially
about but are actually very important. In this kind of application, the user can actually
report to the problem, you know, in another system, reporting the failure is not easy
and the other problem that we have is you can apply the workaround as many times as
you want and the page reloads completely and we're looking into the page reloads completely
the page, you know, you don't want to be in trouble with the database where you are actually
trying to insert as many information as you want, to try to solve the problem, so, you
have the problem of the stability of the state. So, it's not that generalizable. We see big
improvement in being able to generate as--the equivalent sequences, I will show the summary,
and I think we can do that. We--one thing that's actually important, we really need
more and more of that and more and more experiments. Now, let me spend one word on vector and to
conclude really, I want to say Google Maps is not just suited for this audience. It's
also important for us because of specific advantage that you find only on Google Maps,
which is we do have a very well document that set of a box and we can actually reproduce
the box, which is actually very important for us. And we are thankful to Google for
this because I think that's maybe without even knowing it, that was a big help to our
work and my last message is if you have problems, if you have that repositories, you may not
want to disclose everything to us but just shoot the problem, we'll try and we may tell
you whether the system works and that would be a very big help for us. Thank you.
>> Hi. So, from my experience, developers don't like testing, developers don't like
debugging. The developers like progressing and like to release and have more progress.
Do you worry all the--well, all of this time, ultimate workaround is useful and get to the
user--it discourages the developer from actually fixing the root cause of the bugs by making
it less of an issue for them. >> Oh, that's actually a very interesting
question because we did have a much larger project at some point which was about self
feedings. So, this is part of it actually which is not just trying to find a workaround
but just to fix the whole system and we're working on a European project with the company
developing software for the Onyx on the aerospace and they said, "We don't want this stuff because
we don't really want our engineers thinking that they can develop buggy software just
because they can automatically fix it." But that's a human factor, moreover, you know.
I think yes, you're right. I think--let me say one more word, I think that's the way
I teach actually. Software engineering is first of all an human intellectually intensive
activity. Brain intensive activity. And anytime, you must believe, you must think that the
human is intelligent, you cannot get the human out of the room. Of course if you look at
this as--oh, yeah, I can shift a buggy software because anyhow somebody has to take care of
that, that's going to be a disaster. But if you actually put this in place, in parallel
what we have now, and what is actually improving now, that's going to be much, much better.
Thanks. >> And--if I can add something, from our experience
with all the bugs that we studied on Google Maps basically all except one that already
has a workaround, knowing all the bugs have been fixed anyways. So, I think also the ability
to identify a workaround automatically can also help the developer to find a better fix
because for a workaround that we don't know why it works can be a useful information for
the developer because the developers for sure has more knowledge about the code, so.
>> How does the system know what is the problem? In the demo, we don't see how to use the report
a problem and the second question is how--if you have some equivalent sequences for a particular
problem in your repository, how do you identify--how do you link it with the new problem? What's
the criteria, and who decide that these two problems are actually similar or the same?
>> Okay, the first one is very simple. The user has this little button on the top that
says--I tried to find a workaround eventually--probably, you know, in the demo, you've seen this information
on the topic that was used for the demo, you know, just like saying, "Wrong page, try again,"
or something like that, you know, because I understand that, you know, the classic user
would not say, "Find an automatic workaround," and say, "Wrong page, give me another one,"
you know. And that's exactly--that's a very good point actually. They mentioned that at
the very end it's a very good point because actually that's the major--one of the major
problems in applying these techniques outside the Web application environment because we
really need someone who tells us, "These fail." So, in this case, the user clicks and say,
"Page failure," and that's when we trigger the application. We tried to use it like for
Java Programs and the Java will run it and it doesn't tell you it failed. And then, it's
going to be more difficult >> For the same page, the problem can be that
the map is the map is the wrong map or the scale is different.
>> Oh, yes. Oh, yes. You know, okay. So, the only information we have is, I'm not going
back to the first question is that some point the user sees something that does not satisfy.
It clicks – new page please or try to fix it please, wrong page, we fix it. That's actually
it. The second question is if you look at the different workarounds that we had, okay,
there is a constant--we did--actually we do have a formal evasion of the workarounds that
actually tells you how to apply them. You can see that for example, the first one says,
"If you have an open info window, you may actually set the Timeout. Now, before now,
if you don't have any open info window, it doesn't apply. It applies only to pages with
an open info window. And another one says, you can actually--instead of I do varlet,
you can switch between other of the varlet and that show or remove a varlet and hide
okay? Again, if you don't have any of these occurrences in the code, it doesn't really
help you. So, it's basically by context. And going back to the question, that actually
where I say it's not magic. It doesn't solve the problem all the times and may solve the
wrong the problem, in fact, you've seen, you know, in the second example, it tries seven
times before actually get into the problem. The question is not, you know, so, let's say
you should not see these as an alternative to a full tolerance system on your favorite
aircraft, you know, don't take that, actually, you know. It's more like, you know, I'm--
I really wish I knew what's in that position and they can't get to the info window, okay,
can you try yes once more? Can you try once more, I've been mainly, you know, in fact
in many cases we've failed completely and, you know, the user tries two to three times
to say, "Look, I don't care anymore what's on this info window." So, you can think of
that not just like the absolute solution. So, for example, if the problem is that he
wants to drag the page and it doesn't drag, okay, of course, then, you know, there maybe
actually workaround at the price of that. So, in fact, what Alessandra showed you are
two things was failure, click so that you only fold, try to see whether it solved the
problem, you've seen, for example, that just in some cases, the two marker disappears,
"Gee, no, it's not what I want." It may actually have been what you wanted you know but it
was not so--we don't--the system does not know what's right, it does not even know what
is wrong. It simply knows that there are some elements that can be actually fixed. And fixing
maybe wrong. So, but the point is in one third of the case, it has been correct and that's
actually what it incorrectly says. >> Thanks, and this also sort of ends the
second question. >> Yeah, thanks.
>> Because, the user has to click something, have you--thoughts about trying this with
real users and see how many times they would actually click, fix this automatically, is
it--your average user if something takes more than a second to load is going to get it right
so, that I want to meet another website because I'm bored of this. They can and publicly fixes
automatically once if they're going to click it at all and then made wrong. Have you thought
on how you're going to tackle that option? >> Well, you mentioned two kinds of users,
the real users and the average users. Our users are students so they are real but not
average. So, no, they – I need to be honest no, the prototype has been in the early stages
we do actually work on prototypes, we do actually work with students--we cannot afford to work
with real users yet. You now, if there's average users so and of course, you know, if I tell
students, "Can you try and see how many times it works?" So, they would try, and try, and
try, and try, they would not stop. So, this, again, they see many factors of this psychological
factor that say, "How many times you tried before giving up, you know, I can't report
you, you know, and you probably know them very well, you know." many users do not go
through beyond the second page, the second click, do not--although, you know, the Google
is encouraging--I think is this one, which is in many cases, the first time you click
you get the right page. So, assume you open your, let's say, they invest you and you've
gone off the work page and you open the Google Map and, you know, there is a really nice
marker and you want to get information and it doesn't get anything and you really want
to rip it through and to come to the United States and to gather some--you click again
and you see that you got the information and I think once is worthwhile but no, we didn't
actually--it makes sense then..