Imaging the Antikythera Mechanism

>>
TINGWEI: Welcome to Google TechTalks. My name is Donald Tingwei and today, we have a special
guest talking to us. His name is Tom Malzbender. He's from Hewlett-Packard Laboratories just
down the street in Palo Alto. And Tom is a senior researcher from HP Labs. He has wrote
many things at HP Labs including one of the earliest tablet devices and some brainy cognitive
kinds of thing before moving into computer graphics, I guess about 23 years ago. And
he's done things like volumetric rendering and Polynomial Texture Maps and things like
that. And today, he's going to talk about one of his really exciting applications of
his techniques about The Antikythera Mechanism. And so, this is an intersection of computer
vision, computer graphics in archeology. And take it away, Tom.
>> MALZBENDER: Great. Thanks, Donald. So, I guess I want to start off and say that if
you have questions during the talk, feel free to ask, and the best thing would be to go
up to the microphone and ask a question. So, I don't have to repeat all the questions but,
that would be ideal. So, basically, in 2001, a fellow HP Labs researcher and I developed
a method called Polynomial Texture Mapping and it's an interesting technique and it actually
allows you to see more detail on the surface of objects and you can typically see even
holding them in your hand. And, in 2005, we got sent to Greece to apply the technique
to a mechanism called an Antikythera Mecahnism and this talk will cover both the imaging
method itself and the Antikythera Mechanism. And then, at the end of the talk, if there's
time left, I'll go through just some more recent developments that we have on the imaging
method over the last 10 years or so. So, it's well-known that as you change lighting direction,
you can see more or less detail on the surface of the object. This is something that we use
all the time. We hold objects relative to a light source and move them around. But what's
not so well understood is that if you actually change the material properties of the object
itself, you can often see more detail than is possible with the original material properties
of the object. So, the technique I've developed in 2001, at that time, they were two main
ways of playing textures to the surface of objects through computer graphics and we did
developed--this is a method for computer graphics. Although, nowadays, it gets used more in fields
like archeology and forensics, then 3D graphics, but, you know, one of those techniques is
texture wrapping that got developed in the mid '70s and texture mapping you've all seen,
when you play a videogame, the characters on it are all texture mapped, meaning, an
image has been applied to the surface of them to give you a feel that there's more of complexity
there than the geometrically modeled. A great technique, the problem with it is you change
lighting interactively on texture maps. The lighting doesn't do the right thing to represent
the underlying geometry. So, Jim Blinn addressed that problem in the late '70s by technique
called bump mapping, and what you do with that is you keep around surface normal for
every pixel that you have in the texture map and then you can use relighting techniques
to use that normal to produce an image or what the surface would look like under new
lights source directions. That also works very well. The problem with it though is that
it's not image based. Typically, you have artists throughout this bump maps and then,
you know, they're applied onto the surface of objects. So, it's not it's not a photographic
technique. The results are typically not photorealistic. So, we developed a polynomial texture maps
which are--well, I'll just show you one, let's just jump right into it. This is a PTM or
polynomial texture map here and what we do--what we have is control interactive control of
lighting here and the way we do this is for every pixel, we keep around a reflectance
function so that's what's shown on the right here. And so every pixel, you know, has this
simple two-dimensional function associated with it. And you can change lighting direction
by probing in the circular in the right and you can look at the what the reflection functions
are that are modeling on the–by probing the image itself for every pixel. So, this
is done independently for every pixel. So, that's a PTM. How do you collect these things?
Well, here are two objects that we've used. The first one was the first thing I've built
basically in my garage, at home, it's just a dell, a wooden dell assembly with [INDISTINCT]
glue and you simply put a digital camera on a tripod on the top of it looking down on
the floor of this thing. And put an object on the floor and move a table lamp to each
face of this icosahedron and take, in that case 40 pictures of the same thing under different
lighting directions; pretty simple technique. And on the right here is just a more elaborate
version that we built a year or two later that has computer control light sources and
so it plugs into a laptop and, you know, with one stroke of a key, you can collect all,
in this case, 50 images of the same thing under different lighting direction. So, no
matter what kind of hardware you use to collect it, what you have is a stack of images where
you've got a measurement of, for each pixel, you have a measurement of the color as a function
of lighting direction. And then, we just fit biquadratic polynomial to that description
of values. So, specifically, what we do is we fit a biquadratic polynomial to the luminance
to the, you know, brightness basically of that pixel and you'll see that these LUs and
LVs, those are just points in that circular space I was moving around before. They're
just projections of the lighting vector onto the plain of the texture map. So, it's just
a two-dimensional parameterization of the light source direction. And these A0 through
A5s are what we keep around in the texture map that coefficient, in addition to the on-scale
color values. So, the polynomial is used to compute the luminance that gets plugged into
this set of equations here where we take our on-scaled RGB values and just multiply them
by those luminance values so that we can compute it to recover what the real RGB values are.
And that's how we do the renderings those are the RGB values that are drawn. Now, one
thing to point out about this equation is that it's really simple. You know, it consist
only of multiplies and adds and therefore, you can use parallel sub-words instructions
in the CPU to evaluate this very quickly. So, the PTM that I'll be showing here don't
use any graphics hardware support. They just run on the CPU directly and, you know, usually
give you real time performance out. So, I should back that a little bit--yeah.
>> Yeah. >> So, you use quadric polynomial here but
when you have shadows, these are step functions as if function of light. Did somebody, you
know, suddenly falls into the shadow of some other part of your scene, you know, how does
the quadratic deal with that? >> MALZBENDER: Yeah, so that's right. So,
you know, for representing sharp details in lighting space, you know, it's not analytical
high degree enough polynomial to represent very sharp edges. So, once it happens when
do apply these fitting functions is that we have smoothing in lighting space, not an image
space because this is done per pixel independently. But it's really equivalent to having lit the
object with fairly large area light sources instead of point lights. So, that's right.
So, you have this low passing operation because you're taking 50 data points and you're reducing
it to five coefficients. So, the four methods that I described before is just one of the
formats that we have and that is a polynomial description of the luminance and then fixed
RGB values. We've also done this with polynomial three RGB component separately and in color
representations where we apply compression techniques look up tables or JPEG-LS to these
representations and they can get quite compact. Okay, so, that's a PTM. What it turns out--what
the interesting part of this really is though is that you can extract all these information
from PTMs that's interesting. So, if you have surfaces at some orientation, if you have
a digital camera that's looking down with that surface, and you ask the question, where
is the light source need to go to maximize the brightness of that pixel? The answer is,
for diffused object: perpendicular to the surface orientation. So, if you find out--if
you find the maximum of the surface orientation, which you can solve for it analytically, you
can come up with SNE--pretty good SNE for the surface normal in this case. And again,
you can drive that analytically from the function itself. So, it's very quickly. So, in the
course of a half a second, we can compute all the surface normals for a PTM. Now, let
me show you how we us that. We basically have two enhancement methods, both of which use
that surface normal, and this is showing one of them. So, let me show it off on this 4,000
year old Cuneiform tablet and it certainly helps to build a better lighting to be able
to see what's on the surface of that. But, it's even more helpful to use those surface
normal for display. So, if I turn this on, you can now see this red cross here--across
here, that just corresponds the maximum of the PTM for any pixel. Well, if you use that
to now synthesized synthetic specular highlights, you've now change the material properties
of the object. So, we've got a slighter here that controls the amount of diffused coefficient
of the object. So, I've just turned that all the way down. This slider here controls the
degree of specular reflection off the surface, I'll leave up high. And then this slider,
just controls the specular exponents. In other words, how wide or how narrow these specular
highlights are of this object. And so when you do this, it turns out, you can now see
a whole bunch of detail that was difficult to see before. So, specifically, look at the
two calms of text in the middle here. I go back to the original rendering, it's you know,
it's difficult to perceive those. And while I have this up, if you take a look at some
of the grooves that occurred here, on the top here, those turned out to be the fingerprints
that described 4,000 years ago that was holding this tablet when the tablet was still wet
and that was not--that was not known to be on the surface of this tablet at the time
we photographed it. So, that's specular enhancement. That's probably our most powerful technique
for bringing out more surface detail. Another one is very useful is something we called
diffuse gain. And in this case, there's less of a physical analogue to what this is doing.
There's really no--there's really no physics in real material objects that does this sort
of thing. But, if you look at--if you probe the reflectance functions of any object that's
diffused, they're very slowly varying. You know, the brightest of an object doesn't change
much as a function of lighting direction. Well, it turns out we can leave the estimate
of the surface normal the same but increase the curvature of the second derivative of
the reflectance function. And when you do that, you can basically increase contrast
quite a bit in a geometric sort of way. So, we have a slider here that is tied to this,
the amount of gain that we have on the curvature on this second derivative here. And, we can,
you know, we can see more or less detail on the surface of this object, in these inscriptions
here then you can typically see. So, this is–this object is a 3,000-year old object,
Egyptian artifact called a shabti and it was typically wrapped up with mummies to help
them in the afterlife. Yeah, thanks though. Okay. So, let's--all I want to say about the
imaging technique at this point. So, I am a member of HP Lab. I'm also a member of what's
known as The Antikythera Mechanism Research Project and that consist of basically four
groups of people. First, you've got largely British astronomers. You've got Greek epigraphers,
people that study ancient writing. And then, you've got a team of people from a company
called Extech, which is a microfocus CT imaging company in the UK. And you have a couple of
us from HP Labs that have done our surface imaging work. And if you'd say right off to
bat, this group was pulled together by these two gentlemen, Tony Freeth and Mike Edmunds
here and a lot of the breakthroughs that you're really going to hear about here were done
by Tony Freeth and he also published the--or wrote this site of American article that appeared
in December. If you want more information on this, we're free to that. So, the story
starts with a shipwreck that occurred off the Island of Antikythera which is right next
to the island of Kythera and it was a Roman boat that was carrying loot pillage from perhaps
at the Corinth area, perhaps the roads area. It's really not exactly known where the Antikythera
Mechanism originated from, it's a matter of dispute. But any case, the ship did get close
to this island of Antikythera. It hit a storm and it was sunk and this happened in the first
century of B.C. and the boat was underwater for roughly 2,000 years. And in 1900, another
group of--another boat appeared on the scene, a group of sponge divers who were also hiding
out from a similar storm. And when they went--when they woke the next morning and dove down to
look for sponges in that area, they found what they thought were limbs all over the
surface of the ocean floor. And what these were, were life-size and larger than life-size
marble and bronze statues of--from the ship--from the shipwreck. And, consequently, the Greek
government recovered as much of the treasures they got from this wreck and much of it now
resides in the National Archaeological Museum in Athens. There was statues, there was glassware
that was found, and then there was also a lump of clay that was brought up in a bucket
and basically left in the courtyard at the museum to dry. And when it dried out, it split
open and revealed gears, bronze gears in the inside of this lump of clay and it was at
that point they knew they had something pretty interesting here. This is a microfocus CT
study of the largest fragment in the mechanism fragment A and you can see the numerous gears
here. There are 30 million gears in the mechanism. They're probably were more than that, but
in this one fragment alone, there are 27 gears that you can see with CT, so a very complex
device. At the time it was known that the Greeks used gears but very primitively for
mechanical things like turning a water wheel. The fact that they did calculations like this,
as-–which turn out to be astronomical calculations, it was definitely not understood at the time
of this discovery. So just to give you a feel for the size of these, the scale of these,
of the artifact, these are the three fragments that are--appear in the National Archaeological
Museum in Athens. But they are not just three or four, there are actually 82 of these fragments.
And so, in 2005, we went over to Greece and spent a week applying our imaging technique
to the front and back sides of all the artifacts. We took four and a half thousand photographs
during that week and we produced 82 separate PTMs of all the surfaces and these are all
publicly available now on, in HP Labs website that anyone can get access to, anyone has
access to as public and it has been downloaded often by people studying the Antikythera Mechanism.
So it was appreciated in-–by German scientist in 1905 that this was a-–some sort of an
astronomical calculator that was found. But real breakthroughs were not made until Derek
de Solla Price came on the scene and he actually wrote a scientific American article in the
year I was born, in 1959 on the mechanism, that's also very informative. And one of the
most important things Price did was he involved a Greek radiologist, physicist who took x-rays
of the mechanism and clearly you can see, you know, the complexity of the gear assemblies
in it from the s-rays. What you can also see, is that one of the gear teeth had-–gear
wheels had a 127 teeth on it which is kind of an odd number to put a gear. And so, Price,
you know, correctly determine that this was-–it had something to do with the Metonic cycle
which has twice that number of months, sidereal months in the–-in its cycle. And that this
was in fact, some sort of astronomical computer. And Price hypothesized a model of the mechanism
which was right overall. It was closely correct but has details that are incorrect on particularly
on all the gear trains that appear in it. So there, as I mentioned there are two groups
that were sent to do–-apply imaging techniques to the mechanism. The first was this company,
Extech that does microfocus CT work, CAT scan work. They used energy levels that are much
higher than what you're capable of using for medical applications and so they can get very
precise fine images out from this technique. Unfortunately, the museum did not allow us
to remove any of the artifacts from the museum including Antikythera Mechanisms, so we had
to bring all the equipment into the–-into Athens, into the National Archaeological Museum
which wasn't too bad for us but for these guys, it was tricky. This is a 12-ton CT machine
and they literally shut down the streets of Athens for a few hours to bring this into
the museum, so it was brought in on truck into Athens. So let me just show you one more
data set. The sequence we saw earlier was also from their CT data. This is just a cut
away, cutting through the mechanism; just take a look at the complexity of the gears
that you see here. And by the way, none of the gears are intact. It's not just a simple
matter of counting the gear teeth so it's not watching how many teeth each gear had
is a tricky process on its own. Okay and we were the other group that got sent down there.
This is the PTM assembly that we took down there, imaging fragment C. So it's just a
Nikon D70 camera and 50 light sources that were programmable. So as I mentioned already,
we put all our data on the web at the full resolution. Let me show you some of the things
that we captured here and I guess before I pulled it up, I should say that this one fragment
actually it tells you enough to take the mechanism roughly itself. But, for dating it was also
helpful that they found coins on the surface–-on the ship itself that date back from 86 to
60 BC. They rail to radio carbon date the timbers, the mass of the ship itself, that's
from–-so the ship was from about 200 BC. But let me show this fragment in all bit of
detail here. So, again, helpful, it's a very lying direction to study it but, when you
turn on the specular enhancement technique; it's really quite easy to see the writing
on the surface of it. So the first thing that jumps out on you–-at you, is that, most
of the characters were on together. Well, it turns out that the ancient Greeks didn't
use spaces except the demarcating numbers. So it's pretty clear to see that this is a
number that you have to be both, both be astronomical numbers. You can also date it from the writing
itself. So if you'll notice that this sigma here is-–the top and bottom parts are somewhat
splayed out. That's indicative of the second century BC writing style, as is the fact that,
on the spy down here, this leg of the pie is slightly shorter than this leg of the pie
that's, again, indicative of second century B.C. writing. So this one little fragment
that's--about an intra-cross was a–-is quite helpful in dating the device. Here is just
a block of some of the numbers that occur. These are all astronomical cycles that occur,
that the mechanism describes in all of–-I got to slightly to define all these in a minute
but basically there are 76 years or something, notice the Callippic cycle, there's 19 years
in the Metonic cycle, and 323 months, sidereal months in a Saros cycle. Okay, so certainly
one of the things that we did was, allow a new modality for people to see more detail
on the surface of the object. But, actually, another contribution was just providing high
resolution photographs that they think the scholars. This turns out to be the best photograph
that was publicly available before our work--the scholars that were studying the mechanism
of that same fragment. So it was just good to update it all. This is one more example
of a fragment of the mechanism. We zoom into a little part of that and you can certainly
see a moving light source around that you've got some detail on the surface of that. But,
again, it turn out one of the enhancement techniques and that detail becomes pretty
obvious that you've got some ancient writing all over the surface of this thing, much of
this have been decoded now. For those of you that can read ancient Greeks, this is actually
upside down so I apologize for that.
But we basically we're able to go from being able to read--well, in combination with the
CT work, we're able to go from reading about 800 characters to being able to read about
3,000 characters and it turns out that jump really does contribute a lot to the understanding
of what this thing was and how it worked. So this is what the mechanism probably looked
like; the front and back side of it. It had a large dial end pointer on the front that
indicated basically the zodiac, it's a calendar dial. There's both an Egyptian and a Greek
calendar on the surface of that. And on the back side is what is known as the Metonic
dial and that the Saros dial. These are both--first of all, it's the first known scientific instrument
in history. These are the first dials, graduated dials that occur in any artifact. And, basically,
what these two in the back demonstrate is this is showing you the Metonic cycle, which
it has to do with the correspondence between the Earth's--the moon's rotation and the Earth's
rotation around the sun. And, this is showing what's known as the Saros cycle which is useful
for predicting eclipses both Solar and Lunar eclipses, hundreds of years out. The data
that this seems to be based on is probably about 500 years worth of astronomical observations
that the Babylonians actually made before the Greeks. So how do we know that it's hand-cranked?
We don't strictly know that it's hand-cranked. It could have been driven by something like
a water wheel but, given that it's a--of quite small compact object, it probably have little
crank on the side of it. This is just, you know, an axle that connects to the main drive
wheel that it has an obvious little slot that looks like--it sure looks like you would stick
a handle into that and crank it, so that's the evidence for that. This is an animation
I want to show here that Tony Freeth produced. It is of the complete mechanism as it's understood
now. All 29 of the gears were involved in this animation, and you can see the--just
the remarkable complexity, this thing. One of the things I want to point out real quickly
is this little black--what looks black right now is actually little sphere that's both
black and white and as it rotates around, it's telling you and showing you the phase
of the moon. So it's white, it's a full moon, when it's black it's a new moon. So this is
the mechanism as it operated without the boxer around the side of it, you know, to cut away.
And this is, how Tony and our group in general, believes how the mechanism worked and how
did the gears went together. I'm not going to any detail on this at all, but you can
see that the, you know, the hand-cracked here input drives both the front dials which are
the calendar dial again and the back which are both, you know, the little solar calendar
and the eclipse prediction the Saros cycle. Okay, so let me define some of these terms
I've been using. So what you think of as a full moon to a full moon cycle. It takes 29
and a half days, that's called a Synodic month. Now if you look at that from the references
of the sun instead of the Earth, you know, obviously, the Earth is rotating around the
sun, as this is always happening. So the--in the reference system of the sun, for the moon
to get to get to the same position, that's called the Sidereal month. It's so much shorter,
27 and a half days. The Metonic cycle--well, we know that the orbit of the moon is not
face-locked with the orbit of the sun--sorry, the orbit of the Earth around the sun. But
they do coincide roughly every 19 years ago, every 235 Synodic months and that's called
the Metonic cycle and that is--it's displayed by the mechanism. If you add one day to every
fourth year of that, you get a cycle now that's 76 years long, that's called the Callippic
cycle, that's also shown, likely it was shown on the, this surface of the mechanism. The
Saros cycle is interesting. So it was known that eclipses can reoccur every 223 Synodic
months and that cycle is called the Saros cycle and is definitely one of the more complex
things that the mechanism shows. Again, if you--now, that repeat of eclipses turns out
to be eight hours off and so you have these eclipses occurring at different parts of the
planet with the Saros cycle. So if you repeat the cycle three times, you get what's known
as the Exeligmos cycle which is more accurate and in that it's possible that an eclipse
can happen--that happen in one place on the Earth will occur at a similar spot on the
Earth, you know, 54 years later. And then, the last thing that the Antikythera Mechanism
shows is what's called the first Lunar anomaly and that is the--the orbit of the moon is
not perfectly circular. It's somewhat elliptical, and so the moon speeds up and slows down,
that's called the first Lunar anomaly and believe it or not this mechanism even predicts
that or even demonstrates that anomaly. Okay, so let me give you the three examples of how
the imaging was used to uncover some of the details of the mechanism here. So if you look
at the largest fragment, fragment A, and the back side of it, on the right side there,
you'll see very faint markings, and if I highlight those, you can see where they are and if they're
spaced uniformly. What you can also see is that every fifth or sixth spacing apart, either
one, five, or six spacing apart, there are--there is writing that you can see. These little
glyphs and if you apply the PTM techniques to that, you can certainly pull out some of
that writing and it quite have been more visible. Now, this is showing a spiral where all the
known locations of these markings occur and if you extrapolate and basically put them
back into the four cycles of the dial and both of Metonic and the Saros dials were spiral,
they were not circular. You wind up with 223 months as again we know is the number of months
in a Saros cycle. And if you now look at the location of the known glyphs that are on that,
they are located here. And we can certainly pull all those off into one view, so this
is the 16 known glyphs that are visible. You know some of these that are little easier
to write appear on the surface and are pulled up by PTMs. Some of the other once that are
problematic or, you know, you can only see with CTE renderings. They're not quite as--quite
as crisp, but you can highlight some of the text that occurs on this and what you'll see
repeating over and over are Hs. We're sure not to stand for Helios, ancient Greek for
Sun. And the Sigmas are a shorthand for Selene or Selene which is a word for the Moon. And
so this is referring to both Lunar and Solar eclipses. And the rest of the writing below
actually tells you the hour and the day that these glyphs could possibly occur at. So if
you now go back and you place the known glyphs, you can make predictions about where the remaining
glyphs must have occurred on the surface of that dial. Okay, the second thing I want to
point out was something known as the pointer-follower. And as I mentioned already, the dials in the
back are spiral in shape and so it's not clear from just a pointer itself, which of the four
or five arms of the spiral, you should be reading off when you use this mechanism. We'll
it turns out we found, you know, here's an arm of a pointer and here's a little knob
that sticks down in the CTE renderings and this is a rendering that Tony produced of,
you know, what it probably looked like new. And now there's this thing is a pointer-follower.
So the sleeve is free to slide on the surface of this pointer and this pointer-follower
rides in little grooves that are in the underside of the mechanism and let me just show you
an example of that rendering of that, again Tony Freeth produced. You know, as these dials
rotate around, the pointer follower slides out. It tells you which of the thickier four
or five rings to read out at any point in time. And presumably when its gets to the
end of it's travel, you have to manually reset it back to he original but we really don't
know if that's true or not. Okay, so the last things I want to point, I find the--I find
just remarkable. Let me show that to you on a CTE or on a PTM renderings. Again, we're
looking at back side of fragment A and this time let's zoom into the area around these
two sets of gears here. And let me turn on the enhancement techniques again so you can
see the surface a little bit better. So, I want you to keep your eye on this little notch
down here. Let's take it out of the gear wheel and, you know, as you can see it's a little
cut out. And, at first, it was thought that this cut out had something to do with the
fact that may be the Antikythera Mechanism was repaired. But it was noticed that that
this, that in the cut out there's actually a slight pin, there's a tiny little circular
area which you can see better in the CTE renderings, so let me pull that up. Right here, here's
the same little notch and here's the pin that we're talking about. So it turns what this
was, was a little pin slide arrangement that hide two gears together that were sitting
on top of each other. And, this is an epicyclical arrangement meaning that, first of all, that
the access of these two gears were slightly offset and they were mounted on another gear
that was free to rotate. And basically what this did without going into the exact detail
on the mechanism was it predicted this first literal anomaly. It actually caused the gear
that's responsible for keeping track of the moon's procession to slow down and speed up
by exactly the amount that was known at that time or estimated at the time of that, you
know, epicyclical behavior. So this is, I mean, to me, this is one or more amazing parts
of this mechanism. These guys, you know, in around--this mechanism was probably built
around 150 BC. You know, the time when North Americans were living in Teepees, these guys
were building mechanical calculators using bronze gears, you know, hand cut of these
complexity that model the slow down and the speed up of the moon as it went around it's
orbit. It's just amazing. And there is nothing--there's nothing comparable and mechanical complexity
in any civilization until you get to about 1500 or 1600 when European start building
clocks. So this thing, you know it's 1500 years ahead of it's time, it clearly was so
complex that there was no way that this could have been the first device of it's kind that
was built. It's just there are too many mechanical subtleties to that, to allow that, but it's
the only one of its kind that's ever been recovered which is a bit surprising. Probably,
it's explained though by the fact that bronze throughout the years has been an extremely
valuable metal. And, things that were bronze got melted down to form cannon balls and,
you know, and canons typically. And the only reasons that the Antikythera Mechanism survives--it
was underwater for 2,000 years. So let me just show you an animation of that last what
was called Hipparcos Mechanism for some time. So here are the two pairs of gear teeth on
the back and you can see the little pin slot arrangement that ties the top gear onto the
bottom gear. Again, animation was produced by Tony Freeth. Okay, so that's really all
I want to say about the mechanism itself. I want go a little bit into some of the imaging
work that's been done based on our technique since then. And this first slide is just an
overview of some of the devices that have been built in the meantime. This is not--mostly
it's really not by our group. A couple of them were built by a non-profit called Cultural
Heritage Imaging, and the basic idea is that, you know, you just want to get up--a collection
of photographs from different laying directions of the same thing under, you know, under [INDISTINCT]
laying directions. So let me just show you a couple of those devices. This is, again,
it was built by Mark Mudge, Cultural Heritage Imaging. A very simple device, you just got
a digital camera on the top of the tripod here. He's got a stand that's holding a light
source and he's just moving that light source to different locations. Those locations are
marked on a template--a piece of paper here that's got, you know, markings of how high
you should have the light and exactly where it should be. So he's just manually going
through a collective dozen images of the same thing. This is also a very low cost way to
approach it. This is a PTM assembly that was put together for a couple of hundred dollars
by Walter Verhesen in the Netherlands. This is just a low end digital camera with an extension
on the flash and then he bought a $3 starter from Dome and drilled holes in the right places
and just moves the flash assembly itself into the holes one at a time and it takes a number
of pictures, so very simple. And then on a high end you have devices like this is that,
again, was built by a Cultural Heritage Imaging that have a very carefully controlled--color
controlled light sources that are brought in with optic fibers onto the surface of this.
With these kinds of devices you can make assurances of it. You're not exposing any of the museum's
valuable artifacts to excessive amounts of radiation. You can carefully quantify exactly
how much radiation you're applying, you know, with the frequency just [INDISTINCT] and so
on. So this has been picked up by a couple of dozens groups around the world. And the
reason is that all of our tools are pretty easy to use. They are on the web, anybody
can download them. There's no licensing or anything like that. So basically, what you
do is you create a stack of images and then you create with text editor which called an
LP file which just tells you how many images you have in the data set, [INDISTINCT] images,
and what the light source--vector as it normalize light source direction, so what direction
of lighting came from? That's it. And then you feed that into what we call the PTM feeder
and it spits out a PTM for you. And I should just make mention of this figure right here.
It shows you 50 of the original samples of luminance and then the function that was eventually
fit to that. And you can that will low pass again in this lighting space typically preserving
details quite well even in wide space. By the way the feeders are available also for
download and so you can just produce your own PTMs. So, lots of people have done, lot's
of things with it. We had an initial case I just want to mention with the FBI and actual
serial murder investigation. The FBI will prefer not show the original data publicly
so this is kind of a made up version. But basically what we had was a serial murder
was keeping very detailed notes on what he was doing in a spiral bound notebook. He unfortunately
had managed to rip those out before they arrested him. And what they had--what the FBI had were
very faint indentations on blank pieces of papers underneath those, those papers and
we were able to bring out the indented writing and this would shows another example of indented
writing that I made up, it turns out if you use this diffuse gaining technique, you can
really increase the contrast of the stuff and eventually start seeing that you've got
some indentations here. And even with our method you have to, by the way, this is the
quick brown fox jumped over the lazy dog. Even with our method, you need to interactively
do this. Your visual system turns out to pull out much more information under emotion and
looking at different lighting directions. So for instance, you know, the J, let me see
if I can line this up accurately enough here that the J here on the letter jump is best
seen with lightning that's perpendicular to it. If you put the lighting overhead, you
really can't see it that well. If you put it off to the side--it's a little finicky.
You can dial it in here. Put it off to the side, you can really see the perpendicular
brush strokes. So basically you want to put the lights where it's perpendicular to the
structure you're trying to recover. Okay, so it's been used in another criminal investigation.
This is something that California Department of Justice put together. It's a rig that has
an arm that's free to rotate an additional camera on top and it's very useful for capturing
footprint PTM. So, let me show you another feature of the PTM viewer that I haven't shown
off yet. We can actually extrapolate the space beyond the lighting directions that we have
or that are physically acquirable in the first place. I mean it's basically like taking the
light source and gone up with here, moving at below the light source in some ways. So
we can place one light source there and we can certainly take another one, place it off
to some other gracing direction. We take a third light source and put it somewhere else.
And so, you know, eventually you can build up--wearing this letter, that are pretty indicative
of the 3D shape. And what's interesting here is that, you know, once you collect a PTM
like this, you can now perform this analysis much later or rise a new technique, it's got
developed you can try applying them to the device. They've also been used by the National
Gallery in London. This is an arm that we made very simple out of literally just bought
12 flashing, low cost flasher, it's from a camera store mounted them on an arm and, you
know, they're manually activated. Let me show you this PTM of a front hall spanning taken
from the National Gallery in London. So if we move the light source, you know, off to
the side, you can see the vertical brush strokes quite well; if you move it to the top and
see the horizontal brush strokes and certainly in all cases you can see the dust on the surface
of the painting quite well. Okay, so a few years ago this begged the question to me that--okay,
if you have control--interactive control of lighting like this, is there anything analytic
that you can say about the quality of the images that come out as a function of different
lighting? Well can you characterize in effect, you know, how complex the images are as a
function of lighting directions while there's certainly well known measure of information
content and images--image entropy that does correlate as you can see quite well with how
much you can read in lighting direction. That's not analytically derivable from the PTM equation
but fortunately variants which is for Gaussian random variables; it is monotonically related
to entropy. It's possible to compute that analytically from a PTM representation. Basically,
meaning that, once you have a PTM, and, you know, half a second, you can now produce maps
of variance or entropy that tell you where are good places in lighting space to look
at this object. So we're going to see more detail, we're going to see less. And not surprising,
every surface has a different representation of image entropy and different, you know,
region where it's better to look for detail, often it's in the grazing directions here.
Okay. So, this is just a quick slide to show that PTMs can be collected under any frequency
of light. This is using infra-red lighting. Some experiment said the National Gallery
did. So far, we haven't found a great use for IR PTMs, but it's possible to collect
them. This again was produced by Wouter Verhesen in the Netherlands who has interesting microscopy
and you can see, again, both dark field effect and light field effects off a single PTM.
This is the wing of a dragonfly at about 200x magnification. So, this could be applied at
very small scales. You can certainly apply it at very high scales as well. This is a
PTM that I put together of a significant part of Arizona, working with digital elevation
data from geologist name John Soll. John believes that, that a period in the Earth's time, about
three billion years ago, called the Late Heavy Bombardment where the Earth was--had catastrophic
large meteors hit the surface of it, formed very large impact craters. They can still
see evidence for it today, and basically, plate tectonics makes the prediction that
you shouldn't really be able to see this stuff. But, you know, sure enough, you can see circular
structures, like this one here that may be evidence for Late Heavy Bombardment timeframe.
But this is--what you're looking at is a significant chunk of Arizona here so, again, PTMs can
be done in all kinds of the spatial scales. I want to show you that we've now developed
an even easier way to collect these PTMs that requires nothing but additional camera, a
handheld flash, and a black snicker ball, which you can buy for five bucks. So what
you do is you put the black snicker ball into you scene next to the artifact that you want
to capture, and you take pictures moving your light source around to different locations,
and it turns out you can recover the direction of the light source from the reflection of
the black snicker ball, it makes a lot of sense. Well, it turns out--what's nice about
this is you can do this fully automatically. You can--there has been some software written
by a group from the University of Minho in Portugal, that you can just feed it, a directory
worth of images. It automatically finds the black ball. It automatically finds where the
highlights are in the black ball. It automatically extracts the light source direction from those
highlights. And it runs the PTM fitter to produce the PTML from your stock of images,
or hopefully without manual intervention. Now, if there, you know, if it fails and it
doesn't find the black ball, or, you know, automatically, you can go and help it. But
it's automatic, the whole process and made it pretty simple to collect this kind of data.
This is also downloadable from our website at HP Labs. And let me just show you some
you some renderings that were collected from--data collected in that fashion. This is some stone
carvings that dated from about 20 to 30,000 years ago of an antelope here. Here's the
head of the antelope, and the feet, and the body is right here. So, it's really helpful
to vary lighting to see that sort of detail. Okay. And we've also done this in real time.
So, one of the complaints we were getting from forensics and criminal investigators
was that, you know, this is a lot of hard work to take out into the field. It'd be nice
if they can just take things back to their lab and it quickly collect images based on
that. And so, we're just--we built a real-time assembly that consists of a high speed, 500
frames per second video camera here, that's some arms that capture the lighting information
or that [INDISTINCT] lighting. In practice, we use eight light sources and in a 60 of
a second, we can collect all eight images at different lighting, feed all that data
down to a GPU, compute service normals, compute reflectance transformations, and render the
thing which means that [INDISTINCT] video rates you can produce a renderings of things
that you hold in front of this assembly and start seeing the more detail than you typically
can. So, 2 o'clock is the end for this, right? Ten minutes, is that all right or it might--yeah,
I'll save some time. Yup. So, you can produce renderings in real time with this technique.
So, this is just a demonstration of some those results. Here's an original view of the surface
of a basketball. Here is low frequency content being brought out from the basketball, and
here's higher frequency content being displayed on, from the basketball. So it turns out you
can run these normal transformations in such a ways that they have frequency effects, special
frequency effects too so you can bring out different spatial frequency components. So
far, everything I've shown with PTMs has been covering variable lighting. It turns out they're
useful for other things besides this--so this is a just a simple example where I took six
pictures of boring office scene at different focus settings and integrated them into a
PTM. As you can see, we've got continuous interacting patrol of focus direction once
we've done that. So, this, to me, it kind of begs the question, you know, why would
you ever want to fix the folk's conditions at the time you take the pictures as opposed
to collecting data like this, and allow you to play with it after the fact? And the last
topic I'll mention here is, you know, everything I've shown you in terms of lighting with PTM
is from a specific viewpoint; one fixed viewpoint. You'd obviously like to be able to--from museum
applications, capture an object from various viewpoints, allow the user to rotate it around
and change the lighting in real time. So, we've taken some steps towards being able
to do that. Specifically, we've generated this PTM object movies which are quite simple,
you know, conceptually. They're just PTMs taken at different orientations of the object
and, you know, in this case, we still have interactive control of over lighting in real
time, plus in view dependence. Obviously, you would like this not just to occur at static
view locations, you'd like to have continuous control over the rotations. So, we've been
working with researchers at UC Santa Cruz to do just that, to collect--and it turns
out one of the biggest problem here is the amount of data the you have to collect as
you vary view direction and lighting direction is just enormous. And so, we've done a work
in terms of, you know, how you manage that trade off, where you take more lighting directions,
where you take more view directions. Good. And that's really all I had. I want to point
out that this is the URL for the website that we have for--where all the tools are. These
are, you know, it's very easy for, kind of hands on people like you all are to go ahead
and experiment with, so feel free to play around with these tools. And I just want to
thank Dan Gelb specifically developing the technique, and the others that were involved
in the research in the Antikythera Mechanism. Thank you.
>> TINGWEI: That's really cool stuff, Tom. And my favorite was the PTM and that global
scale in Arizona. That was pretty cool. So, I want to plant two things. Definitely, go
to check to his website. Just Google Tom and you'll find the PTM there, I'm sure. But,
interactively, you can change the lighting on some sample of PTMs he already has, and
it's a pretty cool technique. And then, if you have more interest in the Antikythera
Mechanism, check out the December issue of Scientific American. And so we have time for
a few questions, and please use the microphone. >> Would it make any difference if you were
to use coherent light? >> MALZBENDER: Not that I've been able to
figure out. No. I mean, you could use polarized light, you can use coherent light. I don't
really know how to introduce that as, you know, and increase the functionality with
PTM with that. I mean, we've certainly used varying frequency light sources, but I don't
know if--I'm not sure if coherence buys anything with this technique.
>> Have you know anyone else that creates the PTMs using the sun as a light source or
most around? >> MALZBENDER: Yeah. I may have looked at
that. I've taken objects and photographed them automatically through various points
of the day, and you can certainly make a nice, one-dimensional PTM out of that, and I could
probably pull up an example if you give me enough time of that. But, since the sun travels
in a one dimensional path, you don't have the full two dimensions of--that you need
to fully capture a PTM. So, you can't make a two dimensional PTM out of one, but yeah,
we have done one dimensional PTMs. They looked good. Kind of related to that too is--an obvious
thing to do is to take--is to make a PTM of the moon because mostly, the moon is facing
the same direction at the various phases. So, it'd be very simple, you just photograph
the moon every night and combine that into a PTM. Now, you got control over lighting
in the surface of the moon. Unfortunately, the problem with that is the moon actually
rocks back and forth, I guess, like five degrees or so, enough so that the, you know, the images
would not be registered well enough in 3D to make that happen. We would need to be compensated
for it which is all doable but, has not been done yet.
>> What about the color, the number of bits of color depth you have. Is there anything,
you know, is eight bits per, you know, RGB good enough for this technique or do you need
to do anything special on sort of the type of camera that you have and the depth?
>> MALZBENDER: So, this technique really works well for diffused objects and then adding
specular highlights in for that, and for that class of images and data sets, diffused shading,
for example, not to have enough dynamic range where you need to really worry about high
dynamic range. If you're trying to capture true synthetic spectacular highlights, and
we've done work on that, you know, as well since, you know, since this--since the paper--it's
the original of 2001 paper. Yeah, you need high dynamic range for doing that, certainly.
Specular highlights are so bright typically that they blow out the eight bits in dynamic
range. You can't get good data in both the low, you know, the low, the low order and
the higher order, so, yeah. But one of the things we've also looked at how much resolution
you need for displaying the coefficients of the PTM itself, and it turns out, you know,
we've got a trick in here. We have a global scale and bias, scale and biased values that
are in the PTM file that allow basically just store all the PTM coefficients at eight bits
resolution, that's adequate. So, there are definitely resolution tricks you can play.
>> Have there been any applications in movie making?
>> MALZBENDER: In movie making? >> Yeah.
>> MALZBENDER: There's a researcher at USC named Paul Debevec who has done some very
impressive work throughout the years applying, relating sorts of techniques to movies. So,
he's certainly done quite a bit of that and very successful and works closely with Hollywood
doing this, so yes, absolutely. One of the more interesting things that Paul has built
is a large dome with colored lights on the surface of it, so they can actually simulate
the lighting on an actor from a completely synthetic environment. So, they can go ahead
and capture the lighting environment of a synthetic environment or--sorry, of a real
environment, and then produce it synthetically to match the lighting that you're producing
on an actor to that of the--some other environment. Yeah.
>> Not a question, I guess an observation. It seems to remind me a lot of how the plenoptic
or light field cameras work with a... >> MALZBENDER: Yeah
>> ...each pixel being sort of a bundle of incident rays.
>> MALZBENDER: Yup, absolutely. So, you know, we've noticed that parallel as well, you know,
even before we developed this technique. You know, if you look at--if you try to apply
PTM technique through the polynomial itself, two values were taken as you vary camera location
as opposed to varying lighting, it turns out there's, there's just not as much coherence
to the--those pixel values. And so, it makes a low auto representation like this just doesn't
work for representing light fields whereas it works beautifully for this. And, you know,
one of the reasons it works for this is first of all, there's a lot of redundancy in the
data but also, the human visual system really can tolerate quite a bit of sloppiness, you
know, in this. So, when you make one of these PTMs, if you have errors of, you know, even
up to like five percent of your estimation of light source direction, you won't see any
artifacts in the original--or in the PTM that's produced. And the reason for that is quite
simply--yeah, you might have to moved the light source, the synthetic light source to
slightly different location. It might be five degrees off now in the rendering, but in the
end, you'll get a very, you know, a very similar result. So, it's very robust and applied to
light fields, it would not be so robust, so >> Are your photographs done in white light?
Have you ever done like in red, green, blue separately and then combine them and, and
vary those >> MALZBENDER: I mean we typically do in white,
and then we can separate again the PTM representation in each of the colored generals independently,
and then, you know, we surely can have control over color in that way. So yeah, it can be
done under any color light source, but typically white, we do it for generality, so. Great.
Well, thanks very much for your attention.