Originally published 16 Jan 2011
In this post I look at geometry in the service of art, looking at Microsoft’s Kinect sensor technology and Piero della Francesca’s orthographic projection maps of the human head. Viewing a ‘Kinected’ object through infrared light reveals ‘a dissolution of surfaces into a constellation of dots‘. (ref 1) This poetic description was crafted not about Kinect, but about Piero’s head drawings, plotted in 3D space.
The fact that it describes Kinect’s infrared sensor patterns has everything to do with their common approach to a very similar problem.
The challenge that both Kinect developers and Piero della Francesca, Italian artist and mathematician c1415-1492, confronted, was how to create a representation of the human body which appeared convincingly plotted in 3D space, with the Kinect developers facing the additional challenge of perpetual motion.
This challenge breaks down as follows for Piero and Microsoft:
They both needed to improve upon existing methods for doing this in order to achieve greater accuracy
They both sought inspiration from nature
They both use dots on a grid to track the relative position of solids in space
They both needed to improve upon existing methods for doing this in order to achieve greater accuracy.
Kinect developers saw the great potential and success of the Wii, but could see that the Nunchuk™ had its limitations in tracking each player in space, mirroring people back, and encouraging natural gestures. In short they needed to free the body of hardware.
Piero della Francesca was part of an elite community of painters in the early 15th century Italy, centred in Florence, who knew about linear perspective and weren’t afraid to use it. But perspective had its limitations when it came to depicting the human body. Linear perspective could tell the painter where to place a person, and what size they should be, but it couldn’t tell the painter exactly how a partially rotated head might look within the grid – rotations that represent natural movement.
They both sought inspiration from nature
Piero looked to geometry to solve problems of natural representation. Piero, being both a mathematician and a craftsman, was accustomed to using geometry to interpret natural forms. As a member of the merchant class he had already developed advanced skills in ‘volumetric gauging’, amongst other skills in order to do business. (ref 2)
Microsoft looked also to nature, imitating the methods of that most natural of optical devices, the human eye(s). For an interesting view on how Kinect achieves stereo vision see How the Kinect senses depth.
The developers also looked at human perception, which can tell a person apart from other objects and focus on them as they move around.
They both use dots on a grid to track the relative position of solids in space
Kinect definitely became more intriguing when I saw the YouTube videos of the infrared dots.
This is my very basic understanding of what’s happening here:
Kinect blasts a grid of infrared light dots onto the stage and the players on it.
This looks like a tic-tac-toe board with thousands of dots plotted in each square across the x y planes of the space.
The closer the object is to the sensor, the brighter the light is reflected back - Each dot’s shade of grey represents its distance from the sensor (on the z plane)
Kinect then plots a 20 joint skeleton to plausible human forms, and locks onto these forms, now in memory
With a refresh rate of 30 frames per second, it tracks movements in 3D space and reflects back to the players their body movements via their skeletal overlay.
Put another way, the players are tracked in space using a 3D orthographic dot grid matrix. Forms are understood by connecting the dots.
Which brings us to Piero della Francesca. In Book 3 of his treatise De Prospectiva Pingendi (On Perspective in Painting) c. 1480, he looks at depicting the human head in 3D space, front, back, and top view. He plotted the shape of the form on the x, y and z planes with dots. He could then rotate the head in 3D space, maintaining the relative position of each dot on the grids in rotation. This is the first recorded account of orthographic projection. (ref 3)
One of the heads, now tilted in space, looks very much like one of the soldier’s heads in the Resurrection (ref 4), which some speculate to be a self portrait.
It feels very much like the Kinect is a direct descendant of Piero’s math for art’s sake.
Which brings us finally, to the why of this challenge
Why invent Kinect?
People are hooked on immersion in virtual spaces and stories. Kinect offers one step closer to the dream of literally stepping into a different world unencumbered by awkward technology. We’re always on the lookout for that Door Into Summer (a Sci-Fi book written by Robert A. Heinlein, 1957, inspired by his snow hating cat).
Why depict realistic 3D people in sacred paintings?
In a sense Piero’s aspiration can be seen as a door into the ideal.
Piero strove for a natural geometry in his art. Although the move toward naturalism at this point in art history is often seen in terms of a classical revival, it was also a way to move closer to sacred ideals. This is in keeping with the idea that 'earth was a pale reflection of heaven'. (ref 5)
To depict heavenly themes, one needed to get the proportions of earth correct first, and then find a way to transcend the pale.
In Resurrection (fresco circa 1460), that transcendence may be seen in his intentional skewing of the perspective layout, depicting more than one perspective space in one scene. Notice that Christ is viewed straight on in relation to the viewing angle of the sleeping guards in front of him (as if seen from below). In doing this he (possibly) expresses the irrationality of spiritual subjects.
In rising to their own challenge, Microsoft has accomplished unencumbered bodily immersion in their game space. For Piero, his art has survived over 500 years. His mathematical legacy of orthographic projection lies at the heart of technology as diverse as 3D modelling software and neuroimaging.
(See a GD Schott’s fascinating article Piero della Francesca’s projections and neuroimaging today in The Lancet.)
So Piero’s influence has extended to virtual reality in the form of Kinect. I wonder what game he would have created for it?
References 1. Evans, Robin The Projective Cast: Architecture and Its Three Geometries, MIT Press (1995), p. 151.
2. Baxandall, Michael, Painting and Experience in Fifteenth-Century Italy, Oxford Paperbacks; 2 edition (1988), pp. 86-91.
3. Evans, op. cit., p 151.
4. Lavin, Marilyn Aaronberg, Piero della Francesca (Art and Ideas), Phaidon Press Ltd (Mar 2002), pp.220-226.
5. Edgerton, Samuel Y., Rev, of The Mirror, the Window, and the Telescope: How Renaissance Linear Perspective Changed Our Vision of the Universe, by, Samuel Y Edgerton,Nexus Network Journal – Vol. 12, No. 1, 2010: 150.
Resources
Schott, GD, Piero della Francesca’s projections and neuroimaging today, The Lancet – 18 October 2008 ( Vol. 372, Issue 9647, Pages 1378-1379 ) DOI: 10.1016/S0140-6736(08)61576-7
Archived Comments
Hasan Niyazi 20 Jan 2011 Great post! Love the congruencies between Piero and the tech underlying Kinect! It reminds of the parallels between Paolo Uccello’s perspective grid renders and modern 3d modelling applications H
