Understanding Normal Maps

From ZBrushInfo

Surface Normals and Lighting

A surface normal for any flat surface is simply the arrow (or direction) that points directly out from that surface. The angle between that arrow and any line drawn on the surface from the starting point of the arrow is exactly 90 degrees. Put a piece of thin white cardboard on a desk, put the eraser end of a pencil on the cardboard, and point the pencil straight up. The pencil is now a pointer in the direction of the cardboard's surface normal.

Figure: Surface normal for a horizontal piece of cardboard.

If a flat surface is oriented so that its normal points towards a light, that surface will be lit brightly by the light. If the surface is turned so that the normal points farther away from the light, it will become darker. If the surface normal points more than ninety degrees away from the light, the surface will not be lit at all. The white cube below shows this.

Figure: Surface normals and surface brightness.

Interaction of light direction with the surface normals of a white cube. The normal of the top surface points close to the light, and so that surface is the most brightly lit. The normal of the right surface points farther away from the light, and is less brightly lit. The normal of the front side points more than 90 degrees away from the light, so it is dark.

When a renderer calculates how brightly a light illuminates a surface, it (almost always) does it with a simple mathematical calculation involving the angle between the surface normal and the direction of the light.

All of this may seem obvious, but there's an important point here: Lighting calculations are done using the direction of the surface normal, not the orientation of the surface. This makes no difference as long as the surface normal does in fact point straight out from the surface. But if we simply give to the renderer a different value of the surface normal, we can change the apparent intensity with which the surface is lit, and hence the apparent orientation, without changing the actual surface orientation at all. This is the basis of normal mapping, and is discussed in detail in the next section.

Surface Normals with Specular and Reflective Lighting

The previous explanation actually applied to just one type of lighting model, diffuse lighting, where the surface being lit is assumed to scatter incoming light equally in all directions.

Another common type of lighting model is specular lighting, which produces the highlights on smooth or semi-smooth surfaces such as the bright spots on a billiard ball. Specular lighting is a model that assumes light is reflected with some degree of directionality, rather than scattered perfectly in all directions.

We won't go into details, but surface normals can also be used to calculate specular lighting. Basically, the specular brightness depends on the angle between the surface normal and the average of the direction between the light and the camera. This average direction is easy to visualize. Imagine you are on a stage, being lit by one spotlight, with one person in the audience. Now point halfway between the person and the spotlight. That's the average direction between your viewer and the spotlight.

Reflective lighting is a special case of specular lighting. It's specular lighting having no (or negligible) scattering. Images reflected in a mirror or off the surface of a still pond are examples. For realism, reflections are often computed differently than specular lighting, with the aid of a reflection map. Again without going too much into details, this can be handled with normal maps. From a point on the surface, the direction to the camera issurface normal usually means whatever direction is used when doing lighting calculations. From now on, we'll use the graphics meaning, not the mathematical meaning, and in other sections of the paper, the phrase is also used in its graphics, not its mathematical, form.}}

This process can be illustrated by taking the top face of the previously shown, dividing it into four smaller sections, and changing the surface normal in each section. Each section appears with greater or lesser brightness as a result.

Figure: Normal perturbation and effect on lighting.

The surface below is a single flat quadrilateral, divided into four areas. The surface normal in each area for the purposes of lighting calculations (shown by the arrows) is a perturbation of the true surface normal for the quadrilateral. As a result, each area is lit with a different intensity, since brightness is calculated as a function of the angle of the normal to the light direction. In a rendered image, this can give the illusion that different areas of a surface are at different angles, even though the surface is completely flat.

Bump Maps and Surface Normals

Let's talk a bit about surface normals as they relate to bump maps. Bump maps are actually converted to normal maps (invisibly, either by hardware or software) before being drawn on screen, If you understand this process, you understand normal maps.

To see this in action let's look at a single row of pixels from a bump map. The following examples will use lighter colors to indicate higher areas on the bump map, but some applications may do exactly the opposite. The concept is still the same.

Figure: Pixel row from bump map.

A row of pixels from a bump map, looking down. Lighter colors represent higher elevations in this map.

Figure: Converting bump intensity to bump height and normals.

The same bump map, viewed from the side. This shows how (in concept) pixels are raised in height according to their color. A curve is fitted to the resulting profile, and at each coordinate, the normal to that curve is calculated.

Figure: Resulting normals assigned to pixel locations.

The resulting surface normals associated with each pixel, viewed from the side. Once these have been calculated by the hardware or software, the bump map height information is discarded, and the normal directions--which are all that matter for lighting calculations--are retained. The lengths of the normals are not meaningful in this example.

It's often easier to create a bump map via painting than to model the detailed geometry with polygons. In addition, bump maps significantly speed the rendering process, compared to rendering a model with a polygon count high enough to represent the bumps geometrically. In the case of hardware, this can allow the presentation of bumpy surfaces while still animating in real time.

Representing Normal Maps as Textures

A normal is just a direction in space–a vector with three coordinates, usually called the x, y, and z coordinates. To create a normal map, we need to store these three coordinates at each pixel in the map.

Instead of defining a new file format just for this, a simple trick is used. Standard 32-bit images (such as used in many image processing programs) contain four channels of data; one for the red component of the pixel, one for the green, one for the blue, and one for the alpha (which is usually used for transparency). Each of these channels is 8 bits wide, meaning each of them can store one of 256 distinct values.

The x, y, and z coordinates of a vector at a given point can simply be stored in the red, green, and blue channels of the pixel at that point. When used in this way, the alpha channel is ignored. Now every point in the image has a normal direction associated with it, and normal direction is indicated by color in the normal map. This is quite different from bump maps, where normal direction is given by the intensity difference between nearby pixels.

One result of this is that normal maps are not suited for creation by hand. In a normal map, the general "shape" of what will appear on the rendered surface is apparent, but visualizing the exact effect or constructing the map by hand is difficult because of the interplay of the color channels. Constructing a bump map by hand is at least possible, as the grayscale intensities correspond to bump height--though it is very difficult to visualize the exact effect without a render. The left wall is a normal map of a brick surface, the right wall is an equivalent bump map, and the center cube shows how surfaces would render when either map is applied. The normal map actually shows the grain of the brick surface much better than the bump map, but the necessity of getting the right color at each pixel makes it almost impossible to paint by hand. The small differences in bump height over small areas make the bump map look 'smooth', but it could still be painted with an appropriate brush to give an almost invisible graininess to the texture.

Figure: Normal and Bump Map Textures and Render

Normal map (bluish) on the left, bump map on the right, cube mapped with either in the center. (Both the normal map and the bump map will produce the same visual result when rendered.)

Normal Map Generation

This section discusses various ways of generating normal maps. Raytracing is the dominant conceptual model of normal map generation and is the focus of many of the normal mapping tools that have appeared recently. ZBrush's multiple subdivision level editing can also be used to generate normal maps in a different manner, with more accurate results.

Converting From a Bump Map

One of the easiest ways to generate a normal map is to calculate it from an existing bump map. There are various tools to do this, and we won't go into them here. As well as being very simple when it is applicable, this method allows you to use existing bump maps, instead of taking the time to construct new normal maps.

There may seem little reason to do this, but as was mentioned before, one advantage that normal maps have compared to bump maps is that while bump maps require additional information describing how the bump intensity range maps to global space distance units, normal maps do not. If you are finding this to be a disadvantage when working with bump maps, converting to normal maps may well make sense.

There are disadvantages as well, in normal maps through simple conversion of bump maps. Bump maps cannot express as much directional information as a normal map of the same resolution, and so the visual effects available from a converted bump map may not be as striking as those available from a normal map constructed directly.

Normal maps also contain implicit information about object space, tangent space, and the like. Bump maps do not carry such information. As a result, conversion of bump maps is likely to be easy only when the map is one for a flat surface, such as a wall. Other cases may be more problematic.

Raytracing/Raycasting

A useful way of thinking about normal map generation is the idea of raytracing, or raycasting. (Both words are often used in describing this particular technique for normal map generation. The terms come from the raycasting and raytracing methods of rendering, to which they are similar.)

The raytracing method projects rays (the thin solid arrows below) from the surface of the low-resolution mesh (thick lines) to the surface of the high-resolution mesh (thin curve). The normal direction at the end of each ray is used as the normal vector value in the normal map UV coordinate at the start of the ray.

Figure: Raytracing

Rays (solid arrows) are projected from the low-resolution surface to the high-resolution surface. At each point a ray intersects the high-resolution surface, the normal (perpendicular) to the high-resolution surface is calculated at that point, and then applied to the low-res surface as the surface normal at the origin of the initial ray.

Conceptually, this is fairly accurate, but it leaves out a great many details. The most important of these is how points on the low-resolution surface are associated with points on the high-resolution surface. One way of doing this is to simply create two otherwise unrelated meshes having the "same shape" as each other, and to fit the low-resolution mesh inside the high-resolution mesh to generate the map. This may involve ensuring that the mesh rotations are completely identical, that all parts of the low-resolution mesh are inside the high-resolution mesh, and so on, depending on the tools used to do this. Achieving this degree of conformance can be extremely difficult for meshes with any type of shape complexity, such as arms with hands and fingers.

ZBrush can use the raycasting method, but normally uses a different scheme to map points from the first mesh to the second. The subdivision relationship between the high-resolution mesh and the low-resolution mesh is used in matching areas of the surface. This reduces and in many cases eliminates the problems described in the previous paragraph.

Normal Maps in Tangent and Object Space

A normal map specifies the direction of surface normals at points on polygons throughout the model. Direction is relative, however; if we say a normal is pointing up, do we mean up towards the top of the model, or up away from the surface of the polygon? In normal mapping, maps can be created using either type of direction. A normal map which assumes that up is straight up in the model coordinate space is said to be in object space, while a normal map which specifies directions relative to the faces of the polygons to which it is applied is called a tangent space normal map.

The figures below show the effect of this when a (very simple) mesh with assigned surface normals is deformed[11]. In the first case (using object space normals), even as the rightmost polygon changes its orientation, its brightness does not change, because its normal direction does not change with the polygon. This is not what you would expect with "real world" lighting, and illustrates why object space normal maps are not what you want to use if your mesh will deform. The second figure shows how tangent space normals will change direction along with their surfaces, and give the expected changes in surface brightness.

Figure: Effect of Deformation on Lighting with Object Space Normals

On the left of this figure, two adjacent squares are on the same plane, making a flat surface. For lighting purposes, each is assigned a different object space surface normal, so intensity with which each is lit is different.

On the right of the figure, the square with the normal labeled 'B' has been rotated clockwise. Because both normals are defined in object space, their directions remain the same with respect to the object (which itself has not been rotated), and so to the light source. The lighting intensity on each surface remains the same, regardless of the rotation of the right polygon.

Figure: Effect of Deformation on Lighting with Tangent Space Normals

As in the previous figure, one of the two squares has been rotated clockwise. In this case the normals are defined in tangent space, meaning that whenever a surface changes orientation its associated normal changes direction along with it. The result is that the light intensity on the rotated surface changes as one would expect.

One interesting effect of these two different interpretations of the normal map coordinates is that tangent space maps always appear "bluish" in color, while object space maps typically exhibit a full spectrum of colors. The reason is that in either type of map the normal will point "away" from the polygon, as a normal pointing "behind" the polygon would have no meaning. But in a tangent space map, "away from the polygon" means that the z coordinate at that point will always be a positive number. Since the z coordinate of a normal map is stored in the blue channel of the normal map texture, blue becomes the dominant color.

Object Space Map Uses

Object space normal maps can be applied to any object in which the orientation of the polygons relative to the model does not change. This typically means any rigid model. The model may be moved, scaled, or rotated, but not deformed.

Terrain, furniture, bas-reliefs, dueling swords or a door swinging open could be shown very effectively with an object space normal map, using the normal map to generate the details of the blade and pommel of the sword, or gouges or wood grain in the door.

Object space normal maps are easy to generate, and will likely require little if any tweaking to achieve the desired effect and to remove artifacts. They are also computationally quite efficient. Generating an object space map using ZBrush is simply a matter of selecting the appropriate option in the ZMapper plugin. Details are given in other sections.

Tangent Space Map Uses

Tangent space normal maps can be applied to any object, but are required with models that might be animated or deformed (other than through simple scaling). Given how commonly ZBrush is used for organic modeling and how often these models will be deformed, you will likely be generating mostly tangent space normal maps. This is simply a matter of choosing the relevant option in ZMapper.

Compared to world space normal maps, tangent space normal maps require significantly more computational power to render (though it's difficult to predict how much this will affect performance in practice). In addition, they may require quite a bit more tweaking to give you the desired end effect in other applications or renderers. This is more a matter of the fact that working with deforming meshes is always more complex than working with rigid meshes than it is of the conceptual differences between the two types of normal maps or differences in the way they are handled by ZBrush.

How Does Your Application Distinguish Between Tangent and Object Space Normal Maps?

There's a simple answer to this one; your application doesn't. A normal map is just a 2D picture where colors represent directions. It's usually easy to distinguish the two visually (a tangent space map will be predominantly blue in color, while an object space map will usually have a full range of colors), but even this is no guarantee. Certain object space maps could look bluish in the same way as a tangent space map.

Instead, you will simply generate whichever type of map is most appropriate for your model, and then inform your application how to interpret that map relative to the object it is applied to. Depending on the application, this may involve using different shaders, setting a checkbox, or various other methods.

Summary of Object Space and Tangent Space Normal Maps

There's nothing in this section you haven't seen before, but you may find it a convenient summary.

• Tangent space and object space normal maps are simply 2D images where colors are interpreted as directions.
• A major advantage of normal maps over bump maps is that normal maps are not given in terms of distance, and hence do not require adjustment for changes in distance units from one application to the next.

• In a tangent space normal map, the direction indicated by a pixel in the map is taken to be relative to the surface of the polygon to which that pixel applies. In an object space normal map, the direction indicated by a pixel is given in terms of the object's coordinate system.

• Tangent space normal maps can be used with either rigid or deformable objects, but have a higher computational overhead than object space normal maps. Object space normal maps are only useful with rigid bodies (unless you are trying for some truly psychedelic effects), but are faster to render than tangent space normal maps. You'll normally generate object space normal maps for rigid models.

• You may see references to world space normal maps. These are simply object space normal maps where the "object" is the entire world. The computations needed to use a world space map as an object space map are trivial, and generally there is no need to use world space maps.

• Humans can usually visually distinguish object maps and tangent maps (tangent maps will appear bluish), but to rendering software or hardware they are simply maps. You'll specify explicitly in your application whether a map should be applied as an object space map or a tangent space map.

Cavity Shading

We'll briefly mention one more rendering aspect here, since it can be incorporated into normal maps by ZMapper. It is cavity shading.

Cavity shading is a method for implementing an 'ambient occlusion' effect, and ambient occlusion is in turn the visual effect that we all see (but rarely notice) when looking into shallow crevices; they're darker. It isn't really shadowing, since it occurs even in diffuse light where shadows aren't cast. It has more to do with the fact that as light is reflected into a recess, and then back out, the intensity of the light becomes lower due to absorption by the surface of the recess.

This has nothing whatsoever to do with normal maps, except that it turns out to be easy to modify normal maps to include this effect. Or, to put it another way, if cavity shading is calculated at the same time a normal map is generated, then the data necessary to accomplish the cavity shading can be put directly into the normal map. Rendering it later will require no more data, and no more time, that would rendering with a normal map that did not include cavity shading.

The mathematical details are of interest only to the analytically inclined. From an artistic viewpoint, it is much more important that ZMapper does include cavity shading as an option, and this is discussed more thoroughly in the ZMapper instructions.