Untitled Document

Antialiasing

You might have noticed in some of your OpenGL pictures that lines, especially nearly horizontal or nearly vertical ones, appear jagged. These jaggies appear because the ideal line is approximated by a series of pixels that must lie on the pixel grid. The jaggedness is called aliasing, and this section describes antialiasing techniques to reduce it. Figure 6-2 shows two intersecting lines, both aliased and antialiased. The pictures have been magnified to show the effect.

Figure 6-2 : Aliased and Antialiased Lines

Figure 6-3 shows how a diagonal line 1 pixel wide covers more of some pixel squares than others. In fact, when performing antialiasing, OpenGL calculates a coverage value for each fragment based on the fraction of the pixel square on the screen that it would cover. The figure shows these coverage values for the line. In RGBA mode, OpenGL multiplies the fragment's alpha value by its coverage. You can then use the resulting alpha value to blend the fragment with the corresponding pixel already in the framebuffer. In color-index mode, OpenGL sets the least significant 4 bits of the color index based on the fragment's coverage (0000 for no coverage and 1111 for complete coverage). It's up to you to load your color map and apply it appropriately to take advantage of this coverage information.

Figure 6-3 : Determining Coverage Values

The details of calculating coverage values are complex, difficult to specify in general, and in fact may vary slightly depending on your particular implementation of OpenGL. You can use the glHint() command to exercise some control over the trade-off between image quality and speed, but not all implementations will take the hint.

void glHint(GLenum target, GLenum hint);

Controls certain aspects of OpenGL behavior. The target parameter indicates which behavior is to be controlled; its possible values are shown in Table 6-2. The hint parameter can be GL_FASTEST to indicate that the most efficient option should be chosen, GL_NICEST to indicate the highest-quality option, or GL_DONT_CARE to indicate no preference. The interpretation of hints is implementation-dependent; an implementation can ignore them entirely. (For more information about the relevant topics, see "Antialiasing" for the details on sampling and "Fog" for details on fog.)

The GL_PERSPECTIVE_CORRECTION_HINT target parameter refers to how color values and texture coordinates are interpolated across a primitive: either linearly in screen space (a relatively simple calculation) or in a perspective-correct manner (which requires more computation). Often, systems perform linear color interpolation because the results, while not technically correct, are visually acceptable; however, in most cases textures require perspective-correct interpolation to be visually acceptable. Thus, an implementation can choose to use this parameter to control the method used for interpolation. (See Chapter 3 for a discussion of perspective projection, Chapter 4 for a discussion of color, and Chapter 9 for a discussion of texture mapping.)

Table 6-2 : Values for Use with glHint()
Parameter	Meaning
GL_POINT_SMOOTH_HINT, GL_LINE_SMOOTH_HINT, GL_POLYGON_SMOOTH_HINT	Specify the desired sampling quality of points, lines, or polygons during antialiasing operations
GL_FOG_HINT	Specifies whether fog calculations are done per pixel (GL_NICEST) or per vertex (GL_FASTEST)
GL_PERSPECTIVE_CORRECTION_HINT	Specifies the desired quality of color and texture-coordinate interpolation

Antialiasing Points or Lines

To antialias points or lines, you need to turn on antialiasing with glEnable(), passing in GL_POINT_SMOOTH or GL_LINE_SMOOTH, as appropriate. You might also want to provide a quality hint with glHint(). (Remember that you can set the size of a point or the width of a line. You can also stipple a line. See "Line Details" in Chapter 2.) Next follow the procedures described in one of the following sections, depending on whether you're in RGBA or color-index mode.

Antialiasing in RGBA Mode

In RGBA mode, you need to enable blending. The blending factors you most likely want to use are GL_SRC_ALPHA (source) and GL_ONE_MINUS_SRC_ALPHA (destination). Alternatively, you can use GL_ONE for the destination factor to make lines a little brighter where they intersect. Now you're ready to draw whatever points or lines you want antialiased. The antialiased effect is most noticeable if you use a fairly high alpha value. Remember that since you're performing blending, you might need to consider the rendering order as described in "Three-Dimensional Blending with the Depth Buffer." However, in most cases, the ordering can be ignored without significant adverse effects. Example 6-3 initializes the necessary modes for antialiasing and then draws two intersecting diagonal lines. When you run this program, press the `r' key to rotate the lines so that you can see the effect of antialiasing on lines of different slopes. Note that the depth buffer isn't enabled in this example.

Example 6-3 : Antialiased lines: aargb.c

#include <GL/gl.h>

#include <GL/glu.h>

#include <GL/glut.h>

#include <stdlib.h>

#include <stdio.h>

static float rotAngle = 0.;

/*  Initialize antialiasing for RGBA mode, including alpha

 *  blending, hint, and line width.  Print out implementation

 *  specific info on line width granularity and width.

*/

void init(void)

   GLfloat values[2];

   glGetFloatv (GL_LINE_WIDTH_GRANULARITY, values);

   printf ("GL_LINE_WIDTH_GRANULARITY value is %3.1f\n",

      values[0]);

   glGetFloatv (GL_LINE_WIDTH_RANGE, values);

   printf ("GL_LINE_WIDTH_RANGE values are %3.1f %3.1f\n",

      values[0], values[1]);

   glEnable (GL_LINE_SMOOTH);

   glEnable (GL_BLEND);

   glBlendFunc (GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);

   glHint (GL_LINE_SMOOTH_HINT, GL_DONT_CARE);

   glLineWidth (1.5);

   glClearColor(0.0, 0.0, 0.0, 0.0);

/* Draw 2 diagonal lines to form an X */

void display(void)

   glClear(GL_COLOR_BUFFER_BIT);

   glColor3f (0.0, 1.0, 0.0);

   glPushMatrix();

   glRotatef(-rotAngle, 0.0, 0.0, 0.1);

   glBegin (GL_LINES);

      glVertex2f (-0.5, 0.5);

      glVertex2f (0.5, -0.5);

   glEnd ();

   glPopMatrix();

   glColor3f (0.0, 0.0, 1.0);

   glPushMatrix();

   glRotatef(rotAngle, 0.0, 0.0, 0.1);

   glBegin (GL_LINES);

      glVertex2f (0.5, 0.5);

      glVertex2f (-0.5, -0.5);

   glEnd ();

   glPopMatrix();

   glFlush();

void reshape(int w, int h)

   glViewport(0, 0, (GLint) w, (GLint) h);

   glMatrixMode(GL_PROJECTION);

   glLoadIdentity();

   if (w <= h)

      gluOrtho2D (-1.0, 1.0,

         -1.0*(GLfloat)h/(GLfloat)w, 1.0*(GLfloat)h/(GLfloat)w);

   else

      gluOrtho2D (-1.0*(GLfloat)w/(GLfloat)h,

         1.0*(GLfloat)w/(GLfloat)h, -1.0, 1.0);

   glMatrixMode(GL_MODELVIEW);

   glLoadIdentity();

void keyboard(unsigned char key, int x, int y)

   switch (key) {

      case `r':

      case `R':

         rotAngle += 20.;

         if (rotAngle >= 360.) rotAngle = 0.;

         glutPostRedisplay();

         break;

      case 27:  /*  Escape Key  */

         exit(0);

         break;

      default:

         break;

int main(int argc, char** argv)

   glutInit(&argc, argv);

   glutInitDisplayMode (GLUT_SINGLE | GLUT_RGB);

   glutInitWindowSize (200, 200);

   glutCreateWindow (argv[0]);

   init();

   glutReshapeFunc (reshape);

   glutKeyboardFunc (keyboard);

   glutDisplayFunc (display);

   glutMainLoop();

   return 0;

Antialiasing in Color-Index Mode

The tricky part about antialiasing in color-index mode is loading and using the color map. Since the last 4 bits of the color index indicate the coverage value, you need to load sixteen contiguous indices with a color ramp from the background color to the object's color. (The ramp has to start with an index value that's a multiple of 16.) Then you clear the color buffer to the first of the sixteen colors in the ramp and draw your points or lines using colors in the ramp. Example 6-4 demonstrates how to construct the color ramp to draw antialiased lines in color-index mode. In this example, two color ramps are created: one contains shades of green and the other shades of blue.

Example 6-4 : Antialiasing in Color-Index Mode: aaindex.c

#include <GL/gl.h>

#include <GL/glu.h>

#include <GL/glut.h>

#include <stdlib.h>

#define RAMPSIZE 16

#define RAMP1START 32

#define RAMP2START 48

static float rotAngle = 0.;

/*  Initialize antialiasing for color-index mode,

 *  including loading a green color ramp starting

 *  at RAMP1START, and a blue color ramp starting

 *  at RAMP2START. The ramps must be a multiple of 16.

*/

void init(void)

   int i;

   for (i = 0; i < RAMPSIZE; i++) {

      GLfloat shade;

      shade = (GLfloat) i/(GLfloat) RAMPSIZE;

      glutSetColor(RAMP1START+(GLint)i, 0., shade, 0.);

      glutSetColor(RAMP2START+(GLint)i, 0., 0., shade);

   glEnable (GL_LINE_SMOOTH);

   glHint (GL_LINE_SMOOTH_HINT, GL_DONT_CARE);

   glLineWidth (1.5);

   glClearIndex ((GLfloat) RAMP1START);

/*  Draw 2 diagonal lines to form an X */

void display(void)

   glClear(GL_COLOR_BUFFER_BIT);

   glIndexi(RAMP1START);

   glPushMatrix();

   glRotatef(-rotAngle, 0.0, 0.0, 0.1);

   glBegin (GL_LINES);

      glVertex2f (-0.5, 0.5);

      glVertex2f (0.5, -0.5);

   glEnd ();

   glPopMatrix();

   glIndexi(RAMP2START);

   glPushMatrix();

   glRotatef(rotAngle, 0.0, 0.0, 0.1);

   glBegin (GL_LINES);

      glVertex2f (0.5, 0.5);

      glVertex2f (-0.5, -0.5);

   glEnd ();

   glPopMatrix();

   glFlush();

void reshape(int w, int h)

   glViewport(0, 0, (GLsizei) w, (GLsizei) h);

   glMatrixMode(GL_PROJECTION);

   glLoadIdentity();

   if (w <= h)

      gluOrtho2D (-1.0, 1.0,

         -1.0*(GLfloat)h/(GLfloat)w, 1.0*(GLfloat)h/(GLfloat)w);

   else

      gluOrtho2D (-1.0*(GLfloat)w/(GLfloat)h,

         1.0*(GLfloat)w/(GLfloat)h, -1.0, 1.0);

   glMatrixMode(GL_MODELVIEW);

   glLoadIdentity();

void keyboard(unsigned char key, int x, int y)

   switch (key) {

      case `r':

      case `R':

         rotAngle += 20.;

         if (rotAngle >= 360.) rotAngle = 0.;

         glutPostRedisplay();

         break;

      case 27:  /*  Escape Key */

         exit(0);

         break;

      default:

         break;

int main(int argc, char** argv)

   glutInit(&argc, argv);

   glutInitDisplayMode (GLUT_SINGLE | GLUT_INDEX);

   glutInitWindowSize (200, 200);

   glutCreateWindow (argv[0]);

   init();

   glutReshapeFunc (reshape);

   glutKeyboardFunc (keyboard);

   glutDisplayFunc (display);

   glutMainLoop();

   return 0;

Since the color ramp goes from the background color to the object's color, the antialiased lines look correct only in the areas where they are drawn on top of the background. When the blue line is drawn, it erases part of the green line at the point where the lines intersect. To fix this, you would need to redraw the area where the lines intersect using a ramp that goes from green (the color of the line in the framebuffer) to blue (the color of the line being drawn). However, this requires additional calculations and it is usually not worth the effort since the intersection area is small. Note that this is not a problem in RGBA mode, since the colors of object being drawn are blended with the color already in the framebuffer.

You may also want to enable the depth test when drawing antialiased points and lines in color-index mode. In this example, the depth test is disabled since both of the lines lie in the same z-plane. However, if you want to draw a three-dimensional scene, you should enable the depth buffer so that the resulting pixel colors correspond to the "nearest" objects.

The trick described in "Three-Dimensional Blending with the Depth Buffer" can also be used to mix antialiased points and lines with aliased, depth-buffered polygons. To do this, draw the polygons first, then make the depth buffer read-only and draw the points and lines. The points and lines intersect nicely with each other but will be obscured by nearer polygons.

Try This

Take a previous program, such as the robot arm or solar system examples described in "Examples of Composing Several Transformations" in Chapter 3, and draw wireframe objects with antialiasing. Try it in either RGBA or color-index mode. Also try different line widths or point sizes to see their effects.

Antialiasing Polygons

Antialiasing the edges of filled polygons is similar to antialiasing points and lines. When different polygons have overlapping edges, you need to blend the color values appropriately. You can either use the method described in this section, or you can use the accumulation buffer to perform antialiasing for your entire scene. Using the accumulation buffer, which is described in Chapter 10, is easier from your point of view, but it's much more computation-intensive and therefore slower. However, as you'll see, the method described here is rather cumbersome.

Note: If you draw your polygons as points at the vertices or as outlines - that is, by passing GL_POINT or GL_LINE to glPolygonMode() - point or line antialiasing is applied, if enabled as described earlier. The rest of this section addresses polygon antialiasing when you're using GL_FILL as the polygon mode.

In theory, you can antialias polygons in either RGBA or color-index mode. However, object intersections affect polygon antialiasing more than they affect point or line antialiasing, so rendering order and blending accuracy become more critical. In fact, they're so critical that if you're antialiasing more than one polygon, you need to order the polygons from front to back and then use glBlendFunc() with GL_SRC_ALPHA_SATURATE for the source factor and GL_ONE for the destination factor. Thus, antialiasing polygons in color-index mode normally isn't practical.

To antialias polygons in RGBA mode, you use the alpha value to represent coverage values of polygon edges. You need to enable polygon antialiasing by passing GL_POLYGON_SMOOTH to glEnable(). This causes pixels on the edges of the polygon to be assigned fractional alpha values based on their coverage, as though they were lines being antialiased. Also, if you desire, you can supply a value for GL_POLYGON_SMOOTH_HINT.

Now you need to blend overlapping edges appropriately. First, turn off the depth buffer so that you have control over how overlapping pixels are drawn. Then set the blending factors to GL_SRC_ALPHA_SATURATE (source) and GL_ONE (destination). With this specialized blending function, the final color is the sum of the destination color and the scaled source color; the scale factor is the smaller of either the incoming source alpha value or one minus the destination alpha value. This means that for a pixel with a large alpha value, successive incoming pixels have little effect on the final color because one minus the destination alpha is almost zero. With this method, a pixel on the edge of a polygon might be blended eventually with the colors from another polygon that's drawn later. Finally, you need to sort all the polygons in your scene so that they're ordered from front to back before drawing them.

Example 6-5 shows how to antialias filled polygons; clicking the left mouse button toggles the antialiasing on and off. Note that backward-facing polygons are culled and that the alpha values in the color buffer are cleared to zero before any drawing. Pressing the `t' key toggles the antialiasing on and off.

Note: Your color buffer must store alpha values for this technique to work correctly. Make sure you request GLUT_ALPHA and receive a legitimate window.

Example 6-5 : Antialiasing Filled Polygons: aapoly.c

#include <GL/gl.h>

#include <GL/glu.h>

#include <GL/glut.h>

#include <stdlib.h>

#include <stdio.h>

#include <string.h>

GLboolean polySmooth = GL_TRUE;

static void init(void)

   glCullFace (GL_BACK);

   glEnable (GL_CULL_FACE);

   glBlendFunc (GL_SRC_ALPHA_SATURATE, GL_ONE);

   glClearColor (0.0, 0.0, 0.0, 0.0);

#define NFACE 6

#define NVERT 8

void drawCube(GLdouble x0, GLdouble x1, GLdouble y0,

              GLdouble y1, GLdouble z0, GLdouble z1)

   static GLfloat v[8][3];

   static GLfloat c[8][4] = {

      {0.0, 0.0, 0.0, 1.0}, {1.0, 0.0, 0.0, 1.0},

      {0.0, 1.0, 0.0, 1.0}, {1.0, 1.0, 0.0, 1.0},

      {0.0, 0.0, 1.0, 1.0}, {1.0, 0.0, 1.0, 1.0},

      {0.0, 1.0, 1.0, 1.0}, {1.0, 1.0, 1.0, 1.0}

};

/*  indices of front, top, left, bottom, right, back faces  */

   static GLubyte indices[NFACE][4] = {

      {4, 5, 6, 7}, {2, 3, 7, 6}, {0, 4, 7, 3},

      {0, 1, 5, 4}, {1, 5, 6, 2}, {0, 3, 2, 1}

};

   v[0][0] = v[3][0] = v[4][0] = v[7][0] = x0;

   v[1][0] = v[2][0] = v[5][0] = v[6][0] = x1;

   v[0][1] = v[1][1] = v[4][1] = v[5][1] = y0;

   v[2][1] = v[3][1] = v[6][1] = v[7][1] = y1;

   v[0][2] = v[1][2] = v[2][2] = v[3][2] = z0;

   v[4][2] = v[5][2] = v[6][2] = v[7][2] = z1;

#ifdef GL_VERSION_1_1

   glEnableClientState (GL_VERTEX_ARRAY);

   glEnableClientState (GL_COLOR_ARRAY);

   glVertexPointer (3, GL_FLOAT, 0, v);

   glColorPointer (4, GL_FLOAT, 0, c);

   glDrawElements(GL_QUADS, NFACE*4, GL_UNSIGNED_BYTE, indices);

   glDisableClientState (GL_VERTEX_ARRAY);

   glDisableClientState (GL_COLOR_ARRAY);

#else

   printf ("If this is GL Version 1.0, ");

   printf ("vertex arrays are not supported.\n");

   exit(1);

#endif

/*  Note:  polygons must be drawn from front to back

 *  for proper blending.

*/

void display(void)

   if (polySmooth) {

      glClear (GL_COLOR_BUFFER_BIT);

      glEnable (GL_BLEND);

      glEnable (GL_POLYGON_SMOOTH);

      glDisable (GL_DEPTH_TEST);

   else {

      glClear (GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

      glDisable (GL_BLEND);

      glDisable (GL_POLYGON_SMOOTH);

      glEnable (GL_DEPTH_TEST);

   glPushMatrix ();

      glTranslatef (0.0, 0.0, -8.0);

      glRotatef (30.0, 1.0, 0.0, 0.0);

      glRotatef (60.0, 0.0, 1.0, 0.0);

      drawCube(-0.5, 0.5, -0.5, 0.5, -0.5, 0.5);

   glPopMatrix ();

   glFlush ();

void reshape(int w, int h)

   glViewport(0, 0, (GLsizei) w, (GLsizei) h);

   glMatrixMode(GL_PROJECTION);

   glLoadIdentity();

   gluPerspective(30.0, (GLfloat) w/(GLfloat) h, 1.0, 20.0);

   glMatrixMode(GL_MODELVIEW);

   glLoadIdentity();

void keyboard(unsigned char key, int x, int y)

   switch (key) {

      case `t':

      case `T':

         polySmooth = !polySmooth;

         glutPostRedisplay();

         break;

      case 27:

         exit(0);  /*  Escape key  */

         break;

      default:

         break;

int main(int argc, char** argv)

   glutInit(&argc, argv);

   glutInitDisplayMode (GLUT_SINGLE | GLUT_RGB

                        | GLUT_ALPHA | GLUT_DEPTH);

   glutInitWindowSize(200, 200);

   glutCreateWindow(argv[0]);

   init ();

   glutReshapeFunc (reshape);

   glutKeyboardFunc (keyboard);

   glutDisplayFunc (display);

   glutMainLoop();

   return 0;