Building User Interfaces With Video and 3D Graphics For Fun and Profit!

With the availability of cheap video capture and 3D graphics hardware on most desktop computers, we now have the ability to investigate new ways of interaction that were previously beyond the reach of the average hobbyist at home. Rather than viewing everything on a 2D desktop, what possibilities exist for viewing information in 3D? Instead of using a keyboard and mouse, what other more intuitive ways are there to interact with a 3D environment?

This tutorial gives an introduction to some of the less mainstream possibilities that are available on your Linux PC running at home right now. Linux is an ideal platform for performing this type of work with, since there is a wide range of existing source code which can be extended in any way that you can imagine.

The purpose of this tutorial is to encourage developers to think of new possibilities for input devices and user interfaces for their computers. In order to support this, I will also cover some of the less used and understood subsystems available under Linux, including the DRI and OpenGL 3D interfaces in XFree86, accessing video cameras using Video4Linux and the 1394 Firewire drivers, using free libraries such as ARToolkit for 3D object tracking, libraries such as OpenCV for vision recognition, and scene graph libraries for rendering 3D graphics. Many of the subsystems provided for Linux are not documented well and this tutorial will teach from the years of experience that I have in this area.

Another aspect of the tutorial is how to interface custom hardware devices to your computer. I will explain how to implement hardware interfaces using serial, parallel, and USB ports, even for those with limited experience with electronics and no Linux kernel programming knowledge. Finally, I will discuss some of the more exotic hardware possibilities which I explore as part of my research, including the use of head mounted displays and wearable computers. I will be bringing some of my research equipment to the conference for attendees to try out themselves.

License

You are free to copy the text of this tutorial and use it for whatever you wish, although you must ensure that the original author’s name Wayne Piekarski is referenced in an appropriately visible section.

You are free to use the examples in the tutorial text for whatever you wish. If you write an application based on a lot of my examples or things you learned here then a mention in your credits would be much appreciated. I make no guarantee that anything in this tutorial is correct, and so you must use my examples and ideas at your own risk. If your application breaks because of something you read here, you can keep both pieces – please don't send them to me.

The source code included as attachments to the tutorial is all available under the GPL or other open source license, and you must ensure that any applications that you develop based on these comply with their appropriate licenses. Some of the examples were written by others and so I do not own the copyright, but I am distributing them under the terms of their licenses as I downloaded them from the Internet.

Table of Contents

1 Introduction 5

2 Base system install 7

2.1 Distributions 7

2.2 Debian distributions 8

2.3 Debian limitations 9

2.4 Debian package installation 9

2.5 Other configuration 10

2.6 Special Firewire configuration 11

3 3D graphics 12

3.1 GLX and DRI 12

3.2 3D hardware 13

3.2.1 Nvidia cards 13

3.2.2 ATI cards 15

3.2.3 Other cards 16

3.3 Configuring X using xchanger 16

3.4 Programming OpenGL 17

3.4.1 Libraries 18

3.5 Live video display 20

3.5.1 Initialisation 21

3.5.2 New video frame 22

3.5.3 Render 23

3.6 Scene graphs 23

3.6.1 Coin3D 24

3.6.2 Other scene graphs 25

4 Video capture 26

4.1 Video4Linux 26

4.2 1394 Firewire 29

5 Vision tracking 32

5.1 ARToolKit 32

5.1.1 Installation and compiling 32

5.1.2 ARToolKit internal operation 33

5.1.3 Example code 33

5.1.4 Quick and dirty 4x4 matrix tutorial 34

5.1.5 Uses for ARToolKit 35

5.2 Intel OpenCV 36

6 Hardware development 37

6.1 Interfaces 37

6.1.1 Headers 37

6.1.2 Parallel 38

6.1.3 Serial RS-232 39

6.1.4 USB serial 42

6.2 Cheap hacks 43

6.3 Other interfaces 44

7 Conclusion 45

8 References 46

1 Introduction

One of the guiding principles in this tutorial is to be “lazy like a fox”. Most people are very busy and while it would be nice to spend a lot of time on doing something perfectly, we don’t have a lot of time free but we still have projects we would like to build. This tutorial is designed to give a quick overview of useful technology that you can use to build projects at home right now, with the minimal amount of work possible. While some of these technologies are explained elsewhere, the documentation can be a bit lacking or non-existent. While some parts will be only an overview, I will focus in detail on the areas that have important tricks or features and may not be discussed elsewhere. This tutorial is designed to give you useful tools so that you can begin hacking straight away. You can skip the many frustrating hours I have spent playing around with my computers and reading obscure documents and code trying to get things working.

While I try to avoid requiring too much previous knowledge, I will not teach things like basic C programming or OpenGL. These things are easily learned from many other excellent sources and so I want to focus my time on the more obscure areas. However, if you have any C programming experience then you should be able to follow what is happening and then you can catch up on things you missed after the tutorial.

We start the tutorial by briefly explaining how to configure a Linux distribution (mainly Debian and its derivatives) to contain the required packages that will be needed for development. Most modern distributions are suitable and contain pretty much everything you need but there are a few tricks (especially with Firewire and DRI) that I will go through that you must set up correctly and are poorly documented.

Next up is a detailed discussion of 3D graphics support with specific reference to Linux. While many books talk about OpenGL, there is almost no discussion of how 3D is implemented in Linux. Many people think they are running an accelerated server when in fact it is not configured correctly and using software emulation only. I will describe how to configure your server using my xchanger tool that I have written to simplify the configuration of 3D hardware. From a programming perspective, I cover these important details and explain what the various libraries do and how they are used. I discuss in some detail the use of OpenGL to work around X11 limitations and its use for live video display. We finish off with a brief discussion of scene graph libraries and how they can be used to simplify application development.

The two video capture subsystems available under Linux are not commonly used by many, and have only small amounts of documentation that describe how they operate. I will go through in detail the use of V4L and Video1394 to perform video capture, including their various features, problems, and examples.

One of the most exciting uses of video capture is using it to perform real time vision recognition. There are a number of free toolkits such as OpenCV and ARToolkit which can be used to extract out information from video in real-time. These libraries can be used to develop exciting 2D and 3D interactive applications on a desktop without requiring expensive virtual reality equipment.

To finish off the tutorial, I will discuss some useful information for hackers who want to make their own hardware. Building your own hardware opens up a wide range of possibilities for exciting projects with your computers, such as attaching flashing lights and push buttons. I discuss the various interfaces available such as USB, serial, and parallel ports, and how they can be used. I have including lots of example source code to help you get started.

Overall, this tutorial contains approximately forty page of notes, numerous free sample programs, and information which is not normally available elsewhere. I hope that you all will take advantage of this opportunity to learn about lots of new and exciting things that are possible with Linux. I hope you enjoy the tutorial and look forward to talking to you all at Linux Conf Au 2005 in Canberra!

2 Base system install

In this tutorial we will work with some of the more obscure libraries available for Linux, and these libraries tend to be very dynamic and change often. As a result, I would advise you to install the latest distribution that you can get, and preferably including as many packages as possible. While it is possible to go and download each library as a tar ball and compile and install the sources yourself, this is quite time consuming and it is easier if a distribution can manage as much for you as possible.

2.1 Distributions

I have used many Linux distributions over the years in my search for the distribution, and I have found that this journey has been filled with many frustrating experiences. I originally started with Slackware but wanted the benefits of a package management system. I switched to RedHat for a number of years but it was hard to standardise all my systems because things kept changing between versions and the upgrade utilities never seemed to work properly. RedHat also didn't install the more obscure libraries that are available, and some of the packages available in previous releases are no longer available. Furthermore, older RedHat’s did not have any network packaging capability such as Debian’s APT, and so installing new RPMs was always a painful experience with unmet dependencies and so forth.

Over the last two years or so, I have been testing out Debian-based systems as a foundation for a common distribution that I can use between all my systems. I want a standard configuration I can use on my laptops, desktops, and wearable computers, and rapidly deploy it when a new machine arrives. I like Debian for my development work because it has a massive number of packages – pretty much every piece of software ever written seems to be in there somewhere. The APT program makes it very simple to download new packages and resolve dependencies, making building applications much easier. Debian is available in three versions: stable, testing, and unstable. The stable release is safe to use as it has been extensively tested, but the problem is that many of the libraries and tools I require are so out of date they are unusable. Later on we will talk about some of the problems with older libraries that are relevant to this tutorial. While it is possible to install the latest libraries from the testing and unstable releases, these have dependencies on other non-stable packages and so the system ends up performing a massive update when one package is installed. This results in a system which is no longer stable and for those who are not Debian gurus it can be a bit difficult to fix up problems that do occur. Another problem with Debian is that it tends to assume a reasonable amount of knowledge with Linux in order to configure the machine to your liking. Even though I have been using Linux since 1995 and am quite experienced, configuration files vary over time and between distributions, and I’d rather have something that makes simple tasks quick and easy to perform so we can focus on the real work. I don't want to spend a week preparing a new system, I'd really prefer to just drop a CD in and install it, then add my dependencies and the system is ready to go.

2.2 Debian distributions

My initial Debian experience was in late 2003, using the testing version. I used testing because it had up to date versions of the packages I needed compared to the stable distribution. Do not even try to use stable for this tutorial, everything we are using has had large amounts of development since stable was released. I found Debian testing painful to get started with however, configuring things like the hostname for DHCP, setting up wireless, hunting down all the packages I needed, and generally tweaking little things took a lot of work and involved hunting for configuration files all over the system. I was quite surprised that these tasks were still so difficult when they could have been easily solved with a simple GUI tool.

Then I discovered Knoppix, which seemed to be the solution to all of my problems. It was based on Debian so it had all the nice packages that I liked, but it also seemed to be written for users who are not technical. I like these kinds of distributions because they have nice default configurations and simple GUI tools to configure the common things people use. Debian seems to assume you are the author of the tool you are trying to use! Knoppix is based on a mixture of testing and unstable packages which have been selected by Klaus Knopper (the distribution’s author) as being suitable for release, giving us a relatively bleeding endge system which is quite stable. When I installed Knoppix onto my IBM Thinkpad with ATI Radeon 9000, everything including sound, video, and 3D acceleration worked straight away without having to do anything – this is the way things should be! There is no reason that this has to be done the hard way anymore!

Knoppix ran well for a while, but then I started running into big problems. A few months after the install, I would go and try to install a new package I needed. This package depended on some X library, which then needed to be upgraded. By upgrading this X library, a chain of dependencies were unleashed and the next thing you know APT wants to change every package in the entire system! If you let it do the upgrade then something is bound to break and then fixing it is a nightmare. The other problem is that there was no clean way to upgrade to the next version of Knoppix when it comes out, you have to reinstall the system. Klaus didn't keep the deb files he used in an APT repository anywhere, so you couldn't go back to the old packages. Knoppix was fine if you installed the latest CD and grabbed all the packages very soon after, but over time the system becomes stale and can never be upgraded or have new packages installed. If you risk doing an upgrade you may end up with a system that is broken and doesn't work. In my line of work, my systems have to work and I can't risk things breaking. I don't want to have to keep images of my hard drive laying around – that is not the way we should be doing things with Linux!

So while I liked Knoppix a lot, it was unmaintainable for what I was trying to do. I needed something that was well supported by the community, and that I could install once on each of my machines and then upgrade easily and automatically in the future. So I went back to using Debian testing and this time decided that I would have to learn about all the obscure config files of Debian because there was no other easier way. It wasn't too bad, and now I have built up a set of notes that I use to do all my installations. Furthermore, I have create a repository of Debian packages which contain dependencies for all the various packages that I need. Now I just install a base system and then my packages on top, and Debian pretty much does everything else.

2.3 Debian limitations

The problem with Debian testing is that it is always changing. When I did my first testing install, within a few weeks all the kernel header includes for V4L were broken. Another time when upgrading some X packages, two of them contained the same file and dpkg refused to resolve the problem. These problems seem to be caused by when a file is moved from one package to another in the unstable series, but only one package makes it to the testing series. Packages move to the testing series within a few weeks of waiting if no bugs are filed against them. So the problem is that testing may or may not work for you when you need it most. Once the problem is fixed then an upgrade should fix the problem, but what if you are installing a machine in the week when the packages you need are broken? So I see this as being a big problem with Debian, and since stable is so old you can't use that either.

One thing I have been impressed with recently is the Ubuntu distribution. The founders of this project have decided to build a distribution which builds on the strengths of Debian, but with a frequent release schedule so that there is a stable target which we can build systems with. Rather than releasing a stable system every few years (which is eons in Linux development!) the Ubuntu people are doing a release every six months. These releases are designed to be stable with bugs and security patches available. I have only just started using Ubuntu, and I am very impressed with the overall system. They seemed to have learned a lot from distributions such as Knoppix in terms of ease of configuration as well, and I think Ubuntu is a distribution that will really make Debian usable for people like me. I am looking into Ubuntu further and haven't tested it for a long time, so for now this tutorial will focus on Debian testing. However, almost all the Debian examples are relevant to Ubuntu anyway.

2.4 Debian package installation

This tutorial is based on Debian testing systems which were current as of April 2005. Some of the libraries I discuss have experienced changes recently, but hopefully these have stabilised out.

When I build a new system, I only do a standard Debian testing base install with no extra packages. I try to keep my systems minimal so that upgrades are much easier and quicker to perform. I then install a series of packages from my own custom APT repository which contain a number of dependencies on various tools that are needed for my development. These dependencies are designed to work against a standard base install, and you may add extras such as KDE or GNOME without any problems.

To use my dependencies, add my Tinmith APT repository to your /etc/apt/sources.list file:

If you prefer, you may install the dependencies manually of course. I have also provided some packages for Ubuntu that I am currently testing out. Here I will describe all of the packages that I install and the functionality that each of them provides.

2.5 Other configuration

If you are not using a Debian system, it should still be relatively simple to configure your system to get the right libraries installed. As a guide, make sure you have the following software installed in your favourite distribution:

2.6 Special Firewire configuration

For some reason, many distributions do not come with the Firewire devices (particularly video1394) properly configured. Also, there were two naming convention changes between kernel 2.4.18 to 2.4.19 and libDC v8 to v9. Before you begin using your system, if you are using libDC v8 I suggest that you upgrade it immediately. This version is quite old and has a lot of bugs when dealing with multiple cameras and extra features. Version 9 and up are much better, but I do recommend that you run the absolute latest version since the developers keep changing the API in the library! Even worse, there are no version numbers in the header file to allow you to use #ifdefs to write portable code, so you should try to keep up to date. I believe now that the libDC library is stable, so hopefully there will be no more API changes in the future. In newer kernels 2.4.19 and later, the video1394 module uses character major 171 minor 16 devices, while older kernels use character major 172 minor 0. So make sure your devices are numbered like this accordingly. For newer libDC v9 code, the devices are stored in a directory /dev/video1394/* whereas in the older version only a single device is available at /dev/video1394. I have included a make_devices script in the tutorial examples which performs auto-detection, and also some common cases below:

Package Name	Dependencies
tinmith-desktop	auto-apt, bbkeys, blackbox, coriander, cvs, emacs21, rxvt, smbclient, smbfs, sudo, unzip, xterm, nano, zip, make, ssh, nmap, netcat, less, lynx, rsync, dpkg-dev, g++, csh, xawtv, fetchmail, strace, ncftp, module-init-tools, gscanbus, x-window-system-core, x-window-system-dev, gdm, xterm, manpages-dev, dnsutils, minicom, imagemagick, xvkbd, xvncviewer, x11vnc, pptp-linux, hdparm, cupsys, cupsys-client, cupsys-bsd, sharutils, arping, bblaunch, gdm-themes, screen
tinmith-devel	tinmith-intersense, libavc1394-dev, libdc1394-11-dev, libdc1394-examples, libraw1394-dev, libfreetype6-dev, libglut3-dev, freeglut3-dev, libcoin40-dev, libjpeg62-dev, libsoqt-dev, libsoqt20c102, libttf-dev, libx11-dev, libxml2-dev, libxmu-dev, xlibmesa-gl-dev, xlibmesa-glu-dev, xlibs-dev, xlibs-static-dev, libqt3-mt-dev, libncurses5-dev, libsvga1-dev, libmagick6-dev, gdb
tinmith-laptop	wireless-tools, wavemon, pcmcia-cs
tinmith-ubuntu	libasound2-dev, alsa-base, alsa-utils, alsa-oss, sox, aumix, libavc1394-dev, libdc1394-dev, libraw1394-dev, libfreetype6-dev, libglut3-dev, freeglut3-dev, libcoin40-dev, libjpeg62-dev, libsoqt-dev, libsoqt20c102, libttf-dev, libx11-dev, libxml2-dev, libxmu-dev, xlibmesa-gl-dev, xlibmesa-glu-dev, xlibs-dev, xlibs-static-dev, libqt3-mt-dev, libncurses5-dev, libsvga1-dev, libmagick6-dev, gdb, auto-apt, bbkeys, blackbox, cvs, emacs21, rxvt, smbclient, smbfs, sudo, unzip, xterm, nano, zip, make, ssh, nmap, netcat, less, lynx, rsync, dpkg-dev, g++, csh, xawtv, fetchmail, strace, ncftp, module-init-tools, gscanbus, x-window-system-core, x-window-system-dev, gdm, xterm, manpages-dev, dnsutils, minicom, imagemagick, xvkbd, xvncviewer, x11vnc, pptp-linux, hdparm, cupsys, cupsys-client, cupsys-bsd, sharutils, bblaunch, gdm-themes, screen, libopenal-dev, libogg-dev, libvorbis-dev, fluxbox, aterm, wireless-tools, wavemon, pcmcia-cs
tinmith-xchanger	dialog

The other devices such as /dev/raw1394 are typically included by default and have always worked for me without problems. More information on the firewire subsystem can be obtained from http://www.linux1394.org. My tinmith-desktop package corrects any Firewire issues for /dev/video1394 and /dev/raw1394 if they exist. Note that my scripts will not work if you have udev installed, which is a kind of dynamic device file system.

3 3D graphics

One of the core things we will discuss in this tutorial is 3D graphics programming. Pretty much every example involves the display of graphics to the user, controlled via some kind of input mechanism. XFree86 is a graphical X display server used by Linux and many other operating systems to provide a standard way of drawing graphics to the display. Traditional X Windows programs are written as clients, where they run on a server machine somewhere and then send their display commands to an X server typically running on the user’s desk. This architecture is very flexible and allows us to run applications remotely quite easily.

3.1 GLX and DRI

One limitation of the X design is that it always requires two programs to be running – the client and the server. Even when using shared memory or local communication mechanisms on a single processor, the client must still task switch with the server to render a display. Performing any kind of communication to another process involves the kernel performing switching and message passing, and both of these are very time consuming. When we need to render millions of graphical primitives to a display, the X protocol and kernel becomes a large overhead that prevents us from achieving the best performance. In these scenarios, being able to talk directly to the hardware is a must.

When Silicon Graphics (SGI) designed the 3D support into their workstations many years ago, they wanted to support the existing X protocol but extend this to support 3D primitives as well. They developed a protocol extension known as GLX which allows the encapsulation of 3D drawing commands over an X windows session. While GLX allows the transmission of 3D graphics over a network, there are overheads imposed by the network transport. One way to avoid transmitting the graphics commands repeatedly over a network is to use display lists, where the server caches geometry locally to improve performance. For geometry that is continuously changing however, direct hardware access is still needed.

SGI developed further extensions to their X server so that if it detected the client and server were on the same machine, it would allow direct access to the 3D hardware. The IRIX kernel was modified to allow user land applications to safely access the 3D hardware without risking system stability. The user land application then executes OpenGL function calls which are then used to directly write commands into the 3D hardware’s command buffer. The video hardware then draws this to the display and there are minimal communication and switching overheads.

SGI later released some parts of the source code to GLX and their direct rendering extensions, which was then used as a foundation for the Direct Rendering Infrastructure (DRI) project. DRI is used to provide OpenGL direct rendering to hardware under Linux and XFree86, using kernel modules to control access and XFree86 modules to provide the hardware interfaces. While SGI supported almost all of the OpenGL command set fully in hardware, most PC accelerator cards do not, instead relying on software emulation to fill in the gaps. The Mesa3D libraries are also integrated into XFree86 to provide full 3D rendering functionality for any features not provided by the hardware.

With GLX and DRI support under Linux, writing 3D applications using OpenGL is now very simple and easy to access by anyone with any PC and a cheap 3D accelerator card. The major difficulty is getting 3D support to work with your particular video card and distribution. The really nice part about the direct to hardware acceleration is that you can use it for writing fast 2D applications as well. Instead of using X server primitives, simply create an OpenGL window and do everything directly to the hardware using the driver features provided. A good example of this is supporting live video display: previously there were a number of extensions developed to X such as Xvideo (Xv), MIT shared memory extension (MITSHM), direct graphics architecture (DGA), etc, but none of them are standardised amongst all the drivers and still are not as efficient as direct to hardware under OpenGL. Since everyone building 3D hardware supports texture maps and OpenGL, it is a nice and portable way to write applications that are also fast. Note that whenever I refer to XFree86, I am also referring to derivatives such as XOrg as well.

3.2 3D hardware

Most 3D chipsets nowadays are quite powerful and able to pass the standard Quake3 test (ie, if you can play games on it then you should be okay). Before buying a computer or graphics card, it is a good idea to check out somewhere like http://dri.sourceforge.net and http://www.xfree86.org to find out what the latest video card support is. Compiling your own XFree86 or XOrg binaries based on the source available is not for the faint of heart however, and so if want to keep things simple and get things working you should install a new distribution or upgrade to get all the latest XFree86/XOrg packages. The other advice is that you can sometimes be better off buying a slightly older video card (don’t get the latest bleeding edge one) because odds are the developers have had some time to get the drivers out for them. Since the card manufacturers do not typically provide drivers for video cards, the DRI developers can only begin thinking about a driver once the hardware is out.

3.2.1 Nvidia cards

Nvidia produce powerful 3D graphics hardware that is capable of performing a number of complex OpenGL functionality. While the TNT2 is a reasonably old 3D design, it is still very capable and able to run many of today’s games at a reasonable frame rate and resolution. The TNT2 is great for most development and you can get them for free from people throwing them away.

The GeForce2 is a few generations ahead of the TNT2 and is capable of rendering much more complicated models. A nice feature of the GeForce2 is it is powerful enough to support real time video texture mapping – you can capture video data and load it into a texture in real time, and then render it onto a polygon. This ability is what separates cheap low end cards from the more expensive higher end cards. The GeForce2 is also a very cheap card to purchase or acquire for free from a friend since it is also not very new. The thing to realise with 3D hardware is that if you have a quality card, you do not need the latest release to get good performance – they are all quite good and you only notice the difference on the most demanding of games. For everything else it does not matter. GeForce2s and above are available in many laptops as well, particularly larger desktop replacement units. They seem to be less common in smaller laptops than Radeons, perhaps this is to do with different power consumption. I have used Nvidia chips in Dell Inspiron 8100 and Toshiba Tecra M2/M3 laptops, and they both work absolutely perfectly. I would particularly recommend the latest Tecra M3 model, it is very Linux compatible, the hardware is good quality, and is available for a very competitive price.

Nvidia cards are interesting to run under Linux. Nvidia does not publically release any documentation about their cards, and so there are no GPL drivers in XFree86 to support the 3D acceleration features. However, Nvidia do provide their own binary 3D driver that reliably supports every 3D card they have ever made. The binary driver is wrapped up so that it can compile against whatever distribution and kernel you are using, and I have tried it under many distributions with only minor problems. Occasionally you will get a kernel version like 2.6.4 or broken build tools that just won't compile the Nvidia driver properly, and if you end up in this situation then just upgrade one until you get it sorted out. Don't be tempted to play with the problem for days and days like I did unless you have to. In my case, whenever I build the Nvidia kernel module it would insist on rebuilding the entire kernel again! The final result would work, but I didn't want to have to rebuild the kernel just for this. Currently I am using kernel 2.6.8 and Nvidia driver v6111 with no problems, so if you get stuck then try this particular combination out because I know it works. In my experience, once you can get the module to compile and install, it will work forever. If you ever do an upgrade of your Debian X packages, be sure to reinstall the Nvidia driver because some of the files get modified and your X server will crash when 3D applications start up.

The Nvidia binary driver shares much of its code base with the Windows Nvidia driver, and so the performance is excellent and all features of the card such as dual head are nicely supported on even the latest hardware. I have used the Nvidia drivers for a number of years and never had any problems with them, although you do read a few people on Slashdot complaining about them occasionally. Since the drivers from Nvidia are closed source, you cannot edit the core code which does all the magic of talking to the 3D card. However, there is a wrapper around it which people have modified to support new kernels and so forth, so it can be changed in some cases if needed. It can be a bit of a pain installing binary drivers, particular under Debian which seems to go out of its way to make this more difficult that it should be. I build and install my own kernel, and then just use the standard install file from Nvidia to build the driver. I don't bother with the Debian packaged version for now, since it only works with the standard Debian kernels which are not good enough for what I do. Anyway, back to the subject of Nvidia and its closed source binaries. When I have presented at conferences previously, you always get one or two people who get mad at you for using binary only code and so forth. I have been developing 3D applications for a number of years, mostly on mobile laptops, and only in the last few years has it been possible to get any decent 3D chipsets at all. So when we purchased our first Dell 8100 laptop with 3D support, Nvidia was the only powerful hardware which had working drivers available. The equivalent Radeon chips of the time still had very beta and buggy drivers, and were not useable for our work. The people who fund our work want to see demonstrations, and so we had no choice but to use the Nvidia drivers. In the next section I will discuss ATI cards though, which have now matured to a better state, although I still think the Nvidia drivers are much more reliable and stable.

3.2.2 ATI cards

ATI are the main competitors to Nvidia and produce 3D graphics hardware with similar capabilities. I have probably slightly more experience using Nvidia chips than ATI, although I have quite a few ATI based laptops and I will discuss these experiences. From what I have read, a Radeon chipset is slightly better than a TNT2, and a Radeon 7500 is slightly better than a GeForce2, so if you want to do intensive operations like live video texture mapping then go for at least a Radeon 7500 or above. Other older cards like Rage128 are many generations old and you should probably avoid these if possible, especially the Rage (Mach64) chipset which is even older and less supported. On desktops, Radeon cards are very cheap to buy, and Radeon chips are common in many laptops now too.

Getting a Radeon 7500 and up in a laptop is reasonably common these days, with many smaller units coming with them now. In the past I have found that sometimes the type of chipset contained in a laptop is not reported accurately by the manufacturer, so you should check http://www.tuxmobil.org and read the feedback people post before purchasing. Be aware that not all Radeons are supported any more, so it is very important that you chose wisely!

The DRI project directly produces their own drivers for the ATI Radeon cards, supporting Radeon models up to the 9200. The 9200 is the last of the R200 series, and all models upwards are R300 models and are not supported. ATI is not releasing the specifications for the R300 models, and the open source driver will not do 3D acceleration on these systems. XFree86 only mentions this briefly in its output logs however, and so you can think your system is using acceleration when in fact it is not. ATI have released a binary driver to support all of their Radeon models, and this is required if you want to get 3D acceleration on an R300 card in particular. The ATI binary driver is nowhere close to the Nvidia driver in terms of quality and ease of installation, and I am undecided as to whether the DRI or ATI driver is better. They are both average drivers with their plusses and minuses.

We have used the DRI Radeon driver on an IBM Thinkpad T40p and a Dell Inspiron 600m, which contain an ATI Radeon FireGL 9000 and Radeon 9000 respectively. Many distributions like Knoppix will get the 3D acceleration configured automatically, which is very nice. Both the DRI and the ATI driver generally work, but under a heavy rendering and processing load, we have observed the rendering slowing down and then panicking and crashing the kernel. We are not sure what causes these problems, but they are worrying because they can happen at any time and this is not acceptable for the systems I build. On the contrary, I have never experienced similar reliability issues with the Nvidia driver. There are a number of other problems with the Radeon drivers as well, mainly the DRI version I am most familiar with. Some software including my own renders flashing textures and there seems to be no fix for this problem. It seems almost impossible for the driver to control how the LCD and external displays are used. If the BIOS decides to send all output to the monitor on boot up then the LCD screen can not be forced on with an XFree86 configuration file option. If you can trick the BIOS into booting the system with both displays on, the DRI driver won't support 3D acceleration in dual-head mode, further limiting its usefulness. I have also had problems with the frame buffer kernel support, if the modules are loaded at startup then the DRI driver cannot enable 3D acceleration. It seems that under Linux there is no way to prevent a module from being loaded without deleting the module or recompiling the kernel.

As you can see, I have had a frustrating experience with Radeon chips. In general, I am a bit disappointed in the driver quality, especially when years ago people were telling me about how stable they were! The Nvidia driver is superior in every way in that it doesn't have any bugs I can see and supports full external display control from the XFree86 configuration file. I will never buy a Radeon based laptop ever again as long as I can find a suitable model with an Nvidia chipset inside it.

3.2.3 Other cards

If you are not using one of the big two graphics chipsets (Nvidia or ATI) then support varies depending on the chipset that you have. While the chipsets by Intel that are integrated into motherboards are okay for simple 3D applications, they are not powerful enough to handle the live texture map example described previously. The integrated Intel graphics aren't too bad though, and they would be my third place recommendation if you can't get one of the big two in your laptop.

If you are out to buy a video card for your computer, I would recommend you stick with Nvidia or ATI – a lot of people use them, lots of testing has been done, and you can get help on various forums. I would steer clear of other brands because they are typically much cheaper and more cut down. This is fine for Windows where the drivers are supplied, but for Linux this gets risky and its not really worth the trouble.

Many years ago there used to be a project called Utah-GLX which was designed to provide 3D on very old cards such as ATI Rage and 3dfx Voodoo. This support works quite well on these chipsets, but the hardware is so old that it really isn’t suitable for the work we will perform in this tutorial. If you have a desktop then I suggest you get another card like the ones recommended above, although if you have a laptop you don’t really have a choice and so you can use this to get it working. I personally have used Utah-GLX with XFree86 v3.3 on an ATI Rage Mobility laptop and got it to work quite well with standard OpenGL applications, although the live video texture map slowed performance down to about 2 frames per second.

In summary, get yourself an Nvidia or an ATI card. The low end models are basically free from people throwing them out, and so there is no reason not to have one in your desktop. On a laptop you are stuck with what you have, but if you are purchasing a new machine then try to get a good 3D chipset.

3.3 Configuring X using xchanger

I am amazed at the number of Linux systems out there which have 3D hardware installed but do not have it configured correctly. Many distributions don't write out configuration files that support 3D, and XFree86 gives hopeless debugging information that doesn't tell you why it is disabled. Forums are full of helpful people who know nothing, giving out incorrect advice that just makes things worse. No one seems to know anything about it, and so I want to talk about it in this tutorial to help everyone out.

The thing I find most frustrating is that by now you would have thought that generating a configuration file for XFree86 would be a solved problem. There are thousands of example config files floating around on the Internet when there really doesn't have to be. Furthermore, it seems like it is still impossible to change the display resolution or configuration on the fly without editing configuration files by hand. The last thing you want to do when trying to connect your laptop to a projector and playing with many resolutions is to have to edit configuration files! While there are extensions such as XRANDR, not all servers support them, and usually not with 3D acceleration at the same time either.

As a temporary measure until these problems are addressed properly, I have written a GPL'ed program called xchanger. It takes in template configuration files and uses the C preprocessor to substitute in resolution and display configuration information, generating an output file suitable for XFree86. There is a small program written in shell and using dialog which allows the user to pick the resolution of their displays, and pick the layout of the displays. I have supplied template files for Radeon and Nvidia systems that I have tested on every machine that I have. If you follow my advice and buy only Radeon or Nvidia systems then you will be able to use my standard configuration files and have working 3D support with no problems. I would like to encourage everyone to standardise on a configuration file that works for as much hardware as possible, so that the majority of hardware will work to its full potential with no messing around by the user.

I have provided a copy of xchanger as part of the example files included with this tutorial. There is also a package named tinmith-xchanger you can install which will automatically configure itself on your system. The xchanger tool is released under the GPL, and I would like to hear about what others think about it. I would also be interested in letting someone else maintain xchanger for Debian since I'm not really an expert at building proper packages, so please contact me if you are interested.

3.4 Programming OpenGL

This tutorial will not cover too much about how to program OpenGL programs themselves. We will focus on the Linux specific issues and leave you to find out more about OpenGL from other places. The best place to learn OpenGL is from what we graphics people call “The Red Book”. Pretty much everyone I know who does GL coding learned how by reading this book, and it is very well written with many excellent examples.

By Mason Woo, Jackie Neider, Tom Davis, Dave Shreiner, OpenGL Architecture Review Board

If you Google for ‘opengl red book’ then you will find many references to it, and I have linked to a version that you can download yourself. There is another reference manual called “The Blue Book”, but this one is just a dump of all the man pages and is not very useful. If you use Google or visit http://www.opengl.org/developers/documentation there is a lot of free stuff on the web you can use to look up what the functions do. The red book also comes with a wide range of examples that are available online at the location http://www.opengl.org/developers/code/examples/redbook/redbook.html.

There are many free tutorials available on the web. While many of these are written by Windows people, they are mainly written to use the GLUT toolkit which means the source code is portable across most platforms, and so are still useful. Of all the web sites I’ve seen, probably the most useful is Nate Robbin’s http://www.xmission.com/~nate/tutors.html site. Nate has built example tools which allow you to interactively tweak OpenGL function parameters to see what will happen without having to write your own demo application. These tools are very useful to work out what values to use very quickly and easily.

3.4.1 Libraries

When you write applications for OpenGL, there are a number of libraries which you may or may not use depending on what kind of functionality you need.

libGL is the core OpenGL functionality, with all functions named something like gl[NAME]. On the original SGI implementation, every one of these functions was fully implemented in hardware for optimised performance. This library provides primitive rendering, texture mapping, and matrix transformations. When you want to perform hardware accelerated rendering under X11, your driver (DRI, Nvidia, etc) will provide a complete libGL shared library object for you to use. If you do not have one then a software emulation like Mesa will be used.

libGLU is the OpenGL utility library and sits on top of the core GL functions, with all functions named glu[NAME]. There are some functions for assisting with camera placement, and to handle advanced functionality such as tessellating triangles. A triangle tesselator is a program that takes an arbitrarily sized polygon and breaks it up into triangles for you. I have found that the standard GLU tessellator functions provided by Mesa are very buggy and fail to successfully tessellate any kind of complex triangle. To get around this, I compiled up a copy of the SGI tesselator which has been released as open source, and this works much more reliably.

libGLUT is the OpenGL utility toolkit and sits on top of the code GL functions. While the GL functions allow code to render to the hardware, no provision is provided to configure a window on the display or interact with the rest of the window system. The GLUT library is very nice because it provides a set of commonly used functions for opening the display and handling user input. The GLUT library is available on most architectures (Linux, Windows, MacOS, etc) and so if you write your code to be portable and only use GL, GLU, and GLUT calls then it will work on any of these platforms. In general, if you are able to write your application using only GLUT calls, do it because it will save you a lot of grief. You can do everything yourself but it can be quite painful and there isn’t too much documentation for it.

GLX is the interface that we use if we want to write OpenGL applications specifically for X servers. While the GLUT toolkit is nice, it cannot open multiple windows and does not provide the ability to access X server specific structures (because it is designed to be generic across platforms). So if you are going to write an application optimised for best performance with your own custom software architecture, GLX is definitely the way to go. Note that there are other ways of embedding OpenGL into your applications. GUI toolkits such as Qt, GNOME, and Motif all provide widgets that you can write OpenGL to. Note that these widgets were written using the GLX interface to fit with the rest of the toolkit. The strange part with GLX is that it is very hard to find much information about this, all the docs you read either tell you to use GLUT or use an existing widget. I have included an example below which opens up an X window but also configures it for OpenGL drawing as well.

gen_fatal ("Could not open connection to X server on path [%s] - check the DISPLAY variable", XDisplayName(NULL));

/* Now create a GLX context on the server - do not share lists (NULL), and use direct

/* Map the window to the display, waiting for it to appear before continuing, if we do

/* Get the context, and make it the active one for OpenGL commands. This function

/* Add event listening to the connection so we know when things happen in the server */

After this code you can then execute standard OpenGL commands. The next thing to do is flip the display when we have completed drawing because we are using double buffered mode. To do this, we have to make sure to tell GLX to flip over the buffers:

gen_fatal (“OpenGL error code %d detected, graphics system has failed - %s”, glerror, gluErrorString (glerror));

3.5 Live video display

OpenGL is a very powerful graphics library that is capable of performing pretty much everything you could ever imagine. Since you can perform anything Xlib can do, but with arbitrary transformations and viewpoints, you can write your applications in pure OpenGL like I have been doing lately. You can mix Xlib and OpenGL commands but you must ensure you flush the pipeline so they can be kept synchronised properly. Back to using OpenGL though, the Red Book goes into a lot of examples showing the kinds of things you can do with OpenGL, and so here we will cover an example of something a bit different and less mainstream.

As described previously, there are a number of X extensions for displaying live video, but there is no real portable interface supported by all video cards. OpenGL supports two different ways to send large amounts of pixel data to the card very quickly. The first method is to use the glDrawPixels() function call. This method takes in an image and maps it directly to the pixels on the display, and in theory should work quite well. In practice, this function causes the rendering pipeline to stall because it must flush all existing operations currently in progress. This function is not normally used and so hardware manufacturers typically do not optimise it either. One thing that is highly optimised is texture mapping however, and so this is the recommended way to draw images quickly in OpenGL. So we load the texture into the video card, and then draw polygons with the image as its texture. By drawing a square onto the display we can achieve an object that looks just like the texture was copied to the display, but the operation takes advantage of the optimised texture rendering hardware. Using texture maps has the following advantages over glDrawPixels and standard X windows methods:

So as we can see, using OpenGL for video display is both easy to use and very powerful. Applications such as MPlayer and ARToolKit have display code that is capable of using this rendering technique. One catch with using textures in OpenGL is that there is a restriction that the texture must be a power of two in each dimension. So if you have a 320x240 image from your camera, you must supply to OpenGL a texture image which is rounded to 512x256. By forcing this requirement OpenGL is able to accelerate texture performance further. While this limitation may seem difficult to deal with, the important thing to realise is that a scaled image like this is only required for the first frame! We don’t really want to implement our own function to pad an image as each frame comes in from the camera, because we aren’t gurus at writing optimised image handling code. Instead, OpenGL supplies a function glTexSubImage2D() which allows us to replace the existing image with a sub-image that does not have to be a power of two. This function then magically copies the image data over and fills it into the texture correctly, and we will assume that the people who wrote this function did a good job of it. It would be nice if all the texture functions didn’t have this restriction but we will have to live with these decisions, I am sure they were made for good reasons. With the above exceptions explained, we are now ready to explain the process of mapping live video to a textured polygon. We will explain video capture in a separate section later on, that is another problem of its own.

Let us assume for this example we have a 320x240 input stream which is 24-bit RGB formatted. The first step is to perform a one time initialisation to get started, and then we will explain the rendering part which is repeated every time the frame is redrawn. I have written a demo program that draws some polygons with live texture mapped video, and it is included in the demo section of my area on the conference CD.

3.5.1 Initialisation

We will assume that everything else in OpenGL has already been configured before this point. The only major trick is to remember to turn on an appropriate texture mode somewhere during startup, so we do this:

The first thing we must do is turn on texturing and then make sure we turn it off once we are done here because otherwise it will affect any other primitives drawn afterwards:

We now need to create the padded buffer which is rounded to the nearest power of two (512x256). The contents of the buffer are unimportant but I have initialised it to all 0xFF which will make it white if rendered. This buffer is supplied to OpenGL so it can initialise itself, and we can destroy it after because OpenGL makes a copy of the buffer rather than only a reference:

Now we need to initialise a texture map in the video card’s memory. Note that the card allocates a handle called texid that we will need to keep for later use when we want to draw with the particular texture. Whatever you do, make sure you do not run this code more than once per texture (ie, don’t put it in the render loop) because you will run out of memory. Think of the glGenTextures() as a kind of malloc() call that needs to be freed up later:

/* Set some parameters for the texture. We want it to be tiled and also to not perform any special filtering to improve performance */

Now we will load in the blank image we previously created. An interesting thing to note here is the use of GL_RGBA8 even though we specify GL_RGB as the format. What does this mean you may ask? The second GL_RGB is the format of the input image from the camera and tells OpenGL how to interpret the data we have supplied. The first GL_RGBA8 is the internal format to use on the video card however. GL_RGBA8 represents the image with transparency support and 32-bit padding internally in the video card memory, and gives the best performance and texture quality on hardware I have tested it on. You can use others but watch out for performance or quality problems:

glTexImage2D (GL_TEXTURE_2D, 0, GL_RGBA8, round_width, round_height, 0, GL_RGB, GL_UNSIGNED_BYTE, round_buffer);

Now we finish off and disable texturing and we can do any other initialisation work:

3.5.2 New video frame

When a new video frame arrives, we need to capture the video frame, enable texturing, and then pass the image on to the video card:

/* Load in the new image as a sub-image, so we don’t need to pad the image out to a power of two.

Note that some libraries return video in BGR format, so you may need to replace GL_RGB with

glTexSubImage2D (GL_TEXTURE_2D, 0, 0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE, in_data);

For those of you who may be considering the use of threads to keep the video capture and render loops separate, be warned that this is dangerous. Most Linux libraries (including OpenGL and Xlib) are not thread safe by default and you will cause big problems if you try to make calls from two threads simultaneously. The best way to implement threading is to have the frame capturing code copy the new images into a memory buffer, and then have the rendering loop be the only code which makes OpenGL calls. You can try to implement locking but these introduce performance overheads because either the kernel has to get involved, or you have to waste CPU cycles waiting for spin locks to clear.

3.5.3 Render

The render code may be run even when there is no new camera data available. The user might want to change their viewpoint and so we may need to refresh the display even if the video has not changed. To do this, we do a standard OpenGL texture render operation:

/* Any polgons we draw from now on up to glDisable() will have the texture mapped to their surface */

/* Draw a square of size 1.0x1.0 with the video exactly mapped to fit it, taking into account

the padding that was required to make the image a binary power of two in size. */

/* We can draw any other polygons here if we wanted to as well, this is good because we can keep

the same texture loaded into memory without requiring a new one to be swapped in. */

3.6 Scene graphs

OpenGL is designed to render primitive shapes such as lines and polygons with texturing and lighting effects. OpenGL is a very low level graphics toolkit and while it is possible to write complete applications using it, the programmer has to supply much of their own code to handle common tasks. You can think of OpenGL as being equivalent to Xlib in terms of functionality, and most people do not write their applications directly using it. Instead they use a higher level toolkit such as Qt (KDE) or GTK (GNOME) which provides more useful functionality such as widgets.

The most powerful high level programming library for OpenGL would have to be Open Inventor, again developed by SGI. This library was originally designed back in 1992 to provide a more functional environment for programming complex 3D applications. It provided features such as a powerful object model in C++, a scene graph, user interface widgets, object selection, engines for animated objects, and a file format which forms the basis for VRML. Even though it is quite old, the design of the toolkit is still excellent and there is nothing that even compares in terms of functionality and design – the people at SGI who designed Inventor and GL really knew what they were doing.

To provide a more detailed description, a scene graph is a description of an environment that also contains relationships between various objects. To represent a human in a scene graph, the body of the human would be the root node, with the head, arms, and legs attached to it. The nose and ears are attached to the head, and the hands and feet are attached to the arms and legs respectively. A scene graph is able to calculate transformations so that if you move the body, the rest of the parts will move with it. If you rotate the head for example, the nose and ears will move to match this motion. Scene graphs are very useful when representing complex mechanical systems and make rendering them very simple. Another bonus from using a scene graph is that the renderer is able to cache the results of previous runs to improve performance on the renders later on. OpenGL supports a number of features such as display lists and vertex arrays which can be used to improve performance.

3.6.1 Coin3D

SGI have recently released the source code for Inventor to the public, and groups have taken this code and cleaned it up. A commercial group have developed their own version of Inventor named Coin3D from scratch and released it as GPL code to the public. The free version may be used in GPL applications at no cost but commercial applications without source code require a license fee to be paid. A number of my colleagues who use Inventor all swear that the Coin version of Inventor is much better than the SGI sources (less bugs, less problems, etc) and so you probably should have a look at Coin first. There is a book called “The Inventor Mentor” by Josie Wernecke which describes all of the functionality of Inventor, but unfortunately other documentation seem to be a bit hard to come by. Coin have a complete generated set of documentation available on their web site, but it is more to be used as a reference and there are no examples. You can however download the collection of the examples from the Inventor Mentor book, and these are probably the most useful.

The Coin libraries have a number of nice features apart from the standard Inventor object collection. It has the ability to read in VRML files and natively support them in the scene graph, so you can use it as a generic VRML parser. Secondly, it is possible to use only the rendering code (and not the user interface widgets) and use Coin as your renderer in your application. This requires a bit of messing around but is possible and means you can use Coin instead of other libraries such as OpenVRML. When I was building VRML support into my application, I found that OpenVRML was not documented enough to be able to work out how to embed it into my existing scene graph, whereas Coin worked very easily.

/* This is my special hacked callback function which forces the solid flag to be false to ensure that

SoCallbackAction::Response adjust_vifs (void *, SoCallbackAction *, const SoNode *node)

/* We can safely cast the object because Inventor wouldn't have put us here otherwise */

/* Set the attributes we need - just turn off solid and that ensures we have no backface culling */

Abstract