Here are some guidelines I've found extremely useful when writing math libraries for game development, as well as the odd bit of code. Most of these guidelines are C++ specific, though the rationale may make sense in other languages as well.

Barely Objects

First off, if you're the type that has Strong Views about OO and cringes at the sight of public fields, you need to take a deep breath and brace yourself, because the first rule of math types is to work with them on as low a level as you can manage.

Public Fields

Please, please, please don't ever write the following:

struct vec2
{
    float x() const { return x_; }
    void x( float v ) { x_ = v; }

    float y() const { return y_; }
    void y( float v ) { y_ = v; }

private:
    float x_, y_;
};

As lovely as encapsulation is in most instances, a low-level math library simply isn't the place for it. There are several reasons for this:

Performance

Foremost is performance. Now, yes, the compiler would inline and optimize out the method calls, so you may think I'm being silly and old-fashioned, but I'm really not. Remember, you're going to be making debug builds quite often. In fact, you'll probably spend more time running debug code than you ever do playing your game's actual release build. And compilers don't inline in debug builds.

I've seen object-y math libraries (and the STL, but that's another rant) lead to debug builds that take fifteen minutes to process a data set that the release build crunches through in less than one. This is not a joke, you will go insane dealing with this sort of slowdown when the bug you need to trap in the debugger only happens at the very end of processing.

I've even heard one horror story where the team ended up with a game that was simply unplayable unless compiled with optimizations enabled, which meant that they had to debug the whole thing basically using printf.

Math code is going to be found in every corner of your game. Save yourself the pain up front and skip the nice OO-y accessors.

Speaking of Debugging

And while we're on the subject of debugging, you're going to be stepping through your code a lot. Vector fields are going to be accessed all over the place, often right in the middle of complex expressions. Take the following:

float bad = madness( a.x() * b.x() + a.y() * b.y() );
float good = sparta( a.x * b.x + a.y * b.y );

So there you are, in your debugger, stepping along when you hit one of those two lines of code, and you want to step in. Problem is this: you can't step into madness with your "Step In" command without first stepping into x() and y(). Twice. For each of a and b.

Stepping into sparta, on the other hand, is quick and painless.

(This is another one of those things that I really hate about the STL.)

Data Layout Should be Public

Your math code is probably going to have to interact with half a dozen other libraries. Each of those libraries will likely define its own math types. And those math types are almost always going to be implemented in the obvious way. Take a look at the following:

struct vec2 { float x, y; };
struct MyVec2 { float X, Y; };
struct CrazyVec2 { float x; float y; };
struct Float2 { float x, y; };

All of those vector definitions make their data members and their layout part of the public interface. That's incredibly useful because when your game code hands you an array of vec2 and your physics middleware wants an array of Float2 you can be assured that simply casting the pointer will work. You can also be sure that things like the following will work:

struct point
{
	vec3 location;
	vec3 normal;
};
 
//...
 
point *points = //whatever
 
glVertexPointer( 3, GL_FLOAT, sizeof( point ), &points[0].location );
glNormalPointer( GL_FLOAT, sizeof( point ), &points[0].normal );

Fused Operators Rock

Sometimes there's just some weird manipulation that you need to do somewhere in your code. You end up with a bit of inline per-component math just sitting there in the file. And this...

v.x *= a + b * sinf( t );

...is a heck of a lot cleaner and easier to read than this...

v.x( v.x() * (a + b * sinf( t )) );

...especially when the expression on the right turns into a real monster.

What's There to Encapsulate, Anyway?

Encapsulation is typically only important when the details of an implementation might need to change. But these are math types. Their definition is pretty damn fixed. It's not likely that someone's going to publish a proof tomorrow that vector addition actually involves multiplication by \(\pi\).

These implementations are really never going to change, except to make use of SIMD instructions on certain platforms. I've yet to really see a case where that changes the type or order of the vector elements where you'd be able to keep an encapsulated interface untouched, so I don't think this is much of an issue.

Constructors! Can Live Without 'Em!

Putting constructors in these types has two down sides, so I tend to avoid them. Note that there is work in the new C++ spec to fix some of these issues, however game developers often end up targeting old half-supported compilers. It's going to be years before the standard is available on all platforms, so here goes...

Default Constructors

I'm assuming we're all on the same page when I say that if you're going to put constructors on these types, then they should have a default constructor which does nothing, too. That's just basic mindfulness when dealing with performance-central code...

Unions

Anyway, the first real issue is that types with constructors can't go into unions. And types which contain types with constructors have constructors, too. And unions are useful. There's really no reason that a core library which will turn up everywhere in a project should make it impossible (or at the very least tedious) to use unions anywhere.

Overloads

The second issue is that of overloads. See, it's really nice to be able to initialize a vector's components to all be the same value (useful for specifying the origin or uniform scales), but it's also really nice to be able to initialize the vector using a pointer to some floats (which will pop up all over the place when dealing with libraries). This leaves us with a slightly irritating problem:

struct vec2
{
	float x, y;
	vec2() { } //no initialization!
	/* 1 */ vec2( float xy ) : x( xy ), y( xy ) { }
	/* 2 */ vec2( float x, float y ) : x( x ), y( y ) { }
	/* 3 */ vec2( const float *v ) : x( v[0] ), y( v[1] ) { }
};
 
//...
 
vec2 v = 0;

Which constructor did we just call? It looks like we called #1. However, we didn't. We got a compile error instead, because the literal `` is an int, which does not match any of the constructors but implicitly converts to both a float and a float*, leaving us with an ambiguous match between #1 and #3.

Now you get a nice compiler error, and you can go in and trivially fix the code by replacing the `` with a 0.0F, and everything's fine, yeah? The ambiguity's mildly annoying, but totally harmless...

...except on a compiler (which will not be named) where it matches #3, crashing the program when that constructor tries to read from a NULL pointer.

A few other "ambiguous match" issues pop up as well with the math types which implement automatic conversion operators.

So What Do We Use?

So what do we use if not constructors? Two answers spring immediately to mind: C's initializer syntax, and some simple inline functions to cover the cases where it can't be used (basically when all you want is an rvalue). The inline functions also have the advantage of having their own names, which lets us split up the sometimes large number of overloads into something more manageable, which lets us be lazy with our literals without leading to silliness when resolving overloads.

Note that I'm making a bit of an exception to my "inline functions are bad" rule up above. The reasoning is simple: initializers aren't going to be called anywhere near as often as the fields of the type they return will be accessed. If they're going to slow our debug build down, it's going to be in a few well-defined locations, rather than suffusing the whole codebase with an uncontrollable number of small cuts. These functions do suffer from the "debugger step" issue I pointed out above, but they're no worse than constructors in that regard.