How is right-handedness ensured for coordinate systems in dimensions greater than three?

First, from the mathematical point of view.

Say we have an \(n\)-dimensional vector space over the reals. Any two ordered bases of \(V\) are related by a unique linear transformation \(T\). If \(\det T > 0\) then we say the two bases have the same handedness; otherwise (\(\det T < 0\)) we say they have opposite handedness. (Note that the determinant can't be zero, because then \(T\) would have deficient rank, i.e., its image wouldn't span \(V\).) It is easy to see that this is well-defined (e.g., if \(B_1, B_2\) have opposite handedness, and \(B_2, B_3\) have opposite handedness, then \(B_1, B_3\) have the same handedness.) So we can partition the set of all ordered bases into two subsets by handedness. We then arbitrarily choose one of the two subsets and call all its members "right-handed"; the other bases are then left-handed. Each of the two possible choices is called an orientation. (So we can formally define an orientation to be an equivalence class of ordered bases.)

If \(V\) is \(\mathbb{R}^n\) then by convention we define the standard ordered basis \((e_1, \ldots, e_n)\) to be right-handed. However, this definition also makes sense if \(V\) is some vector space with no obvious canonical basis, such as the tangent space to a given point on a Riemannian manifold. When an orientation is required, it doesn't matter which one you actually choose, as long as you're consistent.

Now suppose we've chosen an orientation of \(V\). Suppose further that \(V\) is an inner product space, with inner product \(\langle \cdot , \cdot \rangle\). Say we have \(n-1\) linearly independent vectors, \(v_1, v_2, \ldots, v_{n-1}\). These generate an \(n-1\) dimensional subspace of \(V\). The orthogonal conjugate of this subspace is a one-dimensional subspace, \(W \subseteq V\). Clearly \(W\) is the union of the zero vector and two rays. Along one ray are vectors \(w\) such that \(v_1, v_2, \ldots, v_{n-1}, w\) form a right-handed ordered basis of \(V\); choosing \(w\) from the other ray would give you a left-handed ordered basis. The choice of the former ray is the "right-hand rule" induced by the chosen orientation of \(V\).

Second, from the physical point of view. (This section is longer because I know physics better than I know math.)

Physics in three dimensions makes extensive use of the cross product, and the ability to form a cross product depends crucially on the right-hand rule. From the above discussion we see that the correct "direction" for the cross product is determined by choosing the standard orientation on \(\mathbb{R}^3\) and using the procedure given in the preceding paragraph. But for four or more dimensions you can't take the cross product of two vectors anymore; you need \(n-1\) vectors, where \(n\) is the number of dimensions. So what do we do?

Well, the cross product in three dimensions takes two vectors and yields a third vector. Our generalization in the previous section suggests how to take \(n-1\) vectors in \(n\) dimensions and get another vector. (We haven't actually dealt with the magnitude, but I'll skip this detail since it's not important.) But the formulation of physics in four and higher dimensions involves a different generalization of the cross product. It takes two vectors, but the result is not a vector. This generalized cross product is called the wedge product, and it yields a bivector.

You might wonder why we don't see bivectors in physics in three dimensions. There are two reasons for this. First, in three dimensions and three dimensions alone, there is a correspondence between bivectors and vectors called Hodge duality. (In \(n\) dimensions, a \(k\)-vector is Hodge dual to a \(n-k\)-vector.) This is what allows us to pretend that objects obtained by taking cross products, such as the magnetic field and the angular momentum, are actually vectors. But in \(n\) dimensions, a bivector requires \(n(n-1)/2\) independent components to represent, and this equals \(n\) only when \(n = 3\). So in higher dimensions we are forced to confront the fact that the magnetic field really is not a vector, nor is the angular momentum. (Thus, there really are \(n(n-1/2\) conserved components of angular momentum in \(n\) dimensions. This is because there are \(n(n-1)/2\) degrees of rotational freedom, and invariance under each one gives a corresponding conserved quantity à la Noether's theorem.)

Second, in physics we usually represent a bivector as an antisymmetric tensor of type (2, 0). The angular momentum, for instance, would be defined as \(L^{ij} = r^i p^j - r^j p^i\). This notation has the advantage of being easier to calculate with, but obscures the intrinsic, coordinate-free existence of bivectors (which live in an algebra called the exterior algebra).

Let's take a look at the magnetic field again. If it's now no longer a vector, how do we deal with it? Well, it's not a vector, but we know we can get a vector from it, in the Lorentz force law,

\[\mathbf{F} = q(\mathbf{E} + \mathbf{v} \times \mathbf{B})\]

Unlike \(\mathbf{B}\), the force \(\mathbf{F}\) is a real vector. We got it from taking the cross product again (remember that you have to take a cross product when you calculate \(\mathbf{B}\) using the Biot–Savart law). Somehow taking the cross product twice makes everything right again. You might think that the solution, in general, is to take the wedge product of the velocity and the magnetic field (now a bivector). Unfortunately, this doesn't do the right thing. It'll give a trivector (or an antisymmetric tensor of type (3, 0)). This is because the cross product in the Biot–Savart law and the cross product in the Lorentz force law are actually different! The correct generalization of the latter is not the wedge product; instead, it's the interior product. This corresponds to the contraction of an antisymmetric tensor with a vector. (You'll have to lower the index on the vector, of course.) Taking the interior product of a 2-vector with a 1-form gives a vector, which corresponds to a 1-form under the canonical isomorphism induced by the metric (the musical isomorphism). (Likewise, taking the interior product of a 2-form with a vector gives a 1-form.)

Notice that handedness has disappeared from the picture entirely! You do not need to choose an orientation in order to define the angular momentum or the magnetic field, when you regard them as bivectors. The orientation is only needed to form the Hodge dual, which is what we do in three dimensions to represent bivectors as vectors. When we don't do this, we don't need a right-hand rule at all.

This information is all captured in the covariant form of the Lorentz force law,

\[\frac{dp^i}{d\tau} = q F^{ij} u_j\]

(My roommate hates index notation, but hopefully it's now clear why it's used: you don't need to understand differential geometry to make sense of it!)

But wait, there's more! A vector obtained through Hodge duality isn't actually a true vector, since it depends on orientation. It's actually an object called a pseudovector. A pseudovector is an object like a vector, except that in addition to transforming contravariantly with a change of basis, it also undergoes an inversion if the new basis has opposite handedness. (Picture a car travelling north. The angular momenta of its tires point west. Notice that if we take the mirror image of this car through a plane perpendicular to the east-west axis, then the angular momenta of its tires still point west! This is because they were flipped by the mirror image, but then they flipped again since they're pseudovectors.)

In classical physics, all pseudovectors (and pseudoscalars, and pseudotensors, which are defined similarly) are produced by Hodge duality. In index notation, taking the Hodge dual is contracting with the Levi–Civita tensor, modulo a multiplicative factor. The idea is that we choose some ordered basis we consider "right-handed", and define the Levi–Civita tensor with the appropriate sign such that contracting it with this basis gives a positive number. In such a way we determine the sign of the Levi–Civita tensor and hence the handeness convention. The generalization of the cross product of \(n-1\) vectors to give a vector is then contraction of the \(n-1\) vectors with the Levi–Civita tensor; hence the sign of the Levi–Civita tensor determines the sign of the "cross product"—which is now exactly as we expect, since we know the cross product's sign depends on orientation.

Note that in index notation we see at a glance which objects are tensors and which are pseudotensors—whenever you contract tensors with the Levi–Civita tensor, they become "tainted" by handedness.

Now, there are some phenomena in physics that genuinely do depend on orientation. In particular, the weak interaction is not symmetric under parity. This prevents us from actually doing physics in a "handedness-free" manner, as I suggested in my discussion of the magnetic field. In the quantum theory of the weak interaction there is an object that serves a similar purpose to the Levi–Civita tensor, called the fifth gamma matrix, \(\gamma^5\). Again, the arbitrary choice of sign of this matrix determines handedness, and objects produced by contracting spinors with \(\gamma^5\) are pseudotensors rather than true tensors.

This is an awful lot of text, and I'm afraid you'll only be able to understand it at first glance if you already know what all the words mean. (If that is the case, then hopefully I've given a clear explanation of how to apply the concepts to physics.) If not, consult a textbook on differential geometry.