natural selection - Why do only two sexes exist for animals?

To get a non-circular answer to why humans and other mammals have only two sexes, it's helpful to take a look at our evolutionary history. While mammals possess several adaptations to a terrestrial life cycle, including internal fertilization and gestation, which require substantial anatomic specialization between males and females, these are all secondary features that evolved long after our aquatic ancestors had acquired two distinct sexes.

Indeed, if we look at animals like fish, which reproduce via external fertilization, it's not at all obvious why they might not have more than two sexes. After all, for many aquatic animals, mating involves little more than the female and the male releasing their respective gametes into the water, where they meet and fuse to a form new zygote, which can then divide and grow into a new adult. Seen that way, there seems to be no reason why there could not be more than two "mating types", as in many fungi, such that gametes of any two distinct types could fuse into zygote.

The answer lies in the fact that the male and female gametes aren't actually that similar: the female gametes, or eggs, are typically large cells that contain all the nutrients necessary for the new zygote to develop into a viable individual, whereas the male gametes, or sperm, are tiny and produced in huge numbers. This asymmetry is known as anisogamy, and modeling its origin has been an important topic in the theoretical study of evolution.

Without going into details on the evolution of anisogamy, once it exists, it clearly forces the mating types to also split into two groups: there's no advantage in two microgametes (sperm) fusing, since the resulting zygote would lack the nutrients it needs to be viable, whereas the fusion of two macrogametes (eggs) would simply be inefficient — eggs, being large, are comparatively rare and expensive, and wasting two of them to produce only one offspring would be suboptimal even if the resulting zygote was viable. Nor is there really room in such a scheme for gametes of intermediate size: they'd be too small to fuse into a viable zygote with sperm, but too large to be produced in sufficient amounts to be effective in fertilizing eggs.

Of course, there's nothing that would stop a single adult from producing both micro- and macrogametes, but such an adult would not really be a third sex — it would just be male and female at the same time, a mating strategy known as (simultaneous) hermaphroditism, which indeed occurs relatively often in nature.

So, if pretty much all animals are anisogamous, why do fungi remain isogamous (and often have multiple mating types), then? Well, one explanation is that the main drivers for the evolution of anisogamy — sperm competition and transportation risk — don't really apply to fungi, which mate when two sessile haploid mycelia grow and come into contact with each other. Since the gametes are not motile, there's no advantage for either/any sex to produce more of them (at the cost of smaller size) in order to increase the chance of successful mating. Thus, isogamous mating works fine for the lifestyle of fungi, and having multiple mating types is then a useful adaptation to make successful mating between neighboring mycelia more likely.