zoology - Why is there a difference in the rotation of the tail fin in fish compared to marine mammals?

While fish tend to move from side to side (lateral undulation) for which a vertical tail makes sense, the land ancestors of marine mammals had their limbs under them and so their spines were already adapted to up and down movement (dorsoventral undulation). When these animals moved to marine environments, they continued up and down movement in their swimming, for which a horizontal tail makes sense.

(The wikipedia article on fins gives some more detail, and links to this webpage on Berkeley.edu. A paper by Thewissen et al. suggests that for cetaceans, dorsoventral undulation as a swimming strategy came first, and the horizontal tail evolved later.

---

More detail:

In a third example beyond fishes and marine mammals, the icthyosaurs and other aquatic reptiles developed vertical tails, even though like marine mammals they evolved from four-footed land animals. This may be because the legs/spines/gaits of land reptiles differ from those of land mammals, so that the earliest icthyosaurs swam with lateral undulation, as reflected in their spinal modifications.

This blog post by Brian Switek gives a really superb run-down on the issue, with figures and citations. I'll quote this part which deals with the dorsoventral undulation theory:

...mammals and their relatives were carrying their legs underneath their bodies and not out to the sides since the late Permian, and so the motion of their spine adapted to move in an up-and-down motion rather than side-to-side like many living reptiles and amphibians. Thus Pakicetus[the pre-cetacean land mammal] would not have evolved a tail for side-to-side motion like icthyosaurs or sharks because they would have had to entirely change the way their spinal column was set up first.

Switek goes on to to talk about exceptions in some marine mammals:

At this point some of you might raise the point that living pinnipeds like seals and sea lions move in a side-to-side motion underwater. That may be true on a superficial level, but pinnipeds primarily use their modified limbs (hindlimbs in seals and forelimbs in sea lions) to move through the water; they aren’t relying on propulsion from a large fluke or caudal fin providing most of the propulsion with the front fins/limbs providing lift and allowing for change in direction. This diversity of strategies in living marine mammals suggests differing situations encountered by differing ancestors with their own suites of characteristics, but in the case of whales it seems that their ancestors were best fitted to move by undulating their spinal column and using their limbs to provide some extra propulsion/direction.