history - Why was the size of the solar system not defined by Mercury Transits?

It is indeed possible to measure the AU using transits of Mercury, and Edmund Halley tried to do just that in 1677. However, there are two advantages to a transit of Venus. The first is that during the transit, Venus is only 0.28 AU from Earth, whereas Mercury is about 0.7 AU away. This makes the parallactic effect twice as big. The second difficulty is that you need to measure the time between second contact and third contact (i.e., the first and last time that the planet is entirely in front of the Sun). Mercury is so small that it is hard to determine when second and third contact are. (In fact the limiting effect for the measurement of the AU during the transit of Venus was the so-called "black drop effect", which made it impossible to measure the times of second and third contact to a precision less than about ten seconds.)

A former professor of mine from when I was in graduate school has a really excellent writeup of the history of these measurements:

http://www.astronomy.ohio-state.edu/~pogge/Ast161/Unit4/venussun.html