Roughly how many humans are there in the Star Trek universe?

As James wrote, Earth has 9 billion in First Contact.

There might have been a slight baby boom after WWIII, but developed societies tend to move towards negative population growth, so perhaps Earth's population reaches 10 billion before this happens.

On Mars, the next largest human population, there are 133.8 million (ST: Star Charts), and Luna has 50.2 million (ST: Star Charts and First Contact) by the late 24th century. That adds up to 10.184 billion.

Other human populations:

• Deneva colony had just under a million (TOS: "Operation -- Annihilate!") by the 23rd century.


• Alpha Eridani II had a population of 12 million (ST: Maps) by the 23rd century.


• Moab IV/the Genome colony has a few thousand colonists (TNG: "The Masterpiece Society") by the 24th century.


• The Skagaran colony had 6000 humans (ENT: "The North Star") by the time they were discovered in the 2150s.


• The 37's colony had over 100,000 in 2371. (VOY: "The 37's")


• The Mariposa and Bringloid V colonies totalled only 60 (TNG: "Up the Long
  Ladder") by the 24th century.

Many other human colonies were similarly wiped out or reduced to only a dozen or perhaps a few dozen colonists by the time they were contacted by Starfleet.

Other non-destroyed human colonies mentioned but whose populations are unknown:

• Turkana IV (birthplace of Tasha Yar)


• Tendara colony (birthplace of Seven of Nine)


• Vega colony / Vega IX (where mirror universe Kirk killed 5000 parents to teach a lesson to a group of psychic children who incited rebellion)


• Earth colony II


• Earth colony 5 (Alpha V)


• Proxima colony


• Alpha Centauri colony

Then there are of course penal colonies and mining colonies, but these generally don't have very significant populations. So if you add up the confirmed population figures, that only totals about 10.2 billion. We can be generous and add another billion for the colonies with unknown populations (though, given the average figures, they're likely to total to only 100 million at most), so that gives us 11.2 billion.

The Federation has 700 colonies (website for Star Trek: TMP) by the 23rd century, and that's split between 120 worlds. Earth, we know, is one of the younger (in terms of space-faring history) species, though we're portrayed as having a greater spirit of exploration. Given all of this, and considering the small (<1 million) populations on most colonies, it's hard to imagine there being more than 12 billion humans by the 24th century.

We know that a Federation census reported 985 billion individuals in 2370. Assuming all individuals are member species (of which there should be around 183, assuming one species per member) and not Federation affiliates (of which there are 7128), then that averages out to 5.38 billion individuals per species.

So I think the 12 billion uppper limit by the 24th century is reasonable, assuming that we're not an especially populous or prolific species. (The census was apparently from the Star Trek: Star Charts, not a canonical source.)

---

Edit: As ilinamorato points out, the 9 billion figure is for the Borg Earth in the alternate timeline. First Contact shows the planet pretty much entirely technofied/assimilated:

Compare with Earth in the original timeline:

Borg drones have no need for recreation, personal space, or other quality of life considerations. Their machine-like efficiency, spartan living requirements and complete disregard for the environment enable them to completely colonize the planet surface. Therefore, if there are only 9 billion life signs on the surface, we can assume this is the approximate upper limit of humanoid life that the planet can support.

The only figure we have on the Earth population in the regular timeline is the Star Trek: Star Charts 2370 census which found 4.2 billion humans on Earth at that time. Even though this isn't canon, it's reasonable to assume that there probably isn't much more than 9 billion humans on Earth in the 24th century.