Playing Alliance or Horde-dominated realm?

Does it make any sense to play on side which holds the minority? Won't it be too easy to completely dominate the other side by sheer numbers when I choose a server in which Allies dominate? Or is there something that I got wrong determining what realms actually are? Some people say that it will be very inconvenient to play like that, but I don't know why. Can somebody elaborate?