I'm 24, my husband is 25 and we have a (nearly) 18month old. EVERYONE told us we were too young, but we are having the time of our lives! Our son is the best thing that's ever happened to us.. We both have careers (my husband is a high school teacher and I am a makeup artist) and interests of our own, but our son now comes first.
We have lost friends over our decision to start a family, as most of them are still partying hard, but we have also made lots of new friends!
At the end of the day, you need to make the decision that is best for you (and your partner!) because everyone will have an opinion on what is right or wrong.
Good luck with whatever you choose!