Leveling too early and ignoring the story?

I have been playing for several weeks now, and am very slow at progressing the story. I just got past the meeting with Killerblade and fought the horse dancer... And I'm level 55. My friend is saying that I need to do the story before the game difficulty out scales what I'm am capable of handling. I have just been going all around Qinghe finding oddities and doing random encounters.

What are your opinions? I know I have to do it eventually, but am I really hurting my progression by not doing it?