Differences between server game clients?

Ignoring the particular expansion or client version a server uses, is the base download more or less the same? I've got limited data and in the event I change up servers, I'd rather just download a handful of files rather than the entire client all over again.