Is it offensive to refer to my upbringing and circumstances of birth as "middle class"?

TL;DR - My parents worked lower-middle-class jobs growing up but my dad inherited \~$1.5-2m USD.

This is just a cultural rather than financial question, but growing up in the US \[26M\] I believed my family was poor. My parents had modest jobs (pre-K teacher, part-time plant vendor), we qualified for free lunch, and from ages 10 to 14 money was genuinely tight as we shared one car and one computer, lived with a broken washing machine for a year, and my parents fought over finances constantly. On the flip side, we owned a home in the same neighborhood with professors, doctors, engineers, and accountants, who sent their children to the same schools as me.

This inevitably lead to some questions from multiple teachers across multiple grades as I was the only kid in the class whose family income qualified me for free lunch even though my teachers knew where we lived. Multiple colleagues at my parents' church would try to subtly ask how my parents owned a house there because my parents accidentally told me they bought the house in cash and I casually mentioned it as a kid because I thought that was a normal thing everyone could do. Regardless, at 13 I gave up on my dream college because I was sure we couldn't afford it out-of-state. Around then I also discovered my parents were in over $150,000 of credit card debt.

My dad sat me down after I confronted him about never being able to afford my dream school and how I didn't see the point in trying so hard in school for it. He was apparently born to an affluent family but had a huge falling out with his siblings, and he told me about my grandfather, who died a year before I was born. He left a trust for 22 of his children to be paid out at his death and again 15 years later, with my dad being one of them. At his death in 1999, my dad had inherited $600,000 and used it to buy our house in cash and raise me. He expected another payout of over $1,000,000 before my 15th birthday, and promised that if I proved myself as a student, I'd never carry student debt. I remember calling him a delusional liar and resenting him for what I assumed was a story to calm me down. I ignored him and moved on, but a week later he put me in touch with the aunts and uncles I'd never met, who confirmed it. The money actually hit his account a little over a year later.

As an adult I describe my upbringing as modest or middle class at best. But my wife and others say that framing is extremely misleading, even offensive, to people who took me at my word about growing up without much or coming from a "modest family." Is it genuinely offensive to call my family and upbringing middle class? I justify it because it's treating an upper-middle-class inheritance as canceling out a working-class income, thus creating the middle class experience.