Elite 2 controller

I have an elite 2 controller that I use for both my XSX and PC. I'm using it on PC via USB, however, after I turn off the controller and try to use it to turn on my xbox it won't. I have to manually turn on my xbox then reconnect the controller to xbox like it's new again. What am I doing wrong?