Using a nVidia Tesla P40, but only 4 / 24 GB being used?

Built a rig with the intent of using it for local AI stuff, and I got a Nvidia Tesla P40, 3D printed a fan rig on it, but whenever I run SD, it is doing like 2 seconds per iteration and in the resource manager, I am only using 4 GB of VRAM, when 24 GB are available.

Any ideas?

Edit to add: Using linux, have the most up to date drivers. Here is a shot of the manager: https://i.imgur.com/VVhYXkc.png