To install CUDA for GPU VM, use [run] file as follows.

apt update
wget developer.download.nvidia.com/compute/cuda/12.1.1/local_installers/cuda_12.1.1_530.30.02_linux.run
sudo sh cuda_12.1.1_530.30.02_linux.run

* Notice that, uncheck the [driver] install option to avoid overwriting GPU driver.

* CUDA versions up to 12.2 are supported.(Not supported after 12.3)