Can I Decouple Nvidia Driver and Cuda Installation?

Like others, I've been very confused about instructions for how to install a specific CUDA version for the purposes of deep learning.

Today's main deep learning libraries (Tensorflow and PyTorch) don't support the latest CUDA version which is 11.2. But, when I install the recommended NVIDIA driver after a fresh Ubuntu installation, I end up getting CUDA 11.2 by default as can be seen when I run nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.39       Driver Version: 460.39       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090    Off  | 00000000:09:00.0  On |                  N/A |
| 30%   37C    P8    40W / 350W |    443MiB / 24259MiB |     12%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       954      G   /usr/lib/xorg/Xorg                 59MiB |
|    0   N/A  N/A      1470      G   /usr/lib/xorg/Xorg                161MiB |
|    0   N/A  N/A      1603      G   /usr/bin/gnome-shell              125MiB |
|    0   N/A  N/A      2031      G   ...AAAAAAAAA= --shared-files       59MiB |
+-----------------------------------------------------------------------------+

I tried following this Medium guide to downgrade CUDA to 11.0, using these instructions from NVIDIA for installing the version I want. But a couple of things are confusing me:

• Why is the Medium guide telling me to delete nvidia? I want to leave the drivers the way they are and just change CUDA.
• The rm -rf also looks scary. Feels wrong. Is it?
• I end up getting an error on trying to install CUDA 11.0. Something about lots of dependencies missing. I don't have it anymore because I've jumped ship and wiped everything for a fresh start.