When creating a tensor in PyTorch, we usually need to copy it to the GPU as shown in the following code snippet.

import torch
a = torch.randn(1000, 1000)
a_gpu = a.to('cuda')

Here, a is created in the host memory. People may also call it the main memory, random access memory (RAM), or CPU memory. They all refer to the same thing: the memory space on a normal computer with a CPU.

a_gpu is a copy of a stored in device memory. People may also call it global memory, high bandwidth memory (HBM), or GPU memory. They all refer to the memory that is attached to the GPU. We will dive deeper into the GPU memory hierarchy in the next chapter.

Just saying "memory" can be confusing here since there are two of them. That is why people explicitly use contrasting pairs of terms like host memory vs. device memory, or CPU memory vs. GPU memory.

In most MLSys engineering cases, we only care about the device memory instead of the host memory. We may just load the data from external storage into the host memory. Then, we copy the data from the host memory to the device memory. After that, we only work with the data in the device memory.

The total amount of data a memory can hold is its capacity. Host memory is usually much larger and cheaper than device memory. You can easily have a computer with 128 GB of host memory for $2000, while a high-end GPU like the NVIDIA A100 has only 80 GB of device memory and costs more than $10,000. Running out of device memory is the most common practical constraint in MLSys engineering. It determines which models you can load and how large your batch sizes can be.