In machine learning, we also need to do operations on tensors, like adding them, multiplying them, applying ReLU and softmax on them. These operations on tensors are known as ops.

There is no universal agreement on what the word "op" is short for. It could be either "operation" or "operator". Operation just means we perform a math operation on the tensors. For the word operator, you can think of it as an operator overloaded function in Python, as shown in the following snippet.

class Tensor:
    ...
    def __add__(self, other):
        # Implement addition operation here.
        ...

# Example usage:
a = Tensor(...)
b = Tensor(...)
c = a + b  # This calls the __add__ method.