Every machine learning algorithm with any ability to generalize beyond the training data that it sees has some type of inductive bias, which are the assumptions made by the model to learn the target function and to generalize beyond training data.

For example, in linear regression, the model assumes that the output or dependent variable is related to independent variable linearly (in the weights). This is an inductive bias of the model.

What is inductive bias?

Pretty much every design choice in machine learning signifies some sort of inductive bias. "Relational inductive biases, deep learning, and graph networks" (Battaglia et. al, 2018) is an amazing 🙌 read, which I will be referring to throughout this answer.

An inductive bias allows a learning algorithm to prioritize one solution (or interpretation) over another, independent of the observed data. [...] Inductive biases can express assumptions about either the data-generating process or the space of solutions.

Examples in deep learning

Concretely speaking, the very composition of layers 🍰 in deep learning provides a type of relational inductive bias: hierarchical processing. The type of layer imposes further relational inductive biases:

More generally, non-relational inductive biases used in deep learning include:

• activation non-linearities,
• weight decay,
• dropout,
• batch and layer normalization,
• data augmentation,
• training curricula,
• optimization algorithms,
• anything that imposes constraints on the learning trajectory.