What is the difference between L1 and L2 regularization?

Explanation:

L1 and L2 regularization are techniques used to prevent overfitting in machine learning models by adding a penalty term to the loss function. This penalty discourages complex models by shrinking the model coefficients:

• L1 Regularization (Lasso): Adds the absolute value of coefficients as a penalty term. It can lead to sparse models where some coefficients become exactly zero, effectively performing feature selection.

• L2 Regularization (Ridge): Adds the squared value of coefficients as a penalty term. It generally results in smaller, non-zero coefficients, but does not inherently produce sparse models.

Key Talking Points:

• L1 Regularization:

  • Adds absolute values of coefficients to the loss.
  • Can lead to sparse models (feature selection).
  • Useful when you suspect many features are irrelevant.

• L2 Regularization:

  • Adds squared values of coefficients to the loss.
  • Tends to shrink coefficients uniformly.
  • Useful for multicollinearity problems.

NOTES:

Reference Table:

• L1 Regularization: You're limited by weight, so you only bring the essentials, leaving some items behind (zero coefficients).
• L2 Regularization: You're limited by volume, so you compress everything uniformly to fit everything in, but nothing is left out completely (no zero coefficients).

Pseudocode:

Here is a Python snippet using scikit-learn to demonstrate L1 and L2 regularization with linear regression:

from sklearn.linear_model import Lasso, Ridge
from sklearn.datasets import make_regression

# Create synthetic data
X, y = make_regression(n_samples=100, n_features=20, noise=0.1)

# L1 Regularization (Lasso)
lasso = Lasso(alpha=0.1)
lasso.fit(X, y)
print("Lasso coefficients:", lasso.coef_)

# L2 Regularization (Ridge)
ridge = Ridge(alpha=0.1)
ridge.fit(X, y)
print("Ridge coefficients:", ridge.coef_)

Follow-Up Questions and Answers:

1. Why might you choose L1 over L2 regularization, or vice versa?

   • Answer: Choose L1 if you want feature selection and suspect many features are irrelevant. Choose L2 if you have multicollinearity issues or want to maintain all features while reducing overfitting.

2. How does regularization affect bias and variance?

   • Answer: Regularization increases bias but decreases variance, helping to prevent overfitting at the cost of a potentially higher error on the training set.

3. Can you combine L1 and L2 regularization?

   • Answer: Yes, this is known as Elastic Net, which combines both L1 and L2 penalties. It's useful when you want the benefits of both regularization types, especially when dealing with highly correlated features.

4. How do you choose the regularization parameter ( \lambda )?

   • Answer: The regularization parameter ( \lambda ) can be chosen using techniques like cross-validation, where you test different values to see which one minimizes validation error.