Stochastic Gradient Descent (SGD) Example

This is a simple example of Stochastic Gradient Descent (SGD) using Python and the scikit-learn library.

SGD Overview

Stochastic Gradient Descent (SGD) is an optimization algorithm commonly used for training machine learning models, especially in the context of large datasets. Unlike traditional gradient descent, which computes the gradient using the entire dataset, SGD updates the model parameters using a small random subset of the data (mini-batch) at each iteration. This makes it computationally more efficient and suitable for online learning.

Key concepts of SGD:

• Gradient Descent: An iterative optimization algorithm for finding the minimum of a function (minimizing a loss function).
• Stochastic Updates: Model parameters are updated using a random subset (mini-batch) of the training data.
• Learning Rate: Controls the step size in the parameter space during optimization.
• Iterations: The number of passes through the entire dataset (epochs).

SGD is widely used in training various machine learning models, including linear models, neural networks, and support vector machines.

Python Source Code:

# Import necessary libraries
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import SGDRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# Generate synthetic data
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train SGDRegressor model
sgd_model = SGDRegressor(max_iter=100, eta0=0.01, random_state=42)
sgd_model.fit(X_train, y_train.ravel())

# Predict on the test set
y_pred = sgd_model.predict(X_test)

# Plot the model's predictions
plt.figure(figsize=(10, 6))
plt.scatter(X_test, y_test, color='black', label='True Data Points')
plt.plot(X_test, y_pred, color='red', label='SGDRegressor Prediction')
plt.title('Stochastic Gradient Descent (SGD) Regression')
plt.xlabel('X')
plt.ylabel('y')
plt.legend()
plt.show()

Explanation:

• Import Libraries: Import necessary Python libraries, including NumPy for numerical operations, Matplotlib for plotting, and scikit-learn for SGDRegressor.
• Generate Synthetic Data: Generate synthetic data for demonstration purposes.
• Split Data: Split the data into training and testing sets.
• Train SGDRegressor Model: Train the SGDRegressor model with specified hyperparameters (max_iter and eta0).
• Predict and Plot: Predict on the test set and plot the model's predictions.