Open Machine Learning Course

Author: Yury Kashnitsky
Translated by Sergey Oreshkov
All content is distributed under the Creative Commons CC BY-NC-SA 4.0 license.

Assignment #8 (demo)

Implementation of online regressor

Here we'll implement a regressor trained with stochastic gradient descent (SGD). Fill in the missing code. If you do evething right, you'll pass a simple embedded test.

Linear regression and Stochastic Gradient Descent

In [1]:
import numpy as np
import pandas as pd
from tqdm import tqdm
from sklearn.base import BaseEstimator
from sklearn.metrics import mean_squared_error, log_loss, roc_auc_score
from sklearn.model_selection import train_test_split
%matplotlib inline
from matplotlib import pyplot as plt
import seaborn as sns
from sklearn.preprocessing import StandardScaler

Implement class SGDRegressor. Specification:

  • class is inherited from sklearn.base.BaseEstimator
  • constructor takes parameters eta – gradient step ($10^{-3}$ by default) and n_epochs – dataset pass count (3 by default)
  • constructor also createы mse_ and weights_ lists in order to track mean squared error and weight vector during gradient descent iterations
  • Class has fit and predict methods
  • The fit method takes matrix X and vector y (numpy.array objects) as parameters, appends column of ones to X on the left side, initializes weight vector w with zeros and then makes n_epochs iterations of weight updates (you may refer to this article for details), and for every iteration logs mean squared error and weight vector w in corresponding lists we created in the constructor.
  • Additionally the fit method will create w_ variable to store weights which produce minimal mean squared error
  • The fit method returns current instance of the SGDRegressor class, i.e. self
  • The predict method takes X matrix, adds column of ones to the left side and returns prediction vector, using weight vector w_, created by the fit method.
In [2]:
class SGDRegressor(BaseEstimator):
    # you code here
    def __init__(self):
    def fit(self, X, y):
    def predict(self, X):

Let's test out the algorithm on height/weight data. We will predict heights (in inches) based on weights (in lbs).

In [3]:
data_demo = pd.read_csv('../../data/weights_heights.csv')
In [4]:
plt.scatter(data_demo['Weight'], data_demo['Height']);
plt.xlabel('Weight (lbs)')
plt.ylabel('Height (Inch)')
In [5]:
X, y = data_demo['Weight'].values, data_demo['Height'].values

Perform train/test split and scale data.

In [6]:
X_train, X_valid, y_train, y_valid = train_test_split(X, y,
In [7]:
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.reshape([-1, 1]))
X_valid_scaled = scaler.transform(X_valid.reshape([-1, 1]))

Train created SGDRegressor with (X_train_scaled, y_train) data. Leave default parameter values for now.

In [8]:
# you code here

Draw a chart with training process – dependency of mean squared error from the i-th SGD iteration number.

In [9]:
# you code here

Print the minimal value of mean squared error and the best weights vector.

In [10]:
# you code here

Draw chart of model weights ($w_0$ and $w_1$) behavior during training.

In [11]:
# you code here

Make a prediction for hold-out set (X_valid_scaled, y_valid) and check MSE value.

In [12]:
# you code here
sgd_holdout_mse = 10

Do the same thing for LinearRegression class from sklearn.linear_model. Evaluate MSE for hold-out set.

In [13]:
# you code here
linreg_holdout_mse = 9
In [14]:
    assert (sgd_holdout_mse - linreg_holdout_mse) < 1e-4
except AssertionError:
    print("Something's not good.\n Linreg's holdout MSE: {}"
          "\n SGD's holdout MSE: {}".format(linreg_holdout_mse, 
Something's not good.
 Linreg's holdout MSE: 9
 SGD's holdout MSE: 10