examples/02_model_content_based_filtering/vowpal_wabbit_deep_dive.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "<i>Copyright (c) Recommenders contributors.</i>\n",
                "\n",
                "<i>Licensed under the MIT License.</i>"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# Vowpal Wabbit Deep Dive"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "<center>\n",
                "<img src=\"https://github.com/VowpalWabbit/vowpal_wabbit/blob/master/logo_assets/vowpal-wabbits-github-logo.png?raw=true\" height=\"30%\" width=\"30%\" alt=\"Vowpal Wabbit\">\n",
                "</center>\n",
                "\n",
                "[Vowpal Wabbit](https://github.com/VowpalWabbit/vowpal_wabbit) is a fast online machine learning library that implements several algorithms relevant to the recommendation use case.\n",
                "\n",
                "The main advantage of Vowpal Wabbit (VW) is that training is done in an online fashion typically using Stochastic Gradient Descent or similar variants, which allows it to scale well to very large datasets. Additionally, it is optimized to run very quickly and can support distributed training scenarios for extremely large datasets. \n",
                "\n",
                "VW is best applied to problems where the dataset is too large to fit into memory but can be stored on disk in a single node. Though distributed training is possible with additional setup and configuration of the nodes. The kinds of problems that VW handles well mostly fall into the supervised classification domain of machine learning (Linear Regression, Logistic Regression, Multiclass Classification, Support Vector Machines, Simple Neural Nets). It also supports Matrix Factorization approaches and Latent Dirichlet Allocation, as well as a few other algorithms (see the [wiki](https://github.com/VowpalWabbit/vowpal_wabbit/wiki) for more information).\n",
                "\n",
                "A good example of a typical deployment use case is a Real Time Bidding scenario, where an auction to place an ad for a user is being decided in a matter of milliseconds. Feature information about the user and items must be extracted and passed into a model to predict likelihood of click (or other interaction) in short order. And if the user and context features are constantly changing (e.g. user browser and local time of day) it may be infeasible to score every possible input combination before hand. This is where VW provides value, as a platform to explore various algorithms offline to train a highly accurate model on a large set of historical data then deploy the model into production so it can generate rapid predictions in real time. Of course this isn't the only manner VW can be deployed, it is also possible to use it entirely online where the model is constantly updating, or use active learning approaches, or work completely offline in a pre-scoring mode. "
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "<h3>Vowpal Wabbit for Recommendations</h3>\n",
                "\n",
                "In this notebook we demonstrate how to use the VW library to generate recommendations on the [MovieLens](https://grouplens.org/datasets/movielens/) dataset.\n",
                "\n",
                "Several things are worth noting in how VW is being used in this notebook:\n",
                "\n",
                "By leveraging an Azure Data Science Virtual Machine ([DSVM](https://azure.microsoft.com/en-us/services/virtual-machines/data-science-virtual-machines/)), VW comes pre-installed and can be used directly from the command line. If you are not using a DSVM you must install vw yourself. \n",
                "\n",
                "There are also python bindings to allow VW use within a python environment and even a wrapper conforming to the SciKit-Learn Estimator API. However, the python bindings must be installed as an additional python package with Boost dependencies, so for simplicity's sake execution of VW is done via a subprocess call mimicking what would happen from the command line execution of the model.\n",
                "\n",
                "VW expects a specific [input format](https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Input-format), in this notebook to_vw() is a convenience function that converts the standard movielens dataset into the required data format. Datafiles are then written to disk and passed to VW for training.\n",
                "\n",
                "The examples shown are to demonstrate functional capabilities of VW not to indicate performance advantages of different approaches. There are several hyper-parameters (e.g. learning rate and regularization terms) that can greatly impact performance of VW models which can be adjusted using [command line options](https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Command-Line-Arguments). To properly compare approaches it is helpful to learn about and tune these parameters on the relevant dataset."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# 0. Global Setup"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "System version: 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) \n",
                        "[GCC 7.3.0]\n",
                        "Pandas version: 0.24.1\n"
                    ]
                }
            ],
            "source": [
                "import os\n",
                "import sys\n",
                "import pandas as pd\n",
                "from subprocess import run\n",
                "from tempfile import TemporaryDirectory\n",
                "from time import process_time\n",
                "\n",
                "from recommenders.datasets.movielens import load_pandas_df\n",
                "from recommenders.datasets.python_splitters import python_random_split\n",
                "from recommenders.evaluation.python_evaluation import (\n",
                "    rmse,\n",
                "    mae,\n",
                "    exp_var,\n",
                "    rsquared,\n",
                "    get_top_k_items,\n",
                "    map_at_k,\n",
                "    ndcg_at_k,\n",
                "    precision_at_k,\n",
                "    recall_at_k,\n",
                ")\n",
                "from recommenders.utils.notebook_utils import store_metadata\n",
                "\n",
                "print(\"System version: {}\".format(sys.version))\n",
                "print(\"Pandas version: {}\".format(pd.__version__))\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "metadata": {},
            "outputs": [],
            "source": [
                "def to_vw(df, output, logistic=False):\n",
                "    \"\"\"Convert Pandas DataFrame to vw input format\n",
                "    Args:\n",
                "        df (pd.DataFrame): input DataFrame\n",
                "        output (str): path to output file\n",
                "        logistic (bool): flag to convert label to logistic value\n",
                "    \"\"\"\n",
                "    with open(output, 'w') as f:\n",
                "        tmp = df.reset_index()\n",
                "\n",
                "        # we need to reset the rating type to an integer to simplify the vw formatting\n",
                "        tmp['rating'] = tmp['rating'].astype('int64')\n",
                "        \n",
                "        # convert rating to binary value\n",
                "        if logistic:\n",
                "            tmp['rating'] = tmp['rating'].apply(lambda x: 1 if x >= 3 else -1)\n",
                "        \n",
                "        # convert each row to VW input format (https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Input-format)\n",
                "        # [label] [tag]|[user namespace] [user id feature] |[item namespace] [movie id feature]\n",
                "        # label is the true rating, tag is a unique id for the example just used to link predictions to truth\n",
                "        # user and item namespaces separate the features to support interaction features through command line options\n",
                "        for _, row in tmp.iterrows():\n",
                "            f.write('{rating:d} {index:d}|user {userID:d} |item {itemID:d}\\n'.format_map(row))"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "metadata": {},
            "outputs": [],
            "source": [
                "def run_vw(train_params, test_params, test_data, prediction_path, logistic=False):\n",
                "    \"\"\"Convenience function to train, test, and show metrics of interest\n",
                "    Args:\n",
                "        train_params (str): vw training parameters\n",
                "        test_params (str): vw testing parameters\n",
                "        test_data (pd.dataFrame): test data\n",
                "        prediction_path (str): path to vw prediction output\n",
                "        logistic (bool): flag to convert label to logistic value\n",
                "    Returns:\n",
                "        (dict): metrics and timing information\n",
                "    \"\"\"\n",
                "\n",
                "    # train model\n",
                "    train_start = process_time()\n",
                "    run(train_params.split(' '), check=True)\n",
                "    train_stop = process_time()\n",
                "    \n",
                "    # test model\n",
                "    test_start = process_time()\n",
                "    run(test_params.split(' '), check=True)\n",
                "    test_stop = process_time()\n",
                "    \n",
                "    # read in predictions\n",
                "    pred_df = pd.read_csv(prediction_path, delim_whitespace=True, names=['prediction'], index_col=1).join(test_data)\n",
                "    pred_df.drop(\"rating\", axis=1, inplace=True)\n",
                "\n",
                "    test_df = test_data.copy()\n",
                "    if logistic:\n",
                "        # make the true label binary so that the metrics are captured correctly\n",
                "        test_df['rating'] = test['rating'].apply(lambda x: 1 if x >= 3 else -1)\n",
                "    else:\n",
                "        # ensure results are integers in correct range\n",
                "        pred_df['prediction'] = pred_df['prediction'].apply(lambda x: int(max(1, min(5, round(x)))))\n",
                "\n",
                "    # calculate metrics\n",
                "    result = dict()\n",
                "    result['RMSE'] = rmse(test_df, pred_df)\n",
                "    result['MAE'] = mae(test_df, pred_df)\n",
                "    result['R2'] = rsquared(test_df, pred_df)\n",
                "    result['Explained Variance'] = exp_var(test_df, pred_df)\n",
                "    result['Train Time (ms)'] = (train_stop - train_start) * 1000\n",
                "    result['Test Time (ms)'] = (test_stop - test_start) * 1000\n",
                "    \n",
                "    return result"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "metadata": {},
            "outputs": [],
            "source": [
                "# create temp directory to maintain data files\n",
                "tmpdir = TemporaryDirectory()\n",
                "\n",
                "model_path = os.path.join(tmpdir.name, 'vw.model')\n",
                "saved_model_path = os.path.join(tmpdir.name, 'vw_saved.model')\n",
                "train_path = os.path.join(tmpdir.name, 'train.dat')\n",
                "test_path = os.path.join(tmpdir.name, 'test.dat')\n",
                "train_logistic_path = os.path.join(tmpdir.name, 'train_logistic.dat')\n",
                "test_logistic_path = os.path.join(tmpdir.name, 'test_logistic.dat')\n",
                "prediction_path = os.path.join(tmpdir.name, 'prediction.dat')\n",
                "all_test_path = os.path.join(tmpdir.name, 'new_test.dat')\n",
                "all_prediction_path = os.path.join(tmpdir.name, 'new_prediction.dat')"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# 1. Load & Transform Data"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "metadata": {
                "tags": [
                    "parameters"
                ]
            },
            "outputs": [],
            "source": [
                "# Select MovieLens data size: 100k, 1m, 10m, or 20m\n",
                "MOVIELENS_DATA_SIZE = '100k'\n",
                "TOP_K = 10"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "metadata": {},
            "outputs": [
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "4.93MB [00:00, 6.18MB/s]                            \n"
                    ]
                }
            ],
            "source": [
                "# load movielens data \n",
                "df = load_pandas_df(MOVIELENS_DATA_SIZE)\n",
                "\n",
                "# split data to train and test sets, default values take 75% of each users ratings as train, and 25% as test\n",
                "train, test = python_random_split(df, 0.75)\n",
                "\n",
                "# save train and test data in vw format\n",
                "to_vw(df=train, output=train_path)\n",
                "to_vw(df=test, output=test_path)\n",
                "\n",
                "# save data for logistic regression (requires adjusting the label)\n",
                "to_vw(df=train, output=train_logistic_path, logistic=True)\n",
                "to_vw(df=test, output=test_logistic_path, logistic=True)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# 2. Regression Based Recommendations\n",
                "\n",
                "When considering different approaches for solving a problem with machine learning it is helpful to generate a baseline approach to understand how more complex solutions perform across dimensions of performance, time, and resource (memory or cpu) usage.\n",
                "\n",
                "Regression based approaches are some of the simplest and fastest baselines to consider for many ML problems."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 2.1 Linear Regression\n",
                "\n",
                "As the data provides a numerical rating between 1-5, fitting those values with a linear regression model is easy approach. This model is trained on examples of ratings as the target variable and corresponding user ids and movie ids as independent features.\n",
                "\n",
                "By passing each user-item rating in as an example the model will begin to learn weights based on average ratings for each user as well as average ratings per item.\n",
                "\n",
                "This however can generate predicted ratings which are no longer integers, so some additional adjustments should be made at prediction time to convert them back to the integer scale of 1 through 5 if necessary. Here, this is done in the evaluate function."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 7,
            "metadata": {
                "scrolled": true
            },
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>RMSE</th>\n",
                            "      <th>MAE</th>\n",
                            "      <th>R2</th>\n",
                            "      <th>Explained Variance</th>\n",
                            "      <th>Train Time (ms)</th>\n",
                            "      <th>Test Time (ms)</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>Linear Regression</th>\n",
                            "      <td>0.988433</td>\n",
                            "      <td>0.70988</td>\n",
                            "      <td>0.227276</td>\n",
                            "      <td>0.227286</td>\n",
                            "      <td>62.5</td>\n",
                            "      <td>15.625</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                       RMSE      MAE        R2  Explained Variance  \\\n",
                            "Linear Regression  0.988433  0.70988  0.227276            0.227286   \n",
                            "\n",
                            "                   Train Time (ms)  Test Time (ms)  \n",
                            "Linear Regression             62.5          15.625  "
                        ]
                    },
                    "execution_count": 7,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "\"\"\"\n",
                "Quick description of command line parameters used\n",
                "  Other optional parameters can be found here: https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Command-Line-Arguments\n",
                "  VW uses linear regression by default, so no extra command line options\n",
                "  -f <model_path>: indicates where the final model file will reside after training\n",
                "  -d <data_path>: indicates which data file to use for training or testing\n",
                "  --quiet: this runs vw in quiet mode silencing stdout (for debugging it's helpful to not use quiet mode)\n",
                "  -i <model_path>: indicates where to load the previously model file created during training\n",
                "  -t: this executes inference only (no learned updates to the model)\n",
                "  -p <prediction_path>: indicates where to store prediction output\n",
                "\"\"\"\n",
                "train_params = 'vw -f {model} -d {data} --quiet'.format(model=model_path, data=train_path)\n",
                "# save these results for later use during top-k analysis\n",
                "test_params = 'vw -i {model} -d {data} -t -p {pred} --quiet'.format(model=model_path, data=test_path, pred=prediction_path)\n",
                "\n",
                "result = run_vw(train_params=train_params, \n",
                "                test_params=test_params, \n",
                "                test_data=test, \n",
                "                prediction_path=prediction_path)\n",
                "\n",
                "comparison = pd.DataFrame(result, index=['Linear Regression'])\n",
                "comparison"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 2.2 Linear Regression with Interaction Features\n",
                "\n",
                "Previously we treated the user features and item features independently, but taking into account interactions between features can provide a mechanism to learn more fine grained preferences of the users.\n",
                "\n",
                "To generate interaction features use the quadratic command line argument and specify the namespaces that should be combined: '-q ui' combines the user and item namespaces based on the first letter of each.\n",
                "\n",
                "Currently the userIDs and itemIDs used are integers which means the feature ID is used directly, for instance when user ID 123 rates movie 456, the training example puts a 1 in the values for features 123 and 456. However when interaction is specified (or if a feature is a string) the resulting interaction feature is hashed into the available feature space. Feature hashing is a way to take a very sparse high dimensional feature space and reduce it into a lower dimensional space. This allows for reduced memory while retaining fast computation of feature and model weights.\n",
                "\n",
                "The caveat with feature hashing, is that it can lead to hash collisions, where separate features are mapped to the same location. In this case it can be beneficial to increase the size of the space to support interactions between features of high cardinality. The available feature space is dictated by the --bit_precision (-b) <N> argument. Where the total available space for all features in the model is 2<sup>N</sup>. \n",
                "\n",
                "See [Feature Hashing and Extraction](https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Feature-Hashing-and-Extraction) for more details."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 8,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>RMSE</th>\n",
                            "      <th>MAE</th>\n",
                            "      <th>R2</th>\n",
                            "      <th>Explained Variance</th>\n",
                            "      <th>Train Time (ms)</th>\n",
                            "      <th>Test Time (ms)</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>Linear Regression</th>\n",
                            "      <td>0.988433</td>\n",
                            "      <td>0.70988</td>\n",
                            "      <td>0.227276</td>\n",
                            "      <td>0.227286</td>\n",
                            "      <td>62.500</td>\n",
                            "      <td>15.625</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Linear Regression w/ Interaction</th>\n",
                            "      <td>0.985921</td>\n",
                            "      <td>0.71292</td>\n",
                            "      <td>0.231199</td>\n",
                            "      <td>0.231338</td>\n",
                            "      <td>15.625</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                      RMSE      MAE        R2  \\\n",
                            "Linear Regression                 0.988433  0.70988  0.227276   \n",
                            "Linear Regression w/ Interaction  0.985921  0.71292  0.231199   \n",
                            "\n",
                            "                                  Explained Variance  Train Time (ms)  \\\n",
                            "Linear Regression                           0.227286           62.500   \n",
                            "Linear Regression w/ Interaction            0.231338           15.625   \n",
                            "\n",
                            "                                  Test Time (ms)  \n",
                            "Linear Regression                         15.625  \n",
                            "Linear Regression w/ Interaction          31.250  "
                        ]
                    },
                    "execution_count": 8,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "\"\"\"\n",
                "Quick description of command line parameters used\n",
                "  -b <N>: sets the memory size to 2<sup>N</sup> entries\n",
                "  -q <ab>: create quadratic feature interactions between features in namespaces starting with 'a' and 'b' \n",
                "\"\"\"\n",
                "train_params = 'vw -b 26 -q ui -f {model} -d {data} --quiet'.format(model=saved_model_path, data=train_path)\n",
                "test_params = 'vw -i {model} -d {data} -t -p {pred} --quiet'.format(model=saved_model_path, data=test_path, pred=prediction_path)\n",
                "\n",
                "result = run_vw(train_params=train_params,\n",
                "                test_params=test_params,\n",
                "                test_data=test,\n",
                "                prediction_path=prediction_path)\n",
                "saved_result = result\n",
                "\n",
                "comparison = comparison.append(pd.DataFrame(result, index=['Linear Regression w/ Interaction']))\n",
                "comparison"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 2.3 Multinomial Logistic Regression\n",
                "\n",
                "An alternative to linear regression is to leverage multinomial logistic regression, or multiclass classification, which treats each rating value as a distinct class. \n",
                "\n",
                "This avoids any non integer results, but also reduces the training data for each class which could lead to poorer performance if the counts of different rating levels are skewed.\n",
                "\n",
                "Basic multiclass logistic regression can be accomplished using the One Against All approach specified by the '--oaa N' option, where N is the number of classes and proving the logistic option for the loss function to be used."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 9,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>RMSE</th>\n",
                            "      <th>MAE</th>\n",
                            "      <th>R2</th>\n",
                            "      <th>Explained Variance</th>\n",
                            "      <th>Train Time (ms)</th>\n",
                            "      <th>Test Time (ms)</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>Linear Regression</th>\n",
                            "      <td>0.988433</td>\n",
                            "      <td>0.70988</td>\n",
                            "      <td>0.227276</td>\n",
                            "      <td>0.227286</td>\n",
                            "      <td>62.500</td>\n",
                            "      <td>15.625</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Linear Regression w/ Interaction</th>\n",
                            "      <td>0.985921</td>\n",
                            "      <td>0.71292</td>\n",
                            "      <td>0.231199</td>\n",
                            "      <td>0.231338</td>\n",
                            "      <td>15.625</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Multinomial Regression</th>\n",
                            "      <td>1.112780</td>\n",
                            "      <td>0.75564</td>\n",
                            "      <td>0.020626</td>\n",
                            "      <td>0.050111</td>\n",
                            "      <td>62.500</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                      RMSE      MAE        R2  \\\n",
                            "Linear Regression                 0.988433  0.70988  0.227276   \n",
                            "Linear Regression w/ Interaction  0.985921  0.71292  0.231199   \n",
                            "Multinomial Regression            1.112780  0.75564  0.020626   \n",
                            "\n",
                            "                                  Explained Variance  Train Time (ms)  \\\n",
                            "Linear Regression                           0.227286           62.500   \n",
                            "Linear Regression w/ Interaction            0.231338           15.625   \n",
                            "Multinomial Regression                      0.050111           62.500   \n",
                            "\n",
                            "                                  Test Time (ms)  \n",
                            "Linear Regression                         15.625  \n",
                            "Linear Regression w/ Interaction          31.250  \n",
                            "Multinomial Regression                    31.250  "
                        ]
                    },
                    "execution_count": 9,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "\"\"\"\n",
                "Quick description of command line parameters used\n",
                "  --loss_function logistic: sets the model loss function for logistic regression\n",
                "  --oaa <N>: trains N separate models using One-Against-All approach (all models are captured in the single model file)\n",
                "             This expects the labels to be contiguous integers starting at 1\n",
                "  --link logistic: converts the predicted output from logit to probability\n",
                "The predicted output is the model (label) with the largest likelihood\n",
                "\"\"\"\n",
                "train_params = 'vw --loss_function logistic --oaa 5 -f {model} -d {data} --quiet'.format(model=model_path, data=train_path)\n",
                "test_params = 'vw --link logistic -i {model} -d {data} -t -p {pred} --quiet'.format(model=model_path, data=test_path, pred=prediction_path)\n",
                "\n",
                "result = run_vw(train_params=train_params,\n",
                "                test_params=test_params,\n",
                "                test_data=test,\n",
                "                prediction_path=prediction_path)\n",
                "\n",
                "comparison = comparison.append(pd.DataFrame(result, index=['Multinomial Regression']))\n",
                "comparison"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 2.4 Logistic Regression\n",
                "\n",
                "Additionally, one might simply be interested in whether the user likes or dislikes an item and we can adjust the input data to represent a binary outcome, where ratings in (1,3] are dislikes (negative results) and (3,5] are likes (positive results).\n",
                "\n",
                "This framing allows for a simple logistic regression model to be applied. To perform logistic regression the loss_function parameter is changed to 'logistic' and the target label is switched to [0, 1]. Also, be sure to set '--link logistic' during prediction to convert the logit output back to a probability value."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 10,
            "metadata": {
                "scrolled": true
            },
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>RMSE</th>\n",
                            "      <th>MAE</th>\n",
                            "      <th>R2</th>\n",
                            "      <th>Explained Variance</th>\n",
                            "      <th>Train Time (ms)</th>\n",
                            "      <th>Test Time (ms)</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>Linear Regression</th>\n",
                            "      <td>0.988433</td>\n",
                            "      <td>0.709880</td>\n",
                            "      <td>0.227276</td>\n",
                            "      <td>0.227286</td>\n",
                            "      <td>62.500</td>\n",
                            "      <td>15.625</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Linear Regression w/ Interaction</th>\n",
                            "      <td>0.985921</td>\n",
                            "      <td>0.712920</td>\n",
                            "      <td>0.231199</td>\n",
                            "      <td>0.231338</td>\n",
                            "      <td>15.625</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Multinomial Regression</th>\n",
                            "      <td>1.112780</td>\n",
                            "      <td>0.755640</td>\n",
                            "      <td>0.020626</td>\n",
                            "      <td>0.050111</td>\n",
                            "      <td>62.500</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Logistic Regression</th>\n",
                            "      <td>0.717475</td>\n",
                            "      <td>0.409551</td>\n",
                            "      <td>0.096362</td>\n",
                            "      <td>0.142500</td>\n",
                            "      <td>46.875</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                      RMSE       MAE        R2  \\\n",
                            "Linear Regression                 0.988433  0.709880  0.227276   \n",
                            "Linear Regression w/ Interaction  0.985921  0.712920  0.231199   \n",
                            "Multinomial Regression            1.112780  0.755640  0.020626   \n",
                            "Logistic Regression               0.717475  0.409551  0.096362   \n",
                            "\n",
                            "                                  Explained Variance  Train Time (ms)  \\\n",
                            "Linear Regression                           0.227286           62.500   \n",
                            "Linear Regression w/ Interaction            0.231338           15.625   \n",
                            "Multinomial Regression                      0.050111           62.500   \n",
                            "Logistic Regression                         0.142500           46.875   \n",
                            "\n",
                            "                                  Test Time (ms)  \n",
                            "Linear Regression                         15.625  \n",
                            "Linear Regression w/ Interaction          31.250  \n",
                            "Multinomial Regression                    31.250  \n",
                            "Logistic Regression                       31.250  "
                        ]
                    },
                    "execution_count": 10,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "train_params = 'vw --loss_function logistic -f {model} -d {data} --quiet'.format(model=model_path, data=train_logistic_path)\n",
                "test_params = 'vw --link logistic -i {model} -d {data} -t -p {pred} --quiet'.format(model=model_path, data=test_logistic_path, pred=prediction_path)\n",
                "\n",
                "result = run_vw(train_params=train_params,\n",
                "                test_params=test_params,\n",
                "                test_data=test,\n",
                "                prediction_path=prediction_path,\n",
                "                logistic=True)\n",
                "\n",
                "comparison = comparison.append(pd.DataFrame(result, index=['Logistic Regression']))\n",
                "comparison"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# 3. Matrix Factorization Based Recommendations\n",
                "\n",
                "All of the above approaches train a regression model, but VW also supports matrix factorization with two different approaches.\n",
                "\n",
                "As opposed to learning direct weights for specific users, items and interactions when training a regression model, matrix factorization attempts to learn latent factors that determine how a user rates an item. An example of how this might work is if you could represent user preference and item categorization by genre. Given a smaller set of genres we can associate how much each item belongs to each genre class, and we can set weights for a user's preference for each genre. Both sets of weights could be represented as a vectors where the inner product would be the user-item rating. Matrix factorization approaches learn low rank matrices for latent features of users and items such that those matrices can be combined to approximate the original user item matrix.\n",
                "\n",
                "## 3.1. Singular Value Decomposition Based Matrix Factorization\n",
                "\n",
                "The first approach performs matrix factorization based on Singular Value Decomposition (SVD) to learn a low rank approximation for the user-item rating matix. It is is called using the '--rank' command line argument.\n",
                "\n",
                "See the [Matrix Factorization Example](https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Matrix-factorization-example) for more detail."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 11,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>RMSE</th>\n",
                            "      <th>MAE</th>\n",
                            "      <th>R2</th>\n",
                            "      <th>Explained Variance</th>\n",
                            "      <th>Train Time (ms)</th>\n",
                            "      <th>Test Time (ms)</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>Linear Regression</th>\n",
                            "      <td>0.988433</td>\n",
                            "      <td>0.709880</td>\n",
                            "      <td>0.227276</td>\n",
                            "      <td>0.227286</td>\n",
                            "      <td>62.500</td>\n",
                            "      <td>15.625</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Linear Regression w/ Interaction</th>\n",
                            "      <td>0.985921</td>\n",
                            "      <td>0.712920</td>\n",
                            "      <td>0.231199</td>\n",
                            "      <td>0.231338</td>\n",
                            "      <td>15.625</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Multinomial Regression</th>\n",
                            "      <td>1.112780</td>\n",
                            "      <td>0.755640</td>\n",
                            "      <td>0.020626</td>\n",
                            "      <td>0.050111</td>\n",
                            "      <td>62.500</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Logistic Regression</th>\n",
                            "      <td>0.717475</td>\n",
                            "      <td>0.409551</td>\n",
                            "      <td>0.096362</td>\n",
                            "      <td>0.142500</td>\n",
                            "      <td>46.875</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Matrix Factorization (Rank)</th>\n",
                            "      <td>1.010723</td>\n",
                            "      <td>0.745800</td>\n",
                            "      <td>0.192033</td>\n",
                            "      <td>0.221080</td>\n",
                            "      <td>31.250</td>\n",
                            "      <td>15.625</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                      RMSE       MAE        R2  \\\n",
                            "Linear Regression                 0.988433  0.709880  0.227276   \n",
                            "Linear Regression w/ Interaction  0.985921  0.712920  0.231199   \n",
                            "Multinomial Regression            1.112780  0.755640  0.020626   \n",
                            "Logistic Regression               0.717475  0.409551  0.096362   \n",
                            "Matrix Factorization (Rank)       1.010723  0.745800  0.192033   \n",
                            "\n",
                            "                                  Explained Variance  Train Time (ms)  \\\n",
                            "Linear Regression                           0.227286           62.500   \n",
                            "Linear Regression w/ Interaction            0.231338           15.625   \n",
                            "Multinomial Regression                      0.050111           62.500   \n",
                            "Logistic Regression                         0.142500           46.875   \n",
                            "Matrix Factorization (Rank)                 0.221080           31.250   \n",
                            "\n",
                            "                                  Test Time (ms)  \n",
                            "Linear Regression                         15.625  \n",
                            "Linear Regression w/ Interaction          31.250  \n",
                            "Multinomial Regression                    31.250  \n",
                            "Logistic Regression                       31.250  \n",
                            "Matrix Factorization (Rank)               15.625  "
                        ]
                    },
                    "execution_count": 11,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "\"\"\"\n",
                "Quick description of command line parameters used\n",
                "  --rank <N>: sets the number of latent factors in the reduced matrix\n",
                "\"\"\"\n",
                "train_params = 'vw --rank 5 -q ui -f {model} -d {data} --quiet'.format(model=model_path, data=train_path)\n",
                "test_params = 'vw -i {model} -d {data} -t -p {pred} --quiet'.format(model=model_path, data=test_path, pred=prediction_path)\n",
                "\n",
                "result = run_vw(train_params=train_params,\n",
                "                test_params=test_params,\n",
                "                test_data=test,\n",
                "                prediction_path=prediction_path)\n",
                "\n",
                "comparison = comparison.append(pd.DataFrame(result, index=['Matrix Factorization (Rank)']))\n",
                "comparison"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 3.2. Factorization Machine Based Matrix Factorization\n",
                "\n",
                "An alternative approach based on [Rendel's factorization machines](https://cseweb.ucsd.edu/classes/fa17/cse291-b/reading/Rendle2010FM.pdf) is called using '--lrq' (low rank quadratic). More LRQ details in this [demo](https://github.com/VowpalWabbit/vowpal_wabbit/tree/master/demo/movielens).\n",
                "\n",
                "This learns two lower rank matrices which are multiplied to generate an approximation of the user-item rating matrix. Compressing the matrix in this way leads to learning generalizable factors which avoids some of the limitations of using regression models with extremely sparse interaction features. This can lead to better convergence and smaller on-disk models.\n",
                "\n",
                "An additional term to improve performance is --lrqdropout which will dropout columns during training. This however tends to increase the optimal rank size. Other parameters such as L2 regularization can help avoid overfitting."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 12,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>RMSE</th>\n",
                            "      <th>MAE</th>\n",
                            "      <th>R2</th>\n",
                            "      <th>Explained Variance</th>\n",
                            "      <th>Train Time (ms)</th>\n",
                            "      <th>Test Time (ms)</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>Linear Regression</th>\n",
                            "      <td>0.988433</td>\n",
                            "      <td>0.709880</td>\n",
                            "      <td>0.227276</td>\n",
                            "      <td>0.227286</td>\n",
                            "      <td>62.500</td>\n",
                            "      <td>15.625</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Linear Regression w/ Interaction</th>\n",
                            "      <td>0.985921</td>\n",
                            "      <td>0.712920</td>\n",
                            "      <td>0.231199</td>\n",
                            "      <td>0.231338</td>\n",
                            "      <td>15.625</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Multinomial Regression</th>\n",
                            "      <td>1.112780</td>\n",
                            "      <td>0.755640</td>\n",
                            "      <td>0.020626</td>\n",
                            "      <td>0.050111</td>\n",
                            "      <td>62.500</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Logistic Regression</th>\n",
                            "      <td>0.717475</td>\n",
                            "      <td>0.409551</td>\n",
                            "      <td>0.096362</td>\n",
                            "      <td>0.142500</td>\n",
                            "      <td>46.875</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Matrix Factorization (Rank)</th>\n",
                            "      <td>1.010723</td>\n",
                            "      <td>0.745800</td>\n",
                            "      <td>0.192033</td>\n",
                            "      <td>0.221080</td>\n",
                            "      <td>31.250</td>\n",
                            "      <td>15.625</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>Matrix Factorization (LRQ)</th>\n",
                            "      <td>1.013627</td>\n",
                            "      <td>0.726640</td>\n",
                            "      <td>0.187383</td>\n",
                            "      <td>0.187388</td>\n",
                            "      <td>31.250</td>\n",
                            "      <td>31.250</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "                                      RMSE       MAE        R2  \\\n",
                            "Linear Regression                 0.988433  0.709880  0.227276   \n",
                            "Linear Regression w/ Interaction  0.985921  0.712920  0.231199   \n",
                            "Multinomial Regression            1.112780  0.755640  0.020626   \n",
                            "Logistic Regression               0.717475  0.409551  0.096362   \n",
                            "Matrix Factorization (Rank)       1.010723  0.745800  0.192033   \n",
                            "Matrix Factorization (LRQ)        1.013627  0.726640  0.187383   \n",
                            "\n",
                            "                                  Explained Variance  Train Time (ms)  \\\n",
                            "Linear Regression                           0.227286           62.500   \n",
                            "Linear Regression w/ Interaction            0.231338           15.625   \n",
                            "Multinomial Regression                      0.050111           62.500   \n",
                            "Logistic Regression                         0.142500           46.875   \n",
                            "Matrix Factorization (Rank)                 0.221080           31.250   \n",
                            "Matrix Factorization (LRQ)                  0.187388           31.250   \n",
                            "\n",
                            "                                  Test Time (ms)  \n",
                            "Linear Regression                         15.625  \n",
                            "Linear Regression w/ Interaction          31.250  \n",
                            "Multinomial Regression                    31.250  \n",
                            "Logistic Regression                       31.250  \n",
                            "Matrix Factorization (Rank)               15.625  \n",
                            "Matrix Factorization (LRQ)                31.250  "
                        ]
                    },
                    "execution_count": 12,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "\"\"\"\n",
                "Quick description of command line parameters used\n",
                "  --lrq <abN>: learns approximations of rank N for the quadratic interaction between namespaces starting with 'a' and 'b'\n",
                "  --lrqdroupout: performs dropout during training to improve generalization\n",
                "\"\"\"\n",
                "train_params = 'vw --lrq ui7 -f {model} -d {data} --quiet'.format(model=model_path, data=train_path)\n",
                "test_params = 'vw -i {model} -d {data} -t -p {pred} --quiet'.format(model=model_path, data=test_path, pred=prediction_path)\n",
                "\n",
                "result = run_vw(train_params=train_params,\n",
                "                test_params=test_params,\n",
                "                test_data=test,\n",
                "                prediction_path=prediction_path)\n",
                "\n",
                "comparison = comparison.append(pd.DataFrame(result, index=['Matrix Factorization (LRQ)']))\n",
                "comparison"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# 4. Conclusion\n",
                "\n",
                "The table above shows a few of the approaches in the VW library that can be used for recommendation prediction. The relative performance can change when applied to different datasets and properly tuned, but it is useful to note the rapid speed at which all approaches are able to train (75,000 examples) and test (25,000 examples)."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# 5. Scoring"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "After training a model with any of the above approaches, the model can be used to score potential user-pairs in offline batch mode, or in a real-time scoring mode. The example below shows how to leverage the utilities in the recommenders directory to  generate Top-K recommendations from offline scored output."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 13,
            "metadata": {},
            "outputs": [],
            "source": [
                "# First construct a test set of all items (except those seen during training) for each user\n",
                "users = df[['userID']].drop_duplicates()\n",
                "users['key'] = 1\n",
                "\n",
                "items = df[['itemID']].drop_duplicates()\n",
                "items['key'] = 1\n",
                "\n",
                "all_pairs = pd.merge(users, items, on='key').drop(columns=['key'])\n",
                "\n",
                "# now combine with training data and keep only entries that were note in training\n",
                "merged = pd.merge(train[['userID', 'itemID', 'rating']], all_pairs, on=[\"userID\", \"itemID\"], how=\"outer\")\n",
                "all_user_items = merged[merged['rating'].isnull()].fillna(0).astype('int64')\n",
                "\n",
                "# save in vw format (this can take a while)\n",
                "to_vw(df=all_user_items, output=all_test_path)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 14,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>prediction</th>\n",
                            "      <th>userID</th>\n",
                            "      <th>itemID</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>4.565871</td>\n",
                            "      <td>1</td>\n",
                            "      <td>318</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>4.533308</td>\n",
                            "      <td>1</td>\n",
                            "      <td>64</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>4.530738</td>\n",
                            "      <td>1</td>\n",
                            "      <td>408</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>3</th>\n",
                            "      <td>4.525889</td>\n",
                            "      <td>1</td>\n",
                            "      <td>603</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>4</th>\n",
                            "      <td>4.501398</td>\n",
                            "      <td>1</td>\n",
                            "      <td>483</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "   prediction  userID  itemID\n",
                            "0    4.565871       1     318\n",
                            "1    4.533308       1      64\n",
                            "2    4.530738       1     408\n",
                            "3    4.525889       1     603\n",
                            "4    4.501398       1     483"
                        ]
                    },
                    "execution_count": 14,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# run the saved model (linear regression with interactions) on the new dataset\n",
                "test_start = process_time()\n",
                "test_params = 'vw -i {model} -d {data} -t -p {pred} --quiet'.format(model=saved_model_path, data=all_test_path, pred=prediction_path)\n",
                "run(test_params.split(' '), check=True)\n",
                "test_stop = process_time()\n",
                "test_time = test_stop - test_start\n",
                "\n",
                "# load predictions and get top-k from previous saved results\n",
                "pred_data = pd.read_csv(prediction_path, delim_whitespace=True, names=['prediction'], index_col=1).join(all_user_items)\n",
                "top_k = get_top_k_items(pred_data, col_rating='prediction', k=TOP_K)[['prediction', 'userID', 'itemID']]\n",
                "top_k.head()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 15,
            "metadata": {},
            "outputs": [],
            "source": [
                "# get ranking metrics\n",
                "args = [test, top_k]\n",
                "kwargs = dict(col_user='userID', col_item='itemID', col_rating='rating', col_prediction='prediction',\n",
                "              relevancy_method='top_k', k=TOP_K)\n",
                "\n",
                "rank_metrics = {'MAP': map_at_k(*args, **kwargs), \n",
                "                'NDCG': ndcg_at_k(*args, **kwargs),\n",
                "                'Precision': precision_at_k(*args, **kwargs),\n",
                "                'Recall': recall_at_k(*args, **kwargs)}"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 16,
            "metadata": {
                "scrolled": true
            },
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "RMSE: 0.9859208893212478\n",
                        "MAE: 0.71292\n",
                        "R2: 0.23119931215379363\n",
                        "Explained Variance: 0.2313379575958101\n",
                        "Train Time (ms): 15.625\n",
                        "Test Time (ms): 31.25\n",
                        "MAP: 0.012535184652143394\n",
                        "NDCG: 0.0965940631559385\n",
                        "Precision: 0.0977707006369427\n",
                        "Recall: 0.03761253544606115\n"
                    ]
                }
            ],
            "source": [
                "# final results\n",
                "all_results = ['{k}: {v}'.format(k=k, v=v) for k, v in saved_result.items()]\n",
                "all_results += ['{k}: {v}'.format(k=k, v=v) for k, v in rank_metrics.items()]\n",
                "print('\\n'.join(all_results))"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# 6. Cleanup"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 17,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "application/papermill.record+json": {
                            "rmse": 0.9859208893212478
                        }
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/papermill.record+json": {
                            "mae": 0.71292
                        }
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/papermill.record+json": {
                            "rsquared": 0.23119931215379363
                        }
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/papermill.record+json": {
                            "exp_var": 0.2313379575958101
                        }
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/papermill.record+json": {
                            "train_time": 15.625
                        }
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/papermill.record+json": {
                            "test_time": 0.046875
                        }
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/papermill.record+json": {
                            "map": 0.012535184652143394
                        }
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/papermill.record+json": {
                            "ndcg": 0.0965940631559385
                        }
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/papermill.record+json": {
                            "precision": 0.0977707006369427
                        }
                    },
                    "metadata": {},
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/papermill.record+json": {
                            "recall": 0.03761253544606115
                        }
                    },
                    "metadata": {},
                    "output_type": "display_data"
                }
            ],
            "source": [
                "# Record results for tests - ignore this cell\n",
                "store_metadata('rmse', saved_result['RMSE'])\n",
                "store_metadata('mae', saved_result['MAE'])\n",
                "store_metadata('rsquared', saved_result['R2'])\n",
                "store_metadata('exp_var', saved_result['Explained Variance'])\n",
                "store_metadata(\"train_time\", saved_result['Train Time (ms)'])\n",
                "store_metadata(\"test_time\", test_time)\n",
                "store_metadata('map', rank_metrics['MAP'])\n",
                "store_metadata('ndcg', rank_metrics['NDCG'])\n",
                "store_metadata('precision', rank_metrics['Precision'])\n",
                "store_metadata('recall', rank_metrics['Recall'])"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 18,
            "metadata": {},
            "outputs": [],
            "source": [
                "tmpdir.cleanup()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## References\n",
                "\n",
                "1. John Langford, et. al. Vowpal Wabbit Wiki. URL: https://github.com/VowpalWabbit/vowpal_wabbit/wiki\n",
                "2. Steffen Rendel. Factorization Machines. 2010 IEEE International Conference on Data Mining.\n",
                "3. Jake Hoffman. Matrix Factorization Example. URL: https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Matrix-factorization-example\n",
                "4. Paul Minero. Low Rank Quadratic Example. URL: https://github.com/VowpalWabbit/vowpal_wabbit/tree/master/demo/movielens"
            ]
        }
    ],
    "metadata": {
        "celltoolbar": "Tags",
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.6.8"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 2
}