This tutorial shows how to run the code explained in the solution paper Recommendation Engine on Google Cloud Platform. In order to run this example fully you will need to use various components.
Disclaimer: This is not an official Google product.
This tutorial assumes that you have a Cloud Platform project. To set up a project:
The main steps involved in this example are:
The recommended approach to run Spark on Google Cloud Platform is to use Google Cloud Dataproc. Dataproc is a managed service that facilitates common tasks such as cluster deployment, jobs submissions, cluster scale and nodes monitoring. Interaction with Dataproc can be done over a UI, CLI or API.
Set up a cluster with the default parameters as explained in the Cloud Dataproc documentation on how to create a cluster. Cloud Dataproc does not require you to setup the JDBC connector.
Follow these instructions to create a Cloud SQL instance. We will use a Cloud SQL first generation in this example. To be make sure your Spark cluster can access your Cloud SQL database, you must:
cluster is the name of your cluster and i is a slave number.After you create and connect to an instance, you need to create some tables and load data into some of them by following these steps:
solutions-public-assets/recommendation-spark/sql/table_creation.sql. If for some reason, the UI says that you have no access, you can also copy the file to your own bucket. In a CloudShell window or in a terminal type gsutil cp gs://solutions-public-assets/recommendation-spark/sql/table_creation.sql gs://<your-bucket>/recommendation-spark/sql/table_creation.sql. Then, in the Cloud SQL import window, provide <your-bucket>/recommendation-spark/sql/table_creation.sql (i.e the path to your copy of the file on Google Storage, without the gs:// prefix).
The appengine folder contains a simple HTML website built with Python on App Engine using Angular Material. While it is not required to deploy this website, it can give you an idea of what a recommendation display could look like in a production environment.
From the appengine folder you can run
gcloud app deploy --project [YOUR-PROJECT-ID]Make sure to update your database values in the main.py file to match your setup. If you kept the values of the .sql script, _DB_NAME = 'recommendation_spark'. The rest will be specific to your setup.
You can find some accommodation images here. Upload the individual files to your own bucket and change their acl to be public in order to serve them out. Remember to replace <YOUR_IMAGE_BUCKET> in appengine/app/templates/welcome.html page with your bucket.
The main part of this solution paper is explained on the Cloud Platform solution page. In the pyspark folder, you will find the scripts mentionned in the solution paper:
Both scripts should be run in a Spark cluster. This can be done by using Cloud Dataproc.
Dataproc already has the MySQL connector enabled so there is no need to set it up.
The easiest way is to use the Cloud Console and run the script directly from a remote location (Cloud Storage for example). See the documentation.
It is also possible to run this command line from a local computer:
$ gcloud dataproc jobs submit pyspark \
--cluster <YOUR_DATAPROC_CLUSTER_NAME> \
find_model_collaborative.py \
-- <YOUR_CLOUDSQL_INSTANCE_IP> \
<YOUR_CLOUDSQL_DB_NAME> \
<YOUR_CLOUDSQL_USER> \
<YOUR_CLOUDSQL_PASSWORD>The script above returns a combination of the best parameters for the ALS training, as explained in the Training the model part of the solution article. It will be displayed in the console in the following format, where Dist represents how far we are from being the known value. The result might not feel satisfying but remember that the training dataset was quite small.
After you have those values, you can reuse them when calling the recommendation script.
# Build our model with the best found values
model = ALS.train(rddTraining, BEST_RANK, BEST_ITERATION, BEST_REGULATION)Where, in our current case, BEST_RANK=15, BEST_ITERATION=20, BEST_REGULATION=0.1.
Run the app_collaborative.py file with the updated values as you did before. The code makes a prediction and saves the top 5 expected rates in Cloud SQL. You can look at the results later.
You can use the Cloud Console, as explained before, which would be equivalent of running the following script from your local computer.
$ gcloud dataproc jobs submit pyspark \
--cluster <YOUR_DATAPROC_CLUSTER_NAME> \
app_collaborative.py \
-- <YOUR_CLOUDSQL_INSTANCE_IP> \
<YOUR_CLOUDSQL_DB_NAME> \
<YOUR_CLOUDSQL_USER> \
<YOUR_CLOUDSQL_PASSWORD> \
<YOUR_BEST_RANK> \
<YOUR_BEST_ITERATION> \
<YOUR_BEST_REGULATION>The code posted in GitHub prints the top 5 predictions. You should see something similar to a list of tuples, including userId, accoId, and prediction:
[('0', '75', 4.6428704512729375), ('0', '76', 4.54325166163637), ('0', '86', 4.529177571208829), ('0', '66', 4.52387350189572), ('0', '99', 4.44705391172443)]To easily access a MySql CLI, you can use Cloud Shell and type the following command line then enter your database password.
gcloud sql connect <YOUR_CLOUDSQL_INSTANCE_NAME> --user=<YOUR_CLOUDSQL_USER>Running the following SQL query on the database will return the predictions saved in the Recommendation table by app_collaborative.py:
SELECT
id, title, type, r.prediction
FROM
Accommodation a
INNER JOIN
Recommendation r
ON
r.accoId = a.id
WHERE
r.userId = <USER_ID>
ORDER BY
r.prediction desc