Skip to content

An end to end ML project. Using MLflow for experiment tracking and model registry. Prefect for workflow orchestration. S3 for artifacts storage. AWS Lambda/ ECR for serverless model serving. AWS REST API gateway as endpoint to lambda function. GitHub Actions for CI/CD.

Notifications You must be signed in to change notification settings

Nerdward/Mlops_project

Repository files navigation

Problem Description

A Real Estate company have data on some properties i.e house. Data like the number of bedrooms, size of the front porch etc. They want a model that predicts the price of the house given the house features. This is a regression task. The machine learning engineer (me) sourced data from Kaggle. The dataset contains over 100 columns of different features of houses. He cleaned the data and trained an xgboost model on it.

Cloud

Cloud Infrastructure: cloud

Reproducibility

Run this command to setup the python environment and pre-commit

make setup

Experiment tracking and model registry

MLflow was used for experiment traccking and model registering.

To start up the mlflow server, run

bash orchestration/server.sh

keep this terminal open for the training pipeline

Workflow Orchestration

Prefect was used for workflow orchestration to get the data, preprocess it, run hyperparameter tuning with hyperopt and train the xbgoost model.

To start the training pipeline:

bash orchestration/train.sh

Model deployment

Checkout from develop branch to ci-cd branch

git checkout -b ci-cd develop

Change the infrastructure production variables to your choice.

Commit and push the changes to the ci-cd remote branch Make a pull request and merge with develop branch

Testing

terraform init -backend-config="key=mlops-zoomcamp-prod.tfstate" -reconfigure
terraform output rest_api_url

Change the url in the deployment test file to the rest api url

About

An end to end ML project. Using MLflow for experiment tracking and model registry. Prefect for workflow orchestration. S3 for artifacts storage. AWS Lambda/ ECR for serverless model serving. AWS REST API gateway as endpoint to lambda function. GitHub Actions for CI/CD.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published