Data Science

Logging Metrics with MLflow

Machine learning development must include logging and monitoring metrics with MLflow. Logging metrics using MLflow makes it possible to compare multiple runs, monitor the performance of the models, and make defensible decisions in light of the results. A well-liked tool for monitoring machine learning experiments is MLflow. It allows us to log parameters, metrics, and artifacts from training runs, so you can easily compare and analyze them later. In addition to logging metrics, we can also use MLflow to log parameters and artifacts.

Function to Log Metrics

In MLflow, the mlflow.log_metrics() function is used to log the metrics, which accept a dictionary as an argument, includiding key/name and value pairs.

Pre-requisites

Install the MLflow with the help of the pip command if you haven’t already installed it on the required machine.

pip install mlflow

Import Required Libraries

Import the necessary MLflow library in the python code file.

Example 1

Here is a straightforward example where the code logs the accuracy and loss metrics for a training model in the current run. We can then view these metrics in the MLflow UI or by using the MLflow API:

Code:

import mlflow

def log_model_metrics():

metrics = {"accuracy": 0.9, "loss": 0.1}

mlflow.log_metrics(metrics)

if __name__ == "__main__":

log_model_metrics()

Parameters are key-value pairs that represent the input settings for the training run. Artifacts are arbitrary files that are produced by a training run. We can use the mlflow.log_param() and mlflow to log parameters and artifacts.log_artifact() functions, respectively.

Run the code mentioned above using the bash or python compiler by executing the following commands:

python LogMetrics.py

After the code has been successfully executed, MLflow should log the metrics in the corresponding run folder. This generates a separate file and stores the value for each metric. Below is a screenshot of the directory showing that the run generated two files, accuracy and loss, within the metrics folder:

Example 2

In this example, we will provide a step-by-step guide on how to log and track metrics with MLflow for basic calculator operations:

Scenario:

Here is a simple scenario for a calculator to log results and other information:

  • The user enters two numbers and an operation (+, -, *, or /).
  • The calculation is done on the calculator, and the outcome is recorded in a file.
  • The calculator additionally records the two numbers, the operation, and the time the computation was done.
  • The calculator can also log other information, such as the user’s name, the computer’s IP address, and the browser used.

Import MLflow

Import the essential MLflow library in Python code:

  • The logging module is used for logging information to the console and a file.
  • The time module is used to get the current time.
  • The mlflow module logs metrics, artifacts, and parameters to MLflow.

Code:

import logging

import time

import mlflow

Create Log_Calculation Function

The log_calculation() function is required to input four arguments:

  • Operation: The operation that was performed, such as +, -, *, or /.
  • num1: The first number.
  • num2: The second number.
  • Result: The result of the calculation.

The function aims to carry out the calculator’s fundamental tasks and log the information to mlflow metrics, parameters, and artifacts.

Code:

def log_calculation(operation, num1, num2, result):

Log Information Using Logging

Using the logging module, a common Python module, the code reports the calculation to the console. The info() function logs an INFO level message, including the operation, two numbers, result, and time. The time.strftime() function is used to get the current time in YYYY-MM-DDTHH:MM:SS format. The code then opens the file calculations.log in append mode, writes the calculation, operation, two numbers, result, and time, and closes the file.

Code:

logging.info("Calculation: %s %s %s = %s", operation, num1, num2, result)

with open("calculations.log", "a") as f:

f.write("Calculation: %s %s %s = %s\n" % (operation, num1, num2, result))

f.write("Operation: %s\n" % operation)

f.write("Num1: %s\n" % num1)

f.write("Num2: %s\n" % num2)

f.write("Result: %s\n" % result)

f.write("Time: %s\n" % time.strftime("%Y-%m-%dT%H:%M:%SZ"))

Set the Tracking URI

Define calc_tracking_uri as MLflow tracking server URL and use the mlflow.set_tracking_uri() function for the current run, storing metrics, artifacts, and parameters.

Code:

calc_tracking_uri = "TRACKING_URI"

mlflow.set_tracking_uri(calc_tracking_uri)

Start a Run

At the beginning of an experiment, start an MLflow run to log metrics:

  • It creates a new MLflow run. An MLflow run is a single execution of an experiment.
  • It sets the name of the run to “Basic Calculator Run”.

Code:

with mlflow.start_run(run_name="Basic Calculator Run"):

Log Metrics

Inside the run context, we can log various metrics using the mlflow.log_metric() function. For example:

  • It logs three metrics called First_Number, Second_Number, and Result, along with the values of variables num1, num2 and result to the current MLflow run.

Code:

mlflow.log_metric("First_Number", num1)

mlflow.log_metric("Second_Number", num2)

mlflow.log_metric("Result", result)

Log Parameters

To log parameters used for the experiment using the mlflow.log_param() function:

  • It logs a parameter to the current MLflow run.
  • The parameter is called “Operation” and it is assigned the value of the variable operation.

Code:

mlflow.log_param("Operation", operation)

Log Artifacts

To log the artifacts, including images, files, etc., we use the mlflow.log_artifact() function:

  • The artifact is called “calculations.log,” and it is the file that contains the calculations that were performed.

Code:

mlflow.log_artifact("calculations.log")

Set Parameters

In this case, the name of the tag is self-explanatory. The tag’s value is the name of the user who ran the calculator.

The other two lines of code work similarly. They set the tags “ip_address” and “browser” to the values “127.0.0.1” and “Mozilla”, respectively.

Code:

mlflow.set_tag("User", "Test")

mlflow.set_tag("ip_address", "127.0.0.1")

mlflow.set_tag("browser", "Mozilla")

Call the Main Function

In this case, the code in the if __name__ == “__main__”: block is used to log the results of four calculations to MLflow. The calculations are addition, subtraction, multiplication, and division. The results of the calculations are 30, -10, 200, and 0.5, respectively.

Code:

if __name__ == "__main__":

log_calculation("+", 10, 20, 30)

log_calculation("-", 10, 20, -10)

log_calculation("*", 10, 20, 200)

log_calculation("/", 10, 20, 0.5)

Logging with Context Managers

MLflow provides context managers that can be used to log metrics and parameters:

Viewing Runs and Metrics

To view the logged metrics, parameters, and artifacts using the MLflow UI. Run the following command in the terminal or prompt window to launch the UI:

mlflow ui

This will start a local web server where we can access the MLflow UI by opening a web browser and navigating to http://localhost:5000.

Here is the output of the code with UI:

Conclusion

By seamlessly integrating the mlflow.log_metric() function, developers can effortlessly record and monitor numeric measurements, such as accuracy, precision, and recall, across various model iterations. This logging mechanism empowers teams to analyze and compare model performance, make informed decisions, and drive iterative improvements, fostering a more organized, collaborative, and data-driven approach to machine learning experimentation and development.

About the author

Kalsoom Bibi

Hello, I am a freelance writer and usually write for Linux and other technology related content