Understanding and calculating the F1 score in ML

In machine learning, evaluating the performance of classification models is crucial for ensuring their accuracy and reliability. One of the most significant metrics used for this purpose is the F1 score, also known as the F-measure. This guide will explain what the F1 score is, how it is calculated, its interpretation, and its variations, particularly in multi-class classification scenarios.

Understanding the F1 score

The F1 score is a metric that combines precision and recall to provide a balanced measure of a classification model's performance. It is defined as the harmonic mean of precision and recall, ensuring that both metrics are given equal weight.

Precision and recall

Before diving into the F1 score, it is essential to understand precision and recall:

• Precision: This is the ratio of true positives (TP) to the sum of true positives and false positives (FP). It is calculated as ( \text{Precision} = \frac{TP}{TP + FP} ).
• Recall: This is the ratio of true positives to the sum of true positives and false negatives (FN). It is calculated as ( \text{Recall} = \frac{TP}{TP + FN} ).

Calculating the F1 score

The F1 score is calculated using the harmonic mean of precision and recall. Here is the formula:

[ F1 \text{ score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} ]

This can also be expressed in terms of true positives, false positives, and false negatives:

[ F1 \text{ score} = 2 \times \frac{\frac{TP}{TP + FP} \times \frac{TP}{TP + FN}}{\frac{TP}{TP + FP} + \frac{TP}{TP + FN}} ]

Simplifying this, we get:

[ F1 \text{ score} = \frac{2 \times TP}{2 \times TP + FP + FN} ].

Why use the harmonic mean?

The harmonic mean is used instead of the arithmetic mean because it ensures that the F1 score is influenced more by the smaller of the two values (precision and recall). This is crucial because it prevents the overestimation that could occur if the arithmetic mean were used, especially when precision and recall have significantly different values.

Interpreting the F1 score

The F1 score ranges from 0 to 1, where 0 indicates the worst possible performance and 1 indicates perfect performance. Here’s how to interpret different F1 score values:

• High F1 score: A high F1 score (generally above 0.7) indicates that the model has a good balance between precision and recall, meaning it can effectively identify positive cases while minimizing false positives and false negatives.
• Low F1 score: A low F1 score suggests a trade-off between precision and recall, indicating that the model may have trouble balancing these two metrics.

Variations of F1 score in multi-class classification

In multi-class classification, calculating the F1 score involves different averaging methods to obtain a single overall score.

Macro average F1 score

The macro-averaged F1 score is calculated by taking the arithmetic mean of all per-class F1 scores. This method treats all classes equally, regardless of their support (the number of actual occurrences of each class).

Weighted average F1 score

The weighted-averaged F1 score considers the support of each class and calculates the mean of all per-class F1 scores weighted by their support. This method is particularly useful for imbalanced datasets where the support of classes varies significantly.

Micro average F1 score

The micro-averaged F1 score computes a global F1 score by summing the true positives, false positives, and false negatives across all classes and then applying the F1 formula. This method is equivalent to calculating accuracy in binary classification but extends to multi-class scenarios.

Limitations of the F1 score

While the F1 score is a powerful metric, it has some limitations:

Dataset class imbalance

The regular F1 score may not accurately represent the model's performance in datasets with significant class imbalance. Here, the F-beta score, which allows adjusting the weight given to precision and recall, can be more appropriate.

F-beta score

The F-beta score is a variant of the F1 score that allows for a dynamic blend of recall and precision by adjusting the beta parameter. This can be useful in scenarios where either precision or recall is more critical than the other.

Calculating F1 score in Python

Calculating the F1 score in Python is straightforward using the scikit-learn library. Here is an example:

from sklearn.metrics import f1_score

# Example true labels and predicted labels true_labels = [0, 1, 2, 0, 1, 2] predicted_labels = [0, 2, 1, 0, 0, 1]

# Calculate macro-averaged F1 score macro_f1 = f1_score(true_labels, predicted_labels, average='macro')

# Calculate weighted-averaged F1 score weighted_f1 = f1_score(true_labels, predicted_labels, average='weighted')

# Calculate micro-averaged F1 score micro_f1 = f1_score(true_labels, predicted_labels, average='micro')

print(f"Macro F1 Score: {macro_f1}, Weighted F1 Score: {weighted_f1}, Micro F1 Score: {micro_f1}")

Practical applications of F1 score

The F1 score is widely used in machine learning for evaluating the performance of classification models. It provides a balanced view of precision and recall, making it particularly useful in scenarios where both metrics are critical. Understanding how to calculate and interpret the F1 score, along with its variations in multi-class classification, is essential for developing and optimizing robust classification models.

Contact our team of experts to discover how Telnyx can power your AI solutions.

___________________________________________________________________________________

Sources Cited

• Encord. "F1 Score in Machine Learning Explained." Encord, 18 July 2023, https://encord.com/blog/f1-score-in-machine-learning/.
• KDnuggets. "Micro, Macro & Weighted Averages of F1 Score, Clearly Explained." KDnuggets, 4 January 2023, https://www.kdnuggets.com/2023/01/micro-macro-weighted-averages-f1-score-clearly-explained.html.
• LinkedIn. "How to Calculate and Use the F1 Score for Machine Learning." LinkedIn, 23 December 2023, https://www.linkedin.com/advice/3/how-do-you-calculate-f1-score-machine-learning-6ngoe.
• Serokell. "F1 Score in Machine Learning - Serokell." Serokell, 10 July 2023, https://serokell.io/blog/a-guide-to-f1-score.
• V7 Labs. "F1 Score in Machine Learning: Intro & Calculation - V7 Labs." V7 Labs, 16 December 2022, https://www.v7labs.com/blog/f1-score-guide.

#