Metric is a function that compares the predictions with the actual values and outputs a single number that tells how good the predictions are.
Inputs are predictions and true values. Outputs are a single number.
Different metrics look at different aspects of the predictions.
This week's work used the same telecom churn data set as last week. We began this week by reimporting the data and going through the data preparation steps in a systematic manner.
After removing spaces and making everything lowercase, I set up the training, validation, and test groups. After the validation framework was set up, I deleted the target churn column, so that the data would not be trained on answers already there.
The decision threshold in machine learning is a crucial factor in classifying data into binary classes.
It is the value that determines how likely an instance is to be classified as a positive or negative class based on the model's output probabilities.
For example, in binary classification, a model may output a probability score between 0 and 1, and if the score exceeds the threshold, the instance is classified as positive; otherwise, it is classified as negative.
Just before training the model, I encountered a Python ValueError relating to the y_train data array. On inquiry, Gemini spotted that y_train was all zeros, and it should be ones and zeros, for binary classification. It also pointed out:
None of these recommendations got rid of the ValueError, though it was helpful that Gemini pointed out that y_train was entirely zeros. When it became clear that of the target churn values had been set zero, I turned to Claude.
Claude had a difference of opinion, stating "Gemini’s suggestions about random_state and shuffling do not apply here. Your split code is correct; the target column was already all zeros before the split ran."
Claude then came back with the following useful advice: " On the first run, churn holds the strings 'yes' and 'no', and the line produces the correct 0s and 1s. If the cell runs again without df being reloaded from the CSV, churn already holds integers. No integer equals the string 'yes', so every comparison is False and the whole column becomes 0. Nothing raises an error at that point; the target is silently wiped."
The default threshold is often set to 0.5, but this value may not be optimal for all scenarios, especially in cases with imbalanced classes or where the cost of false negatives outweighs the cost of false positives.