dacapo.experiments.tasks.evaluators.binary_segmentation_evaluator

Attributes

logger

BG

Classes

BinarySegmentationEvaluator

Given a binary segmentation, compute various metrics to determine their similarity. The metrics include:

ArrayEvaluator

Given a binary segmentation, compute various metrics to determine their similarity. The metrics include:

CremiEvaluator

Evaluate the performance of a binary segmentation task using the CREMI score.

Module Contents

dacapo.experiments.tasks.evaluators.binary_segmentation_evaluator.logger
dacapo.experiments.tasks.evaluators.binary_segmentation_evaluator.BG = 0
class dacapo.experiments.tasks.evaluators.binary_segmentation_evaluator.BinarySegmentationEvaluator(clip_distance: float, tol_distance: float, channels: List[str])

Given a binary segmentation, compute various metrics to determine their similarity. The metrics include: - Dice coefficient: 2 * |A ∩ B| / |A| + |B| ; where A and B are the binary segmentations - Jaccard coefficient: |A ∩ B| / |A ∪ B| ; where A and B are the binary segmentations - Hausdorff distance: max(h(A, B), h(B, A)) ; where h(A, B) is the Hausdorff distance between A and B - False negative rate: |A - B| / |A| ; where A and B are the binary segmentations - False positive rate: |B - A| / |B| ; where A and B are the binary segmentations - False discovery rate: |B - A| / |A| ; where A and B are the binary segmentations - VOI: Variation of Information; split and merge errors combined into a single measure of segmentation quality - Mean false distance: 0.5 * (mean false positive distance + mean false negative distance) - Mean false negative distance: mean distance of false negatives - Mean false positive distance: mean distance of false positives - Mean false distance clipped: 0.5 * (mean false positive distance clipped + mean false negative distance clipped) ; clipped to a maximum distance - Mean false negative distance clipped: mean distance of false negatives clipped ; clipped to a maximum distance - Mean false positive distance clipped: mean distance of false positives clipped ; clipped to a maximum distance - Precision with tolerance: TP / (TP + FP) ; where TP and FP are the true and false positives within a tolerance distance - Recall with tolerance: TP / (TP + FN) ; where TP and FN are the true and false positives within a tolerance distance - F1 score with tolerance: 2 * (Recall * Precision) / (Recall + Precision) ; where Recall and Precision are the true and false positives within a tolerance distance - Precision: TP / (TP + FP) ; where TP and FP are the true and false positives - Recall: TP / (TP + FN) ; where TP and FN are the true and false positives - F1 score: 2 * (Recall * Precision) / (Recall + Precision) ; where Recall and Precision are the true and false positives

clip_distance

float the clip distance

tol_distance

float the tolerance distance

channels

List[str] the channels

criteria

List[str] the evaluation criteria

evaluate(output_array_identifier, evaluation_array)

Evaluate the output array against the evaluation array.

score()

Return the evaluation scores.

Note

The BinarySegmentationEvaluator class is used to evaluate the performance of a binary segmentation task. The class provides methods to evaluate the output array against the evaluation array and return the evaluation scores. All evaluation scores should inherit from this class.

Clip distance is the maximum distance between the ground truth and the predicted segmentation for a pixel to be considered a false positive. Tolerance distance is the maximum distance between the ground truth and the predicted segmentation for a pixel to be considered a true positive. Channels are the channels of the binary segmentation. Criteria are the evaluation criteria.

criteria = ['jaccard', 'voi']

A list of all criteria for which a model might be “best”. i.e. your criteria might be “precision”, “recall”, and “jaccard”. It is unlikely that the best iteration/post processing parameters will be the same for all 3 of these criteria

Returns:

List[str]

the evaluation criteria

Raises:

NotImplementedError – if the function is not implemented

Examples

>>> evaluator = Evaluator()
>>> evaluator.criteria
[]

Note

This function is used to return the evaluation criteria.

clip_distance
tol_distance
channels
evaluate(output_array_identifier, evaluation_array)

Evaluate the output array against the evaluation array.

Parameters:

  • output_array_identifier – str the identifier of the output array

  • evaluation_array – ZarrArray the evaluation array

Returns:

BinarySegmentationEvaluationScores or MultiChannelBinarySegmentationEvaluationScores

the evaluation scores

Raises:

ValueError – if the output array identifier is not valid

Examples

>>> binary_segmentation_evaluator = BinarySegmentationEvaluator(clip_distance=200, tol_distance=40, channels=["channel1", "channel2"])
>>> output_array_identifier = "output_array"
>>> evaluation_array = ZarrArray.open_from_array_identifier("evaluation_array")
>>> binary_segmentation_evaluator.evaluate(output_array_identifier, evaluation_array)
BinarySegmentationEvaluationScores(dice=0.0, jaccard=0.0, hausdorff=0.0, false_negative_rate=0.0, false_positive_rate=0.0, false_discovery_rate=0.0, voi=0.0, mean_false_distance=0.0, mean_false_negative_distance=0.0, mean_false_positive_distance=0.0, mean_false_distance_clipped=0.0, mean_false_negative_distance_clipped=0.0, mean_false_positive_distance_clipped=0.0, precision_with_tolerance=0.0, recall_with_tolerance=0.0, f1_score_with_tolerance=0.0, precision=0.0, recall=0.0, f1_score=0.0)

Note

This function is used to evaluate the output array against the evaluation array.

property score
Return the evaluation scores.
Returns:

BinarySegmentationEvaluationScores or MultiChannelBinarySegmentationEvaluationScores

the evaluation scores

Raises:

NotImplementedError – if the function is not implemented

Examples

>>> binary_segmentation_evaluator = BinarySegmentationEvaluator(clip_distance=200, tol_distance=40, channels=["channel1", "channel2"])
>>> binary_segmentation_evaluator.score
BinarySegmentationEvaluationScores(dice=0.0, jaccard=0.0, hausdorff=0.0, false_negative_rate=0.0, false_positive_rate=0.0, false_discovery_rate=0.0, voi=0.0, mean_false_distance=0.0, mean_false_negative_distance=0.0, mean_false_positive_distance=0.0, mean_false_distance_clipped=0.0, mean_false_negative_distance_clipped=0.0, mean_false_positive_distance_clipped=0.0, precision_with_tolerance=0.0, recall_with_tolerance=0.0, f1_score_with_tolerance=0.0, precision=0.0, recall=0.0, f1_score=0.0)

Note

This function is used to return the evaluation scores.

class dacapo.experiments.tasks.evaluators.binary_segmentation_evaluator.ArrayEvaluator(truth_binary, test_binary, truth_empty, test_empty, metric_params, resolution)

Given a binary segmentation, compute various metrics to determine their similarity. The metrics include: - Dice coefficient: 2 * |A ∩ B| / |A| + |B| ; where A and B are the binary segmentations - Jaccard coefficient: |A ∩ B| / |A ∪ B| ; where A and B are the binary segmentations - Hausdorff distance: max(h(A, B), h(B, A)) ; where h(A, B) is the Hausdorff distance between A and B - False negative rate: |A - B| / |A| ; where A and B are the binary segmentations - False positive rate: |B - A| / |B| ; where A and B are the binary segmentations - False discovery rate: |B - A| / |A| ; where A and B are the binary segmentations - VOI: Variation of Information; split and merge errors combined into a single measure of segmentation quality - Mean false distance: 0.5 * (mean false positive distance + mean false negative distance) - Mean false negative distance: mean distance of false negatives - Mean false positive distance: mean distance of false positives - Mean false distance clipped: 0.5 * (mean false positive distance clipped + mean false negative distance clipped) ; clipped to a maximum distance - Mean false negative distance clipped: mean distance of false negatives clipped ; clipped to a maximum distance - Mean false positive distance clipped: mean distance of false positives clipped ; clipped to a maximum distance - Precision with tolerance: TP / (TP + FP) ; where TP and FP are the true and false positives within a tolerance distance - Recall with tolerance: TP / (TP + FN) ; where TP and FN are the true and false positives within a tolerance distance - F1 score with tolerance: 2 * (Recall * Precision) / (Recall + Precision) ; where Recall and Precision are the true and false positives within a tolerance distance - Precision: TP / (TP + FP) ; where TP and FP are the true and false positives - Recall: TP / (TP + FN) ; where TP and FN are the true and false positives - F1 score: 2 * (Recall * Precision) / (Recall + Precision) ; where Recall and Precision are the true and false positives

truth

np.ndarray the truth binary segmentation

test

np.ndarray the test binary segmentation

truth_empty

bool whether the truth binary segmentation is empty

test_empty

bool whether the test binary segmentation is empty

cremieval

CremiEvaluator the cremi evaluator

resolution

Tuple[float, float, float] the resolution

dice()

Return the Dice coefficient.

jaccard()

Return the Jaccard coefficient.

hausdorff()

Return the Hausdorff distance.

false_negative_rate()

Return the false negative rate.

false_positive_rate()

Return the false positive rate.

false_discovery_rate()

Return the false discovery rate.

precision()

Return the precision.

recall()

Return the recall.

f1_score()

Return the F1 score.

voi()

Return the VOI.

mean_false_distance()

Return the mean false distance.

mean_false_negative_distance()

Return the mean false negative distance.

mean_false_positive_distance()

Return the mean false positive distance.

mean_false_distance_clipped()

Return the mean false distance clipped.

mean_false_negative_distance_clipped()

Return the mean false negative distance clipped.

mean_false_positive_distance_clipped()

Return the mean false positive distance clipped.

false_positive_rate_with_tolerance()

Return the false positive rate with tolerance.

false_negative_rate_with_tolerance()

Return the false negative rate with tolerance.

precision_with_tolerance()

Return the precision with tolerance.

recall_with_tolerance()

Return the recall with tolerance.

f1_score_with_tolerance()

Return the F1 score with tolerance.

Note

The ArrayEvaluator class is used to evaluate the performance of a binary segmentation task. The class provides methods to evaluate the truth binary segmentation against the test binary segmentation. All evaluation scores should inherit from this class.

truth
test
truth_empty
test_empty
cremieval
resolution
truth_itk()

A SimpleITK image of the truth binary segmentation.

Returns:

sitk.Image

the truth binary segmentation as a SimpleITK image

Examples

>>> array_evaluator = ArrayEvaluator(truth_binary, test_binary, truth_empty, test_empty, metric_params, resolution)
>>> array_evaluator.truth_itk
<SimpleITK.SimpleITK.Image; proxy of <Swig Object of type 'std::vector< itk::simple::Image >::value_type *' at 0x7f8b1c0b3f30> >

Note

This function is used to return the truth binary segmentation as a SimpleITK image.

test_itk()

A SimpleITK image of the test binary segmentation.

Parameters:

  • test – np.ndarray the test binary segmentation

  • resolution – Tuple[float, float, float] the resolution

Returns:

sitk.Image

the test binary segmentation as a SimpleITK image

Raises:

ValueError – if the test binary segmentation is not valid

Examples

>>> array_evaluator = ArrayEvaluator(truth_binary, test_binary, truth_empty, test_empty, metric_params, resolution)
>>> array_evaluator.test_itk
<SimpleITK.SimpleITK.Image; proxy of <Swig Object of type 'std::vector< itk::simple::Image >::value_type *' at 0x7f8b1c0b3f30> >

Note

This function is used to return the test binary segmentation as a SimpleITK image.

overlap_measures_filter()

A SimpleITK filter to compute overlap measures.

Parameters:

  • truth_itk – sitk.Image the truth binary segmentation as a SimpleITK image

  • test_itk – sitk.Image the test binary segmentation as a SimpleITK image

Returns:

sitk.LabelOverlapMeasuresImageFilter

the overlap measures filter

Raises:

ValueError – if the truth binary segmentation or the test binary segmentation is not valid

Examples

>>> array_evaluator = ArrayEvaluator(truth_binary, test_binary, truth_empty, test_empty, metric_params, resolution)
>>> array_evaluator.overlap_measures_filter
<SimpleITK.SimpleITK.LabelOverlapMeasuresImageFilter; proxy of <Swig Object of type 'itk::simple::LabelOverlapMeasuresImageFilter *' at 0x7f8b1c0b3f30> >

Note

This function is used to return the overlap measures filter.

dice()

The Dice coefficient.

Parameters:

  • truth_itk – sitk.Image the truth binary segmentation as a SimpleITK image

  • test_itk – sitk.Image the test binary segmentation as a SimpleITK image

Returns:

float

the Dice coefficient

Raises:

ValueError – if the truth binary segmentation or the test binary segmentation is not valid

Examples

>>> array_evaluator = ArrayEvaluator(truth_binary, test_binary, truth_empty, test_empty, metric_params, resolution)
>>> array_evaluator.dice()
0.0

Note

This function is used to return the Dice coefficient.

jaccard()

The Jaccard coefficient.

Parameters:

  • truth_itk – sitk.Image the truth binary segmentation as a SimpleITK image

  • test_itk – sitk.Image the test binary segmentation as a SimpleITK image

Returns:

float

the Jaccard coefficient

Raises:

ValueError – if the truth binary segmentation or the test binary segmentation is not valid

Examples

>>> array_evaluator = ArrayEvaluator(truth_binary, test_binary, truth_empty, test_empty, metric_params, resolution)
>>> array_evaluator.jaccard()
0.0

Note

This function is used to return the Jaccard coefficient.

hausdorff()

The Hausdorff distance.

Parameters:

None

Returns:

the Hausdorff distance

Return type:

float

Raises:

None –

Examples

>>> array_evaluator = ArrayEvaluator(truth_binary, test_binary, truth_empty, test_empty, metric_params, resolution)
>>> array_evaluator.hausdorff()
0.0

Note

This function is used to return the Hausdorff distance between the truth binary segmentation and the test binary segmentation.

If either the truth or test binary segmentation is empty, the function returns 0. Otherwise, it calculates the Hausdorff distance using the HausdorffDistanceImageFilter from the SimpleITK library.

false_negative_rate()

The false negative rate.

Returns:

float

the false negative rate

Returns:

if the truth binary segmentation or the test binary segmentation is not valid

Return type:

ValueError

Examples

>>> array_evaluator = ArrayEvaluator(truth_binary, test_binary, truth_empty, test_empty, metric_params, resolution)
>>> array_evaluator.false_negative_rate()
0.0

Note

This function is used to return the false negative rate.

false_positive_rate()

The false positive rate.

Parameters:

  • truth_itk – sitk.Image the truth binary segmentation as a SimpleITK image

  • test_itk – sitk.Image the test binary segmentation as a SimpleITK image

Returns:

float

the false positive rate

Raises:

ValueError – if the truth binary segmentation or the test binary segmentation is not valid

Examples

>>> array_evaluator = ArrayEvaluator(truth_binary, test_binary, truth_empty, test_empty, metric_params, resolution)
>>> array_evaluator.false_positive_rate()
0.0

Note

This function is used to return the false positive rate.

false_discovery_rate()

Calculate the false discovery rate (FDR) for the binary segmentation evaluation.

Returns:

The false discovery rate.

Return type:

float

Raises:

None –

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.false_discovery_rate()
0.25

Note

The false discovery rate is a measure of the proportion of false positives among the predicted positive samples. It is calculated as the ratio of false positives to the sum of true positives and false positives. If either the ground truth or the predicted segmentation is empty, the FDR is set to NaN.

precision()

Calculate the precision of the binary segmentation evaluation.

Returns:

The precision value.

Return type:

float

Raises:

None. –

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.precision()
0.75

Note

Precision is a measure of the accuracy of the positive predictions made by the model. It is calculated as the ratio of true positives to the total number of positive predictions. If either the ground truth or the predicted values are empty, the precision value will be NaN.

recall()

Calculate the recall metric for binary segmentation evaluation.

Returns:

The recall value.

Return type:

float

Raises:

None –

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.recall()
0.75

Note

Recall is a measure of the ability of a binary classifier to identify all positive samples. It is calculated as the ratio of true positives to the total number of actual positives.

f1_score()

Calculate the F1 score for binary segmentation evaluation.

Returns:

The F1 score value.

Return type:

float

Raises:

None. –

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.f1_score()
0.75

Note

The F1 score is the harmonic mean of precision and recall. It is a measure of the balance between precision and recall, providing a single metric to evaluate the model’s performance.

If either the ground truth or the predicted values are empty, the F1 score will be NaN.

voi()

Calculate the Variation of Information (VOI) for binary segmentation evaluation.

Parameters:

  • truth – np.ndarray the truth binary segmentation

  • test – np.ndarray the test binary segmentation

Returns:

The VOI value.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.voi()
0.75

Note

The VOI is a measure of the similarity between two segmentations. It combines the split and merge errors into a single measure of segmentation quality. If either the ground truth or the predicted values are empty, the VOI will be NaN.

mean_false_distance()

Calculate the mean false distance between the ground truth and the test results.

Parameters:

  • truth – np.ndarray the truth binary segmentation

  • test – np.ndarray the test binary segmentation

Returns:

The mean false distance.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.mean_false_distance()
0.25

Note

  • This method returns np.nan if either the ground truth or the test results are empty.

  • The mean false distance is a measure of the average distance between the false positive pixels in the test results and the nearest true positive pixels in the ground truth.

mean_false_negative_distance()

Calculate the mean false negative distance between the ground truth and the test results.

Parameters:

  • truth – np.ndarray the truth binary segmentation

  • test – np.ndarray the test binary segmentation

Returns:

The mean false negative distance.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.mean_false_negative_distance()
0.25

Note

This method returns np.nan if either the ground truth or the test results are empty.

mean_false_positive_distance()

Calculate the mean false positive distance.

This method calculates the mean false positive distance between the ground truth and the test results. If either the ground truth or the test results are empty, the method returns NaN.

Parameters:

  • truth – np.ndarray the truth binary segmentation

  • test – np.ndarray the test binary segmentation

Returns:

The mean false positive distance.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.mean_false_positive_distance()
0.5

Note

The mean false positive distance is a measure of the average distance between false positive pixels in the test results and the corresponding ground truth pixels. It is commonly used to evaluate the performance of binary segmentation algorithms.

mean_false_distance_clipped()

Calculate the mean false distance (clipped) between the ground truth and the test results.

Parameters:

  • truth – np.ndarray the truth binary segmentation

  • test – np.ndarray the test binary segmentation

Returns:

The mean false distance (clipped) value.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.mean_false_distance_clipped()
0.123

Note

This method returns np.nan if either the ground truth or the test results are empty.

mean_false_negative_distance_clipped()

Calculate the mean false negative distance, with clipping.

This method calculates the mean false negative distance between the ground truth and the test results. The distance is clipped to avoid extreme values.

Parameters:

  • truth – np.ndarray the truth binary segmentation

  • test – np.ndarray the test binary segmentation

Returns:

The mean false negative distance with clipping.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.mean_false_negative_distance_clipped()
0.123

Note

  • The mean false negative distance is a measure of the average distance between the false negative pixels in the ground truth and the test results.

  • Clipping the distance helps to avoid extreme values that may skew the overall evaluation.

mean_false_positive_distance_clipped()

Calculate the mean false positive distance, with clipping.

This method calculates the mean false positive distance between the ground truth and the test results, taking into account any clipping that may have been applied.

Parameters:

  • truth – np.ndarray the truth binary segmentation

  • test – np.ndarray the test binary segmentation

Returns:

The mean false positive distance with clipping.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.mean_false_positive_distance_clipped()
0.25

Note

  • The mean false positive distance is a measure of the average distance between false positive pixels in the test results and the corresponding ground truth pixels.

  • If either the ground truth or the test results are empty, the method returns NaN.

false_positive_rate_with_tolerance()

Calculate the false positive rate with tolerance.

Parameters:

  • truth – np.ndarray the truth binary segmentation

  • test – np.ndarray the test binary segmentation

Returns:

The false positive rate with tolerance.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.false_positive_rate_with_tolerance()
0.25

Note

This method calculates the false positive rate with tolerance by comparing the truth and test data. If either the truth or test data is empty, it returns NaN.

false_negative_rate_with_tolerance()

Calculate the false negative rate with tolerance.

Parameters:

  • truth – np.ndarray the truth binary segmentation

  • test – np.ndarray the test binary segmentation

Returns:

The false negative rate with tolerance as a floating-point number.

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.false_negative_rate_with_tolerance()
0.25

Note

This method calculates the false negative rate with tolerance, which is a measure of the proportion of false negatives in a binary segmentation evaluation. If either the ground truth or the test data is empty, the method returns NaN.

precision_with_tolerance()

Calculate the precision with tolerance.

This method calculates the precision with tolerance by comparing the truth and test data. Precision is the ratio of true positives to the sum of true positives and false positives. Tolerance is a distance threshold within which two pixels are considered to be a match.

Parameters:

  • truth – np.ndarray the truth binary segmentation

  • test – np.ndarray the test binary segmentation

Returns:

The precision with tolerance.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.precision_with_tolerance()
0.75

Note

  • Precision is a measure of the accuracy of the positive predictions.

  • If either the ground truth or the test data is empty, the method returns NaN.

recall_with_tolerance()

Calculate the recall with tolerance for the binary segmentation evaluator.

Returns:

The recall with tolerance value.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.recall_with_tolerance()
0.75

Note

This method calculates the recall with tolerance, which is a measure of how well the binary segmentation evaluator performs. It returns the recall with tolerance value as a float. If either the truth or test data is empty, it returns NaN.

f1_score_with_tolerance()

Calculate the F1 score with tolerance.

This method calculates the F1 score with tolerance between the ground truth and the test results. If either the ground truth or the test results are empty, the function returns NaN.

Parameters:

  • truth – np.ndarray the truth binary segmentation

  • test – np.ndarray the test binary segmentation

Returns:

The F1 score with tolerance.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.f1_score_with_tolerance()
0.85

Note

The F1 score is a measure of a test’s accuracy. It considers both the precision and recall of the test to compute the score. The tolerance parameter allows for a certain degree of variation between the ground truth and the test results.

class dacapo.experiments.tasks.evaluators.binary_segmentation_evaluator.CremiEvaluator(truth, test, sampling=(1, 1, 1), clip_distance=200, tol_distance=40)

Evaluate the performance of a binary segmentation task using the CREMI score. The CREMI score is a measure of the similarity between two binary segmentations.

truth

np.ndarray the truth binary segmentation

test

np.ndarray the test binary segmentation

sampling

Tuple[float, float, float] the sampling resolution

clip_distance

float the maximum distance to clip

tol_distance

float the tolerance distance

false_positive_distances()

Return the false positive distances.

false_positives_with_tolerance()

Return the false positives with tolerance.

false_positive_rate_with_tolerance()

Return the false positive rate with tolerance.

false_negatives_with_tolerance()

Return the false negatives with tolerance.

false_negative_rate_with_tolerance()

Return the false negative rate with tolerance.

true_positives_with_tolerance()

Return the true positives with tolerance.

precision_with_tolerance()

Return the precision with tolerance.

recall_with_tolerance()

Return the recall with tolerance.

f1_score_with_tolerance()

Return the F1 score with tolerance.

mean_false_positive_distances_clipped()

Return the mean false positive distances clipped.

mean_false_negative_distances_clipped()

Return the mean false negative distances clipped.

mean_false_positive_distance()

Return the mean false positive distance.

false_negative_distances()

Return the false negative distances.

mean_false_negative_distance()

Return the mean false negative distance.

mean_false_distance()

Return the mean false distance.

mean_false_distance_clipped()

Return the mean false distance clipped.

Note

  • The CremiEvaluator class is used to evaluate the performance of a binary segmentation task using the CREMI score.

  • True and test binary segmentations are compared to calculate various evaluation metrics.

  • The class provides methods to evaluate the performance of the binary segmentation task.

  • Toleration distance is used to determine the tolerance level for the evaluation.

  • Clip distance is used to clip the distance values to avoid extreme values.

  • All evaluation scores should inherit from this class.

test
truth
sampling
clip_distance
tol_distance
test_mask()

Generate a binary mask for the test data.

Parameters:

test – np.ndarray the test binary segmentation

Returns:

A binary mask indicating the regions of interest in the test data.

Return type:

test_mask (ndarray)

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.test = np.array([[0, 1, 0], [1, 1, 1], [0, 1, 0]])
>>> evaluator.test_mask()
array([[False,  True, False],
        [ True,  True,  True],
        [False,  True, False]])

Note

This method assumes that the background class is represented by the constant BG.

truth_mask()

Returns a binary mask indicating the truth values.

Parameters:

truth – np.ndarray the truth binary segmentation

Returns:

A binary mask where True indicates the truth values and False indicates other values.

Return type:

truth_mask (ndarray)

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> mask = evaluator.truth_mask()
>>> print(mask)
[[ True  True False]
    [False  True False]
    [ True False False]]

Note

The truth mask is computed by comparing the truth values with a predefined background value (BG).

test_edt()

Calculate the Euclidean Distance Transform (EDT) of the test mask.

Parameters:

  • self.test_mask (ndarray) – The binary test mask.

  • self.sampling (float or sequence of floats) – The pixel spacing or sampling along each dimension.

Returns:

The Euclidean Distance Transform of the test mask.

Return type:

ndarray

Examples

# Example 1: test_mask = np.array([[0, 0, 1],

[1, 1, 1], [0, 0, 0]])

sampling = 1.0 result = test_edt(test_mask, sampling) # Output: array([[1. , 1. , 0. ], # [0. , 0. , 0. ], # [1. , 1. , 1.41421356]])

# Example 2: test_mask = np.array([[0, 1, 0],

[1, 0, 1], [0, 1, 0]])

sampling = 0.5 result = test_edt(test_mask, sampling) # Output: array([[0.5 , 0. , 0.5 ], # [0. , 0.70710678, 0. ], # [0.5 , 0. , 0.5 ]])

Note

The Euclidean Distance Transform (EDT) calculates the distance from each pixel in the binary mask to the nearest boundary pixel. It is commonly used in image processing and computer vision tasks, such as edge detection and shape analysis.

truth_edt()

Calculate the Euclidean Distance Transform (EDT) of the ground truth mask.

Parameters:

  • self.truth_mask (ndarray) – The binary ground truth mask.

  • self.sampling (float or sequence of floats) – The pixel spacing or sampling along each dimension.

Returns:

The Euclidean Distance Transform of the ground truth mask.

Return type:

ndarray

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> edt = evaluator.truth_edt()

Note

The Euclidean Distance Transform (EDT) calculates the distance from each pixel in the binary mask to the nearest boundary pixel. It is commonly used in image processing and computer vision tasks.

false_positive_distances()

Calculate the distances of false positive pixels from the ground truth segmentation.

Parameters:

  • self.test_mask (ndarray) – The binary test mask.

  • self.truth_edt (ndarray) – The Euclidean Distance Transform of the ground truth segmentation.

Returns:

An array containing the distances of false positive pixels from the ground truth segmentation.

Return type:

numpy.ndarray

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> distances = evaluator.false_positive_distances()
>>> print(distances)
[1.2, 0.8, 2.5, 1.0]

Note

This method assumes that the ground truth segmentation and the test mask have been initialized. The ground truth segmentation is stored in the truth_edt attribute, and the test mask is obtained by inverting the test_mask attribute.

false_positives_with_tolerance()

Calculate the number of false positives with a given tolerance distance.

Parameters:

  • self.false_positive_distances (ndarray) – The distances of false positive pixels from the ground truth segmentation.

  • self.tol_distance (float) – The tolerance distance.

Returns:

The number of false positives with a distance greater than the tolerance distance.

Return type:

int

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.false_positive_distances = [1, 2, 3]
>>> evaluator.tol_distance = 2
>>> false_positives = evaluator.false_positives_with_tolerance()
>>> print(false_positives)
1

Note

The false_positive_distances attribute should be initialized before calling this method.

false_positive_rate_with_tolerance()

Calculate the false positive rate with tolerance.

This method calculates the false positive rate by dividing the number of false positives with tolerance by the number of condition negatives.

Parameters:

  • self.false_positives_with_tolerance (int) – The number of false positives with tolerance.

  • self.truth_mask (ndarray) – The binary ground truth mask.

Returns:

The false positive rate with tolerance.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.false_positives_with_tolerance = 10
>>> evaluator.truth_mask = np.array([0, 1, 0, 1, 0])
>>> evaluator.false_positive_rate_with_tolerance()
0.5

Note

The false positive rate with tolerance is a measure of the proportion of false positive predictions with respect to the total number of condition negatives. It is commonly used in binary segmentation tasks.

false_negatives_with_tolerance()

Calculate the number of false negatives with tolerance.

Parameters:

  • self.false_negative_distances (ndarray) – The distances of false negative pixels from the ground truth segmentation.

  • self.tol_distance (float) – The tolerance distance.

Returns:

The number of false negatives with tolerance.

Return type:

int

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.false_negative_distances = [1, 2, 3]
>>> evaluator.tol_distance = 2
>>> false_negatives = evaluator.false_negatives_with_tolerance()
>>> print(false_negatives)
1

Note

False negatives are cases where the model incorrectly predicts the absence of a positive class. The tolerance distance is used to determine whether a false negative is within an acceptable range.

false_negative_rate_with_tolerance()

Calculate the false negative rate with tolerance.

This method calculates the false negative rate by dividing the number of false negatives with tolerance by the number of condition positives.

Parameters:

  • self.false_negatives_with_tolerance (int) – The number of false negatives with tolerance.

  • self.false_negative_distances (ndarray) – The distances of false negative pixels from the ground truth segmentation.

Returns:

The false negative rate with tolerance.

Return type:

float

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.false_negative_distances = [1, 2, 3]
>>> evaluator.false_negatives_with_tolerance = 2
>>> evaluator.false_negative_rate_with_tolerance()
0.6666666666666666

Note

The false negative rate with tolerance is a measure of the proportion of condition positives that are incorrectly classified as negatives, considering a certain tolerance level.

true_positives_with_tolerance()

Calculate the number of true positives with tolerance.

Parameters:

  • self.test_mask (ndarray) – The test binary segmentation mask.

  • self.truth_mask (ndarray) – The ground truth binary segmentation mask.

  • self.false_negatives_with_tolerance (int) – The number of false negatives with tolerance.

  • self.false_positives_with_tolerance (int) – The number of false positives with tolerance.

Returns:

The number of true positives with tolerance.

Return type:

int

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.test_mask = np.array([[0, 1], [1, 0]])
>>> evaluator.truth_mask = np.array([[0, 1], [1, 0]])
>>> evaluator.false_negatives_with_tolerance = 1
>>> evaluator.false_positives_with_tolerance = 1
>>> true_positives = evaluator.true_positives_with_tolerance()
>>> print(true_positives)
2

Note

True positives are cases where the model correctly predicts the presence of a positive class. The tolerance distance is used to determine whether a true positive is within an acceptable range.

precision_with_tolerance()

Calculate the precision with tolerance.

This method calculates the precision with tolerance by dividing the number of true positives with tolerance by the sum of true positives with tolerance and false positives with tolerance.

Parameters:

  • self.true_positives_with_tolerance (int) – The number of true positives with tolerance.

  • self.false_positives_with_tolerance (int) – The number of false positives with tolerance.

Returns:

The precision with tolerance.

Return type:

float

Raises:

ZeroDivisionError – If the sum of true positives with tolerance and false positives with tolerance is zero.

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.true_positives_with_tolerance = 10
>>> evaluator.false_positives_with_tolerance = 5
>>> evaluator.precision_with_tolerance()
0.6666666666666666

Note

The precision with tolerance is a measure of the proportion of true positives with tolerance out of the total number of predicted positives with tolerance. It indicates how well the binary segmentation evaluator performs in terms of correctly identifying positive samples. If the sum of true positives with tolerance and false positives with tolerance is zero, the precision with tolerance is undefined and a ZeroDivisionError is raised.

recall_with_tolerance()

A measure of the ability of a binary classifier to identify all positive samples.

Parameters:

  • self.true_positives_with_tolerance (int) – The number of true positives with tolerance.

  • self.false_negatives_with_tolerance (int) – The number of false negatives with tolerance.

Returns:

The recall with tolerance value.

Return type:

float

Raises:

ZeroDivisionError – If the sum of true positives with tolerance and false negatives with tolerance is zero.

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.recall_with_tolerance()
0.75

Note

This method calculates the recall with tolerance, which is a measure of how well the binary segmentation evaluator performs. It returns the recall with tolerance value as a float. If either the truth or test data is empty, it returns NaN.

f1_score_with_tolerance()

Calculate the F1 score with tolerance.

Parameters:

  • self.recall_with_tolerance (float) – The recall with tolerance value.

  • self.precision_with_tolerance (float) – The precision with tolerance value.

Returns:

The F1 score with tolerance.

Return type:

float

Raises:

ZeroDivisionError – If both the recall with tolerance and precision with tolerance are zero.

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.recall_with_tolerance = 0.8
>>> evaluator.precision_with_tolerance = 0.9
>>> evaluator.f1_score_with_tolerance()
0.8571428571428571

Note

The F1 score is a measure of a test’s accuracy. It considers both the precision and recall of the test to compute the score. The F1 score with tolerance is calculated using the formula: F1 = 2 * (recall_with_tolerance * precision_with_tolerance) / (recall_with_tolerance + precision_with_tolerance) If both recall_with_tolerance and precision_with_tolerance are 0, the F1 score with tolerance will be NaN.

mean_false_positive_distances_clipped()

Calculate the mean of the false positive distances, clipped to a maximum distance.

Parameters:

  • self.false_positive_distances (ndarray) – The distances of false positive pixels from the ground truth segmentation.

  • self.clip_distance (float) – The maximum distance to clip.

Returns:

The mean of the false positive distances, clipped to a maximum distance.

Return type:

float

Raises:

ValueError – If the clip distance is not set.

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.false_positive_distances = [1, 2, 3, 4, 5]
>>> evaluator.clip_distance = 3
>>> evaluator.mean_false_positive_distances_clipped()
2.5

Note

This method calculates the mean of the false positive distances, where the distances are clipped to a maximum distance. The false_positive_distances attribute should be set before calling this method. The clip_distance attribute determines the maximum distance to which the distances are clipped.

mean_false_negative_distances_clipped()

Calculate the mean of the false negative distances, clipped to a maximum distance.

Parameters:

  • self.false_negative_distances (ndarray) – The distances of false negative pixels from the ground truth segmentation.

  • self.clip_distance (float) – The maximum distance to clip.

Returns:

The mean of the false negative distances, clipped to a maximum distance.

Return type:

float

Raises:

ValueError – If the clip distance is not set.

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.false_negative_distances = [1, 2, 3, 4, 5]
>>> evaluator.clip_distance = 3
>>> evaluator.mean_false_negative_distances_clipped()
2.5

Note

This method calculates the mean of the false negative distances, where the distances are clipped to a maximum distance. The false_negative_distances attribute should be set before calling this method. The clip_distance attribute determines the maximum distance to which the distances are clipped.

mean_false_positive_distance()

Calculate the mean false positive distance.

This method calculates the mean distance between the false positive points and the ground truth points.

Parameters:

self.false_positive_distances (ndarray) – The distances of false positive pixels from the ground truth mask.

Returns:

The mean false positive distance.

Return type:

float

Raises:

ValueError – If the false positive distances are not set.

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.false_positive_distances = [1.2, 3.4, 2.1]
>>> evaluator.mean_false_positive_distance()
2.2333333333333334

Note

The false positive distances should be set before calling this method using the false_positive_distances attribute.

false_negative_distances()

Calculate the distances of false negative pixels from the ground truth mask.

Parameters:

self.truth_mask (ndarray) – The binary ground truth mask.

Returns:

An array containing the distances of false negative pixels.

Return type:

numpy.ndarray

Raises:

ValueError – If the truth mask is not set.

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> distances = evaluator.false_negative_distances()
>>> print(distances)
[0.5, 1.0, 1.5, 2.0]

Note

This method assumes that the ground truth mask and the test mask have already been set.

mean_false_negative_distance()

Calculate the mean false negative distance.

Parameters:

self.false_negative_distances (ndarray) – The distances of false negative pixels from the ground truth mask.

Returns:

The mean false negative distance.

Return type:

float

Raises:

ValueError – If the false negative distances are not set.

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.false_negative_distances = [1.2, 3.4, 2.1]
>>> evaluator.mean_false_negative_distance()
2.2333333333333334

Note

The mean false negative distance is calculated as the average of all false negative distances.

mean_false_distance()

Calculate the mean false distance.

This method calculates the mean false distance by taking the average of the mean false positive distance and the mean false negative distance.

Parameters:

  • self.mean_false_positive_distance (float) – The mean false positive distance.

  • self.mean_false_negative_distance (float) – The mean false negative distance.

Returns:

The calculated mean false distance.

Return type:

float

Raises:

ValueError – If the mean false positive distance or the mean false negative distance is not set.

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.mean_false_distance()
5.0

Note

The mean false distance is a metric used to evaluate the performance of a binary segmentation model. It provides a measure of the average distance between false positive and false negative predictions.

mean_false_distance_clipped()

Calculates the mean false distance clipped.

This method calculates the mean false distance clipped by taking the average of the mean false positive distances clipped and the mean false negative distances clipped.

Parameters:

  • self.mean_false_positive_distances_clipped (float) – The mean false positive distances clipped.

  • self.mean_false_negative_distances_clipped (float) – The mean false negative distances clipped.

Returns:

The calculated mean false distance clipped.

Return type:

float

Raises:

ValueError – If the mean false positive distances clipped or the mean false negative distances clipped are not set.

Examples

>>> evaluator = BinarySegmentationEvaluator()
>>> evaluator.mean_false_distance_clipped()
2.5

Note

The mean false distance clipped is calculated as 0.5 * (mean_false_positive_distances_clipped + mean_false_negative_distances_clipped).