dacapo.experiments.tasks.evaluators

Submodules

• dacapo.experiments.tasks.evaluators.binary_segmentation_evaluation_scores
• dacapo.experiments.tasks.evaluators.binary_segmentation_evaluator
• dacapo.experiments.tasks.evaluators.dummy_evaluation_scores
• dacapo.experiments.tasks.evaluators.dummy_evaluator
• dacapo.experiments.tasks.evaluators.evaluation_scores
• dacapo.experiments.tasks.evaluators.evaluator
• dacapo.experiments.tasks.evaluators.instance_evaluation_scores
• dacapo.experiments.tasks.evaluators.instance_evaluator

Package Contents

Classes

DummyEvaluationScores	Base class for evaluation scores.
DummyEvaluator	Base class of all evaluators.
EvaluationScores	Base class for evaluation scores.
Evaluator	Base class of all evaluators.
MultiChannelBinarySegmentationEvaluationScores	Base class for evaluation scores.
BinarySegmentationEvaluationScores	Base class for evaluation scores.
BinarySegmentationEvaluator	Given a binary segmentation, compute various metrics to determine their similarity.
InstanceEvaluationScores	Base class for evaluation scores.
InstanceEvaluator	Base class of all evaluators.

class dacapo.experiments.tasks.evaluators.DummyEvaluationScores

Base class for evaluation scores.

criteria = ['frizz_level', 'blipp_score']

frizz_level: float

blipp_score: float

static higher_is_better(criterion: str) → bool

Wether or not higher is better for this criterion.

static bounds(criterion: str) → Tuple[int | float | None, int | float | None]

The bounds for this criterion

static store_best(criterion: str) → bool

Whether or not to save the best validation block and model weights for this criterion.

class dacapo.experiments.tasks.evaluators.DummyEvaluator

Base class of all evaluators.

An evaluator takes a post-processor’s output and compares it against ground-truth.

property score: dacapo.experiments.tasks.evaluators.dummy_evaluation_scores.DummyEvaluationScores

criteria = ['frizz_level', 'blipp_score']

evaluate(output_array_identifier, evaluation_dataset)

Evaluate the given output array and dataset and returns the scores based on predefined criteria.

Parameters:

• output_array_identifier – The output array to be evaluated.

• evaluation_dataset – The dataset to be used for evaluation.

Returns:

An object of DummyEvaluationScores class, with the evaluation scores.

Return type:

DummyEvaluationScore

class dacapo.experiments.tasks.evaluators.EvaluationScores

Base class for evaluation scores.

abstract property criteria: List[str]

abstract static higher_is_better(criterion: str) → bool

Wether or not higher is better for this criterion.

abstract static bounds(criterion: str) → Tuple[int | float | None, int | float | None]

The bounds for this criterion

abstract static store_best(criterion: str) → bool

Whether or not to save the best validation block and model weights for this criterion.

class dacapo.experiments.tasks.evaluators.Evaluator

Base class of all evaluators.

An evaluator takes a post-processor’s output and compares it against ground-truth.

property best_scores: Dict[OutputIdentifier, BestScore]

abstract property criteria: List[str]

A list of all criteria for which a model might be “best”. i.e. your criteria might be “precision”, “recall”, and “jaccard”. It is unlikely that the best iteration/post processing parameters will be the same for all 3 of these criteria

abstract property score: dacapo.experiments.tasks.evaluators.evaluation_scores.EvaluationScores

abstract evaluate(output_array_identifier: dacapo.store.local_array_store.LocalArrayIdentifier, evaluation_array: dacapo.experiments.datasplits.datasets.arrays.Array) → dacapo.experiments.tasks.evaluators.evaluation_scores.EvaluationScores

Compares and evaluates the output array against the evaluation array.

Parameters:

• output_array_identifier (Array) – The output data array to evaluate

• evaluation_array (Array) – The evaluation data array to compare with the output

Returns:

The detailed evaluation scores after the comparison.

Return type:

EvaluationScores

is_best(dataset: dacapo.experiments.datasplits.datasets.Dataset, parameter: dacapo.experiments.tasks.post_processors.PostProcessorParameters, criterion: str, score: dacapo.experiments.tasks.evaluators.evaluation_scores.EvaluationScores) → bool

Check if the provided score is the best for this dataset/parameter/criterion combo

get_overall_best(dataset: dacapo.experiments.datasplits.datasets.Dataset, criterion: str)

get_overall_best_parameters(dataset: dacapo.experiments.datasplits.datasets.Dataset, criterion: str)

compare(score_1, score_2, criterion)

set_best(validation_scores: dacapo.experiments.validation_scores.ValidationScores) → None

Find the best iteration for each dataset/post_processing_parameter/criterion

higher_is_better(criterion: str) → bool

Wether or not higher is better for this criterion.

bounds(criterion: str) → Tuple[int | float | None, int | float | None]

The bounds for this criterion

store_best(criterion: str) → bool

The bounds for this criterion

class dacapo.experiments.tasks.evaluators.MultiChannelBinarySegmentationEvaluationScores

Base class for evaluation scores.

property criteria

channel_scores: List[Tuple[str, BinarySegmentationEvaluationScores]]

static higher_is_better(criterion: str) → bool

Wether or not higher is better for this criterion.

static store_best(criterion: str) → bool

Whether or not to save the best validation block and model weights for this criterion.

static bounds(criterion: str) → Tuple[int | float | None, int | float | None]

The bounds for this criterion

class dacapo.experiments.tasks.evaluators.BinarySegmentationEvaluationScores

Base class for evaluation scores.

dice: float

jaccard: float

hausdorff: float

false_negative_rate: float

false_negative_rate_with_tolerance: float

false_positive_rate: float

false_discovery_rate: float

false_positive_rate_with_tolerance: float

voi: float

mean_false_distance: float

mean_false_negative_distance: float

mean_false_positive_distance: float

mean_false_distance_clipped: float

mean_false_negative_distance_clipped: float

mean_false_positive_distance_clipped: float

precision_with_tolerance: float

recall_with_tolerance: float

f1_score_with_tolerance: float

precision: float

recall: float

f1_score: float

criteria = ['dice', 'jaccard', 'hausdorff', 'false_negative_rate', 'false_negative_rate_with_tolerance',...

static store_best(criterion: str) → bool

Whether or not to save the best validation block and model weights for this criterion.

static higher_is_better(criterion: str) → bool

Wether or not higher is better for this criterion.

static bounds(criterion: str) → Tuple[int | float | None, int | float | None]

The bounds for this criterion

class dacapo.experiments.tasks.evaluators.BinarySegmentationEvaluator(clip_distance: float, tol_distance: float, channels: List[str])

Given a binary segmentation, compute various metrics to determine their similarity.

property score

criteria = ['jaccard', 'voi']

evaluate(output_array_identifier, evaluation_array)

Compares and evaluates the output array against the evaluation array.

Parameters:

• output_array_identifier (Array) – The output data array to evaluate

• evaluation_array (Array) – The evaluation data array to compare with the output

Returns:

The detailed evaluation scores after the comparison.

Return type:

EvaluationScores

class dacapo.experiments.tasks.evaluators.InstanceEvaluationScores

Base class for evaluation scores.

property voi

criteria = ['voi_split', 'voi_merge', 'voi']

voi_split: float

voi_merge: float

static higher_is_better(criterion: str) → bool

Wether or not higher is better for this criterion.

static bounds(criterion: str) → Tuple[int | float | None, int | float | None]

The bounds for this criterion

static store_best(criterion: str) → bool

Whether or not to save the best validation block and model weights for this criterion.

class dacapo.experiments.tasks.evaluators.InstanceEvaluator

Base class of all evaluators.

An evaluator takes a post-processor’s output and compares it against ground-truth.

property score: dacapo.experiments.tasks.evaluators.instance_evaluation_scores.InstanceEvaluationScores

criteria: List[str] = ['voi_merge', 'voi_split', 'voi']

evaluate(output_array_identifier, evaluation_array)

Compares and evaluates the output array against the evaluation array.

Parameters:

• output_array_identifier (Array) – The output data array to evaluate

• evaluation_array (Array) – The evaluation data array to compare with the output

Returns:

The detailed evaluation scores after the comparison.

Return type:

EvaluationScores