`dacapo.experiments.tasks.evaluators.evaluator`

Module Contents

Classes

Base class of all evaluators.

Attributes

`OutputIdentifier`
`Iteration`
`Score`
`BestScore`

dacapo.experiments.tasks.evaluators.evaluator.OutputIdentifier

dacapo.experiments.tasks.evaluators.evaluator.Iteration

dacapo.experiments.tasks.evaluators.evaluator.Score

dacapo.experiments.tasks.evaluators.evaluator.BestScore

class dacapo.experiments.tasks.evaluators.evaluator.Evaluator

Base class of all evaluators.

An evaluator takes a post-processor’s output and compares it against ground-truth.

property best_scores: Dict[OutputIdentifier, BestScore]

abstract property criteria: List[str]: A list of all criteria for which a model might be “best”. i.e. your criteria might be “precision”, “recall”, and “jaccard”. It is unlikely that the best iteration/post processing parameters will be the same for all 3 of these criteria

abstract property score: dacapo.experiments.tasks.evaluators.evaluation_scores.EvaluationScores

abstract evaluate(output_array_identifier: dacapo.store.local_array_store.LocalArrayIdentifier, evaluation_array: dacapo.experiments.datasplits.datasets.arrays.Array) → dacapo.experiments.tasks.evaluators.evaluation_scores.EvaluationScores

Compares and evaluates the output array against the evaluation array.

Parameters:

output_array_identifier (Array) – The output data array to evaluate
evaluation_array (Array) – The evaluation data array to compare with the output

Returns:

The detailed evaluation scores after the comparison.

Return type:

EvaluationScores

is_best(dataset: dacapo.experiments.datasplits.datasets.Dataset, parameter: dacapo.experiments.tasks.post_processors.PostProcessorParameters, criterion: str, score: dacapo.experiments.tasks.evaluators.evaluation_scores.EvaluationScores) → bool: Check if the provided score is the best for this dataset/parameter/criterion combo

get_overall_best(dataset: dacapo.experiments.datasplits.datasets.Dataset, criterion: str)

get_overall_best_parameters(dataset: dacapo.experiments.datasplits.datasets.Dataset, criterion: str)

compare(score_1, score_2, criterion)

set_best(validation_scores: dacapo.experiments.validation_scores.ValidationScores) → None: Find the best iteration for each dataset/post_processing_parameter/criterion

higher_is_better(criterion: str) → bool: Wether or not higher is better for this criterion.

bounds(criterion: str) → Tuple[int | float | None, int | float | None]: The bounds for this criterion

store_best(criterion: str) → bool: The bounds for this criterion