dacapo.experiments.tasks.evaluators.evaluation_scores

Classes

EvaluationScores

Base class for evaluation scores. This class is used to store the evaluation scores for a task.

Module Contents

class dacapo.experiments.tasks.evaluators.evaluation_scores.EvaluationScores

Base class for evaluation scores. This class is used to store the evaluation scores for a task. The scores include the evaluation criteria. The class also provides methods to determine whether higher is better for a given criterion, the bounds for a given criterion, and whether to store the best score for a given criterion.

criteria: List[str] the evaluation criteria

higher_is_better(criterion): Return whether higher is better for the given criterion.

bounds(criterion): Return the bounds for the given criterion.

store_best(criterion): Return whether to store the best score for the given criterion.

Note

The EvaluationScores class is used to store the evaluation scores for a task. All evaluation scores should inherit from this class.

property criteria: List[str]

Abstractmethod:

The evaluation criteria.

Returns:

List[str]: the evaluation criteria

Raises:

NotImplementedError – if the function is not implemented

Examples

>>> evaluation_scores = EvaluationScores()
>>> evaluation_scores.criteria
["criterion1", "criterion2"]

Note

This function is used to return the evaluation criteria.

static higher_is_better(criterion: str) → bool

Abstractmethod:

Wether or not higher is better for this criterion.

Parameters:

criterion – str the evaluation criterion

Returns:

bool: whether higher is better for this criterion

Raises:

NotImplementedError – if the function is not implemented

Examples

>>> evaluation_scores = EvaluationScores()
>>> criterion = "criterion1"
>>> evaluation_scores.higher_is_better(criterion)
True

Note

This function is used to determine whether higher is better for a given criterion.

static bounds(criterion: str) → Tuple[int | float | None, int | float | None]

Abstractmethod:

The bounds for this criterion.

Parameters:

criterion – str the evaluation criterion

Returns:

Tuple[Union[int, float, None], Union[int, float, None]]: the bounds for this criterion

Raises:

NotImplementedError – if the function is not implemented

Examples

>>> evaluation_scores = EvaluationScores()
>>> criterion = "criterion1"
>>> evaluation_scores.bounds(criterion)
(0, 1)

Note

This function is used to return the bounds for the given criterion.

static store_best(criterion: str) → bool

Abstractmethod:

Whether or not to save the best validation block and model weights for this criterion.

Parameters:

criterion – str the evaluation criterion

Returns:

bool: whether to store the best score for this criterion

Raises:

NotImplementedError – if the function is not implemented

Examples

>>> evaluation_scores = EvaluationScores()
>>> criterion = "criterion1"
>>> evaluation_scores.store_best(criterion)
True

Note

This function is used to return whether to store the best score for the given criterion.