dacapo.experiments.validation_scores

Classes

ValidationScores

Class representing the validation scores for a set of parameters and datasets.

Module Contents

class dacapo.experiments.validation_scores.ValidationScores

Class representing the validation scores for a set of parameters and datasets.

parameters

The list of parameters that are being evaluated.

Type:

List[PostProcessorParameters]

datasets

The datasets that will be evaluated at each iteration.

Type:

List[Dataset]

evaluation_scores

The scores that are collected on each iteration per PostProcessorParameters and Dataset.

Type:

EvaluationScores

scores

A list of evaluation scores and their associated post-processing parameters.

Type:

List[ValidationIterationScores]

subscores(iteration_scores)

Create a new ValidationScores object with a subset of the iteration scores.

add_iteration_scores(iteration_scores)

Add iteration scores to the list of scores.

delete_after(iteration)

Delete scores after a specified iteration.

validated_until()

Get the number of iterations validated for (the maximum iteration plus one).

compare(existing_iteration_scores)

Compare iteration stats provided from elsewhere to scores we have saved locally.

criteria()

Get the list of evaluation criteria.

parameter_names()

Get the list of parameter names.

to_xarray()

Convert the validation scores to an xarray DataArray.

get_best(data, dim)

Compute the Best scores along dimension “dim” per criterion.

Notes

The scores attribute is a list of ValidationIterationScores objects, each of which contains the scores for a single iteration.

parameters: List[dacapo.experiments.tasks.post_processors.PostProcessorParameters]

datasets: List[dacapo.experiments.datasplits.datasets.Dataset]

evaluation_scores: dacapo.experiments.tasks.evaluators.EvaluationScores

scores: List[dacapo.experiments.validation_iteration_scores.ValidationIterationScores]

subscores(iteration_scores: List[dacapo.experiments.validation_iteration_scores.ValidationIterationScores]) → ValidationScores

Create a new ValidationScores object with a subset of the iteration scores.

Parameters:

iteration_scores – The iteration scores to include in the new ValidationScores object.

Returns:

A new ValidationScores object with the specified iteration scores.

Raises:

ValueError – If the iteration scores are not in the list of scores.

Examples

>>> validation_scores.subscores([validation_scores.scores[0]])

Note

This method is used to create a new ValidationScores object with a subset of the iteration scores. This is useful when you want to create a new ValidationScores object that only contains the scores up to a certain iteration.

add_iteration_scores(iteration_scores: dacapo.experiments.validation_iteration_scores.ValidationIterationScores) → None

Add iteration scores to the list of scores.

Parameters:

iteration_scores – The iteration scores to add.

Raises:

ValueError – If the iteration scores are already in the list of scores.

Examples

>>> validation_scores.add_iteration_scores(validation_scores.scores[0])

Note

This method is used to add iteration scores to the list of scores. This is useful when you want to add scores for a new iteration to the ValidationScores object.

delete_after(iteration: int) → None

Delete scores after a specified iteration.

Parameters:

iteration – The iteration after which to delete the scores.

Raises:

ValueError – If the iteration scores are not in the list of scores.

Examples

>>> validation_scores.delete_after(0)

Note

This method is used to delete scores after a specified iteration. This is useful when you want to delete scores after a certain iteration.

validated_until() → int

Get the number of iterations validated for (the maximum iteration plus one).

Returns:

The number of iterations validated for.

Raises:

ValueError – If there are no scores.

Examples

>>> validation_scores.validated_until()

Note

This method is used to get the number of iterations validated for (the maximum iteration plus one). This is useful when you want to know how many iterations have been validated.

compare(existing_iteration_scores: List[dacapo.experiments.validation_iteration_scores.ValidationIterationScores]) → Tuple[bool, int]

Compare iteration stats provided from elsewhere to scores we have saved locally. Local scores take priority. If local scores are at a lower iteration than the existing ones, delete the existing ones and replace with local. If local iteration > existing iteration, just update existing scores with the last overhanging local scores.

Parameters:

existing_iteration_scores – The existing iteration scores to compare with.

Returns:

A tuple indicating whether the local scores should replace the existing ones and the existing iteration number.

Raises:

ValueError – If the iteration scores are not in the list of scores.

Examples

>>> validation_scores.compare([validation_scores.scores[0]])

Note

This method is used to compare iteration stats provided from elsewhere to scores we have saved locally. Local scores take priority. If local scores are at a lower iteration than the existing ones, delete the existing ones and replace with local. If local iteration > existing iteration, just update existing scores with the last overhanging local scores.

property criteria: List[str]

Get the list of evaluation criteria.

Returns:

The list of evaluation criteria.

Raises:

ValueError – If there are no scores.

Examples

>>> validation_scores.criteria

Note

This property is used to get the list of evaluation criteria. This is useful when you want to know what criteria are being used to evaluate the scores.

property parameter_names: List[str]

Get the list of parameter names.

Returns:

The list of parameter names.

Raises:

ValueError – If there are no scores.

Examples

>>> validation_scores.parameter_names

Note

This property is used to get the list of parameter names. This is useful when you want to know what parameters are being used to evaluate the scores.

to_xarray() → xarray.DataArray

Convert the validation scores to an xarray DataArray.

Returns:

An xarray DataArray representing the validation scores.

Raises:

ValueError – If there are no scores.

Examples

>>> validation_scores.to_xarray()

Note

This method is used to convert the validation scores to an xarray DataArray. This is useful when you want to work with the validation scores as an xarray DataArray.

get_best(data: xarray.DataArray, dim: str) → Tuple[xarray.DataArray, xarray.DataArray]

Compute the Best scores along dimension “dim” per criterion. Returns both the index associated with the best value, and the best value in two separate arrays.

Parameters:

• data – The data array to compute the best scores from.

• dim – The dimension along which to compute the best scores.

Returns:

A tuple containing the index associated with the best value and the best value in two separate arrays.

Raises:

ValueError – If the criteria are not in the data array.

Examples

>>> validation_scores.get_best(data, "iterations")

Note

This method is used to compute the Best scores along dimension “dim” per criterion. It returns both the index associated with the best value and the best value in two separate arrays. This is useful when you want to know the best scores for a given data array. Fix: The method is currently not able to handle the case where the criteria are not in the data array. To fix this, we need to add a check to see if the criteria are in the data array and raise an error if they are not.