dacapo.experiments.validation_scores ==================================== .. py:module:: dacapo.experiments.validation_scores Classes ------- .. autoapisummary:: dacapo.experiments.validation_scores.ValidationScores Module Contents --------------- .. py:class:: ValidationScores Class representing the validation scores for a set of parameters and datasets. .. attribute:: parameters The list of parameters that are being evaluated. :type: List[PostProcessorParameters] .. attribute:: datasets The datasets that will be evaluated at each iteration. :type: List[Dataset] .. attribute:: evaluation_scores The scores that are collected on each iteration per `PostProcessorParameters` and `Dataset`. :type: EvaluationScores .. attribute:: scores A list of evaluation scores and their associated post-processing parameters. :type: List[ValidationIterationScores] .. method:: subscores(iteration_scores) Create a new ValidationScores object with a subset of the iteration scores. .. method:: add_iteration_scores(iteration_scores) Add iteration scores to the list of scores. .. method:: delete_after(iteration) Delete scores after a specified iteration. .. method:: validated_until() Get the number of iterations validated for (the maximum iteration plus one). .. method:: compare(existing_iteration_scores) Compare iteration stats provided from elsewhere to scores we have saved locally. .. method:: criteria() Get the list of evaluation criteria. .. method:: parameter_names() Get the list of parameter names. .. method:: to_xarray() Convert the validation scores to an xarray DataArray. .. method:: get_best(data, dim) Compute the Best scores along dimension "dim" per criterion. .. rubric:: Notes The `scores` attribute is a list of `ValidationIterationScores` objects, each of which contains the scores for a single iteration. .. py:attribute:: parameters :type: List[dacapo.experiments.tasks.post_processors.PostProcessorParameters] .. py:attribute:: datasets :type: List[dacapo.experiments.datasplits.datasets.Dataset] .. py:attribute:: evaluation_scores :type: dacapo.experiments.tasks.evaluators.EvaluationScores .. py:attribute:: scores :type: List[dacapo.experiments.validation_iteration_scores.ValidationIterationScores] .. py:method:: subscores(iteration_scores: List[dacapo.experiments.validation_iteration_scores.ValidationIterationScores]) -> ValidationScores Create a new ValidationScores object with a subset of the iteration scores. :param iteration_scores: The iteration scores to include in the new ValidationScores object. :returns: A new ValidationScores object with the specified iteration scores. :raises ValueError: If the iteration scores are not in the list of scores. .. rubric:: Examples >>> validation_scores.subscores([validation_scores.scores[0]]) .. note:: This method is used to create a new ValidationScores object with a subset of the iteration scores. This is useful when you want to create a new ValidationScores object that only contains the scores up to a certain iteration. .. py:method:: add_iteration_scores(iteration_scores: dacapo.experiments.validation_iteration_scores.ValidationIterationScores) -> None Add iteration scores to the list of scores. :param iteration_scores: The iteration scores to add. :raises ValueError: If the iteration scores are already in the list of scores. .. rubric:: Examples >>> validation_scores.add_iteration_scores(validation_scores.scores[0]) .. note:: This method is used to add iteration scores to the list of scores. This is useful when you want to add scores for a new iteration to the ValidationScores object. .. py:method:: delete_after(iteration: int) -> None Delete scores after a specified iteration. :param iteration: The iteration after which to delete the scores. :raises ValueError: If the iteration scores are not in the list of scores. .. rubric:: Examples >>> validation_scores.delete_after(0) .. note:: This method is used to delete scores after a specified iteration. This is useful when you want to delete scores after a certain iteration. .. py:method:: validated_until() -> int Get the number of iterations validated for (the maximum iteration plus one). :returns: The number of iterations validated for. :raises ValueError: If there are no scores. .. rubric:: Examples >>> validation_scores.validated_until() .. note:: This method is used to get the number of iterations validated for (the maximum iteration plus one). This is useful when you want to know how many iterations have been validated. .. py:method:: compare(existing_iteration_scores: List[dacapo.experiments.validation_iteration_scores.ValidationIterationScores]) -> Tuple[bool, int] Compare iteration stats provided from elsewhere to scores we have saved locally. Local scores take priority. If local scores are at a lower iteration than the existing ones, delete the existing ones and replace with local. If local iteration > existing iteration, just update existing scores with the last overhanging local scores. :param existing_iteration_scores: The existing iteration scores to compare with. :returns: A tuple indicating whether the local scores should replace the existing ones and the existing iteration number. :raises ValueError: If the iteration scores are not in the list of scores. .. rubric:: Examples >>> validation_scores.compare([validation_scores.scores[0]]) .. note:: This method is used to compare iteration stats provided from elsewhere to scores we have saved locally. Local scores take priority. If local scores are at a lower iteration than the existing ones, delete the existing ones and replace with local. If local iteration > existing iteration, just update existing scores with the last overhanging local scores. .. py:property:: criteria :type: List[str] Get the list of evaluation criteria. :returns: The list of evaluation criteria. :raises ValueError: If there are no scores. .. rubric:: Examples >>> validation_scores.criteria .. note:: This property is used to get the list of evaluation criteria. This is useful when you want to know what criteria are being used to evaluate the scores. .. py:property:: parameter_names :type: List[str] Get the list of parameter names. :returns: The list of parameter names. :raises ValueError: If there are no scores. .. rubric:: Examples >>> validation_scores.parameter_names .. note:: This property is used to get the list of parameter names. This is useful when you want to know what parameters are being used to evaluate the scores. .. py:method:: to_xarray() -> xarray.DataArray Convert the validation scores to an xarray DataArray. :returns: An xarray DataArray representing the validation scores. :raises ValueError: If there are no scores. .. rubric:: Examples >>> validation_scores.to_xarray() .. note:: This method is used to convert the validation scores to an xarray DataArray. This is useful when you want to work with the validation scores as an xarray DataArray. .. py:method:: get_best(data: xarray.DataArray, dim: str) -> Tuple[xarray.DataArray, xarray.DataArray] Compute the Best scores along dimension "dim" per criterion. Returns both the index associated with the best value, and the best value in two separate arrays. :param data: The data array to compute the best scores from. :param dim: The dimension along which to compute the best scores. :returns: A tuple containing the index associated with the best value and the best value in two separate arrays. :raises ValueError: If the criteria are not in the data array. .. rubric:: Examples >>> validation_scores.get_best(data, "iterations") .. note:: This method is used to compute the Best scores along dimension "dim" per criterion. It returns both the index associated with the best value and the best value in two separate arrays. This is useful when you want to know the best scores for a given data array. Fix: The method is currently not able to handle the case where the criteria are not in the data array. To fix this, we need to add a check to see if the criteria are in the data array and raise an error if they are not.