Saving optimization metadata when saving optimization history

Users Q&A
1

As I understand it, the most straightforward way to save the results from an executed scenario is to save it as an HDF5 like so: my_scenario.save_optimization_history() or saving the dataset directly with my_scenario.to_dataset().to_hdf()

When post-processing results from a run I want to be able to use all the same post-processing functions as if I were to do the post-processing immediately after execution of the scenario with the scenario object still in memory, including being able to filter data points by whether constraints were met using the mask in my_scenario.to_dataset().misc['optimization_metadata'].feasible_iterations

But as far as I can tell, the contents of the misc field is not included when saving the optimization history, because when loading the hdf5 file for post-processing with import_database all the items in misc are empty.

Is there some way to save and load the optimization metadata as well?

2

Hello Axel!

The scenario.save_optimization_history() method saves an OptimizationProblem to an HDF5 file. The OptimizationProblem class includes constraints, objective, design space, the database… but it does not include the optimization metadata, that is only stored in the OptimizationDataset that you get when you export the OptimizationProblem using the method my_scenario.to_dataset().

One way you can achieve your goal is to export the scenario to an HDF5 and then re-exporting it to a dataset:

scenario.to_hdf("scenario_hdf.h5")
rebuilt_problem = OptimizationProblem.from_hdf("scenario_hdf.h5")
dataset_from_problem = rebuilt_problem.to_dataset()
print(dataset_from_problem.misc['optimization_metadata'].feasible_iterations)

Keep in mind that import_database will only instantiate the Database of the OptimizationProblem, so this fails:

scenario.to_hdf("scenario_hdf.h5")
rebuilt_database = import_database("scenario_hdf.h5")
dataset_from_database = rebuilt_database.to_dataset()
print(dataset_from_problem.misc['optimization_metadata'].feasible_iterations)

If you don’t care too much about the OptimizationProblem instance and just want to post-process the dataset directly, you can use to_pickle and from_pickle to serialize the Dataset object directly:

dataset = scenario.to_dataset()
to_pickle(dataset, "some_dataset.pkl")
rebuilt_dataset = from_pickle("some_dataset.pkl")
print(dataset.misc['optimization_metadata'].feasible_iterations)

I hope this helps!