View source on GitHub
|
Callback to back up and restore the training state.
Inherits From: Callback
tf.keras.callbacks.BackupAndRestore(
backup_dir, save_freq='epoch', delete_checkpoint=True
)
| Used in the guide | Used in the tutorials |
|---|---|
BackupAndRestore callback is intended to recover training from an
interruption that has happened in the middle of a Model.fit execution, by
backing up the training states in a temporary checkpoint file, at the end of
each epoch. Each backup overwrites the previously written checkpoint file,
so at any given time there is at most one such checkpoint file for
backup/restoring purpose.
If training restarts before completion, the training state (which includes
the Model weights and epoch number) is restored to the most recently saved
state at the beginning of a new Model.fit run. At the completion of a
Model.fit run, the temporary checkpoint file is deleted.
Note that the user is responsible to bring jobs back after the interruption. This callback is important for the backup and restore mechanism for fault tolerance purpose, and the model to be restored from a previous checkpoint is expected to be the same as the one used to back up. If user changes arguments passed to compile or fit, the checkpoint saved for fault tolerance can become invalid.
class InterruptingCallback(keras.callbacks.Callback):def on_epoch_begin(self, epoch, logs=None):if epoch == 4:raise RuntimeError('Interrupting!')callback = keras.callbacks.BackupAndRestore(backup_dir="/tmp/backup")model = keras.models.Sequential([keras.layers.Dense(10)])model.compile(keras.optimizers.SGD(), loss='mse')try:model.fit(np.arange(100).reshape(5, 20), np.zeros(5), epochs=10,batch_size=1, callbacks=[callback, InterruptingCallback()],verbose=0)except:passhistory = model.fit(np.arange(100).reshape(5, 20), np.zeros(5),epochs=10, batch_size=1, callbacks=[callback],verbose=0)# Only 6 more epochs are run, since first training got interrupted at# zero-indexed epoch 4, second training will continue from 4 to 9.len(history.history['loss'])6
backup_dir
BackupAndRestore callback of another training run,
or by another callback (e.g. ModelCheckpoint)
of the same training run.
save_freq
"epoch", integer, or False. When set to "epoch"
the callback saves the checkpoint at the end of each epoch.
When set to an integer, the callback saves the checkpoint every
save_freq batches. Set save_freq=False only if using
preemption checkpointing (i.e. with save_before_preemption=True).
delete_checkpoint
True. This BackupAndRestore
callback works by saving a checkpoint to back up the training state.
If delete_checkpoint=True, the checkpoint will be deleted after
training is finished. Use False if you'd like to keep the checkpoint
for future usage.
model
on_batch_beginon_batch_begin(
batch, logs=None
)
A backwards compatibility alias for on_train_batch_begin.
on_batch_endon_batch_end(
batch, logs=None
)
A backwards compatibility alias for on_train_batch_end.
on_epoch_beginon_epoch_begin(
epoch, logs=None
)
Called at the start of an epoch.
Subclasses should override for any actions to run. This function should only be called during TRAIN mode.
epoch
logs
on_epoch_endon_epoch_end(
epoch, logs=None
)
Called at the end of an epoch.
Subclasses should override for any actions to run. This function should only be called during TRAIN mode.
epoch
logs
val_. For training epoch, the values of
the Model's metrics are returned. Example:
{'loss': 0.2, 'accuracy': 0.7}.
on_predict_batch_beginon_predict_batch_begin(
batch, logs=None
)
Called at the beginning of a batch in predict methods.
Subclasses should override for any actions to run.
Note that if the steps_per_execution argument to compile in
Model is set to N, this method will only be called every
N batches.
batch
logs
on_predict_batch_endon_predict_batch_end(
batch, logs=None
)
Called at the end of a batch in predict methods.
Subclasses should override for any actions to run.
Note that if the steps_per_execution argument to compile in
Model is set to N, this method will only be called every
N batches.
batch
logs
on_predict_beginon_predict_begin(
logs=None
)
Called at the beginning of prediction.
Subclasses should override for any actions to run.
logs
on_predict_endon_predict_end(
logs=None
)
Called at the end of prediction.
Subclasses should override for any actions to run.
logs
on_test_batch_beginon_test_batch_begin(
batch, logs=None
)
Called at the beginning of a batch in evaluate methods.
Also called at the beginning of a validation batch in the fit
methods, if validation data is provided.
Subclasses should override for any actions to run.
Note that if the steps_per_execution argument to compile in
Model is set to N, this method will only be called every
N batches.
batch
logs
on_test_batch_endon_test_batch_end(
batch, logs=None
)
Called at the end of a batch in evaluate methods.
Also called at the end of a validation batch in the fit
methods, if validation data is provided.
Subclasses should override for any actions to run.
Note that if the steps_per_execution argument to compile in
Model is set to N, this method will only be called every
N batches.
batch
logs
on_test_beginon_test_begin(
logs=None
)
Called at the beginning of evaluation or validation.
Subclasses should override for any actions to run.
logs
on_test_endon_test_end(
logs=None
)
Called at the end of evaluation or validation.
Subclasses should override for any actions to run.
logs
on_test_batch_end() is passed to this argument for this method
but that may change in the future.
on_train_batch_beginon_train_batch_begin(
batch, logs=None
)
Called at the beginning of a training batch in fit methods.
Subclasses should override for any actions to run.
Note that if the steps_per_execution argument to compile in
Model is set to N, this method will only be called every
N batches.
batch
logs
on_train_batch_endon_train_batch_end(
batch, logs=None
)
Called at the end of a training batch in fit methods.
Subclasses should override for any actions to run.
Note that if the steps_per_execution argument to compile in
Model is set to N, this method will only be called every
N batches.
batch
logs
on_train_beginon_train_begin(
logs=None
)
Get training state from temporary file and restore it.
on_train_endon_train_end(
logs=None
)
Called at the end of training.
Subclasses should override for any actions to run.
logs
on_epoch_end() is passed to this argument for this method but
that may change in the future.
set_modelset_model(
model
)
set_paramsset_params(
params
)
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates. Some content is licensed under the numpy license.
Last updated 2024-06-07 UTC.