pypots.optim package#
pypots.optim.adam#
The optimizer wrapper for PyTorch Adam.
- class pypots.optim.adam.Adam(lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False, lr_scheduler=None)[source]#
Bases:
OptimizerThe optimizer wrapper for PyTorch Adam
torch.optim.Adam.- Parameters:
lr (float) – The learning rate of the optimizer.
betas (Tuple[float, float]) – Coefficients used for computing running averages of gradient and its square.
eps (float) – Term added to the denominator to improve numerical stability.
weight_decay (float) – Weight decay (L2 penalty).
amsgrad (bool) – Whether to use the AMSGrad variant of this algorithm from the paper [32].
lr_scheduler (pypots.optim.lr_scheduler.base.LRScheduler) – The learning rate scheduler of the optimizer.
- add_param_group(param_group)#
Add a param group to the optimizer param_groups.
- load_state_dict(state_dict)#
Loads the optimizer state.
- Parameters:
state_dict – Optimizer state. It should be an object returned from
state_dict().- Return type:
- state_dict()#
Returns the state of the optimizer as a dict.
- Returns:
The state dict of the optimizer, which contains two entries: 1). state - a dict holding current optimization state. Its content differs between optimizer classes. 2). param_groups - a list containing all parameter groups where each parameter group is a dict
- Return type:
state_dict
- step(closure=None)#
Performs a single optimization step (parameter update).
pypots.optim.adamw#
The optimizer wrapper for PyTorch AdamW.
- class pypots.optim.adamw.AdamW(lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01, amsgrad=False, lr_scheduler=None)[source]#
Bases:
OptimizerThe optimizer wrapper for PyTorch AdamW
torch.optim.AdamW.- Parameters:
lr (float) – The learning rate of the optimizer.
betas (Tuple[float, float]) – Coefficients used for computing running averages of gradient and its square.
eps (float) – Term added to the denominator to improve numerical stability.
weight_decay (float) – Weight decay (L2 penalty).
amsgrad (bool) – Whether to use the AMSGrad variant of this algorithm from the paper [32].
lr_scheduler (pypots.optim.lr_scheduler.base.LRScheduler) – The learning rate scheduler of the optimizer.
- add_param_group(param_group)#
Add a param group to the optimizer param_groups.
- load_state_dict(state_dict)#
Loads the optimizer state.
- Parameters:
state_dict – Optimizer state. It should be an object returned from
state_dict().- Return type:
- state_dict()#
Returns the state of the optimizer as a dict.
- Returns:
The state dict of the optimizer, which contains two entries: 1). state - a dict holding current optimization state. Its content differs between optimizer classes. 2). param_groups - a list containing all parameter groups where each parameter group is a dict
- Return type:
state_dict
- step(closure=None)#
Performs a single optimization step (parameter update).
pypots.optim.adagrad#
The optimizer wrapper for PyTorch Adagrad.
- class pypots.optim.adagrad.Adagrad(lr=0.01, lr_decay=0, weight_decay=0.01, initial_accumulator_value=0.01, eps=1e-08, lr_scheduler=None)[source]#
Bases:
OptimizerThe optimizer wrapper for PyTorch Adagrad
torch.optim.Adagrad.- Parameters:
lr (float) – The learning rate of the optimizer.
lr_decay (float) – Learning rate decay.
weight_decay (float) – Weight decay (L2 penalty).
eps (float) – Term added to the denominator to improve numerical stability.
initial_accumulator_value (float) – A floating point value. Starting value for the accumulators, must be positive.
lr_scheduler (pypots.optim.lr_scheduler.base.LRScheduler) – The learning rate scheduler of the optimizer.
- add_param_group(param_group)#
Add a param group to the optimizer param_groups.
- load_state_dict(state_dict)#
Loads the optimizer state.
- Parameters:
state_dict – Optimizer state. It should be an object returned from
state_dict().- Return type:
- state_dict()#
Returns the state of the optimizer as a dict.
- Returns:
The state dict of the optimizer, which contains two entries: 1). state - a dict holding current optimization state. Its content differs between optimizer classes. 2). param_groups - a list containing all parameter groups where each parameter group is a dict
- Return type:
state_dict
- step(closure=None)#
Performs a single optimization step (parameter update).
pypots.optim.rmsprop#
The optimizer wrapper for PyTorch RMSprop.
- class pypots.optim.rmsprop.RMSprop(lr=0.001, momentum=0, alpha=0.99, eps=1e-08, centered=False, weight_decay=0, lr_scheduler=None)[source]#
Bases:
OptimizerThe optimizer wrapper for PyTorch RMSprop
torch.optim.RMSprop.- Parameters:
lr (float) – The learning rate of the optimizer.
momentum (float) – Momentum factor.
alpha (float) – Smoothing constant.
eps (float) – Term added to the denominator to improve numerical stability.
centered (bool) – If True, compute the centered RMSProp, the gradient is normalized by an estimation of its variance
weight_decay (float) – Weight decay (L2 penalty).
lr_scheduler (pypots.optim.lr_scheduler.base.LRScheduler) – The learning rate scheduler of the optimizer.
- add_param_group(param_group)#
Add a param group to the optimizer param_groups.
- load_state_dict(state_dict)#
Loads the optimizer state.
- Parameters:
state_dict – Optimizer state. It should be an object returned from
state_dict().- Return type:
- state_dict()#
Returns the state of the optimizer as a dict.
- Returns:
The state dict of the optimizer, which contains two entries: 1). state - a dict holding current optimization state. Its content differs between optimizer classes. 2). param_groups - a list containing all parameter groups where each parameter group is a dict
- Return type:
state_dict
- step(closure=None)#
Performs a single optimization step (parameter update).
pypots.optim.sgd#
The optimizer wrapper for PyTorch SGD torch.optim.SGD.
- class pypots.optim.sgd.SGD(lr=0.001, momentum=0, weight_decay=0, dampening=0, nesterov=False, lr_scheduler=None)[source]#
Bases:
OptimizerThe optimizer wrapper for PyTorch SGD
torch.optim.SGD.- Parameters:
lr (float) – The learning rate of the optimizer.
momentum (float) – Momentum factor.
weight_decay (float) – Weight decay (L2 penalty).
dampening (float) – Dampening for momentum.
nesterov (bool) – Whether to enable Nesterov momentum.
lr_scheduler (pypots.optim.lr_scheduler.base.LRScheduler) – The learning rate scheduler of the optimizer.
- add_param_group(param_group)#
Add a param group to the optimizer param_groups.
- load_state_dict(state_dict)#
Loads the optimizer state.
- Parameters:
state_dict – Optimizer state. It should be an object returned from
state_dict().- Return type:
- state_dict()#
Returns the state of the optimizer as a dict.
- Returns:
The state dict of the optimizer, which contains two entries: 1). state - a dict holding current optimization state. Its content differs between optimizer classes. 2). param_groups - a list containing all parameter groups where each parameter group is a dict
- Return type:
state_dict
- step(closure=None)#
Performs a single optimization step (parameter update).
pypots.optim.base#
The base wrapper for PyTorch optimizers (https://pytorch.org/docs/stable/optim.html#algorithms), also is the base class for all optimizers in pypots.optim.
The optimizers in pypots.optim are all wrappers for PyTorch optimizers. pypots.optim.optimizers inherent all functionalities from torch.optim.optimizers (so you can see many docstrings are copied from torch), but are more powerful. So far, they are designed to:
1). separate the hyperparameters of models and optimizers in PyPOTS, so that users don’t have to put all hyperparameters in one place, which could result in a mess and be not readable;
2). provide additional functionalities, such as learning rate scheduling, etc.;
- class pypots.optim.base.Optimizer(lr, lr_scheduler=None)[source]#
Bases:
ABCThe base wrapper for PyTorch optimizers, also is the base class for all optimizers in PyPOTS.
- Parameters:
lr (float) – The learning rate of the optimizer.
lr_scheduler (pypots.optim.lr_scheduler.base.LRScheduler) – The learning rate scheduler of the optimizer.
- Attributes:
torch_optimizer – The torch optimizer wrapped by this class.
- load_state_dict(state_dict)[source]#
Loads the optimizer state.
- Parameters:
state_dict – Optimizer state. It should be an object returned from
state_dict().- Return type:
- state_dict()[source]#
Returns the state of the optimizer as a dict.
- Returns:
The state dict of the optimizer, which contains two entries: 1). state - a dict holding current optimization state. Its content differs between optimizer classes. 2). param_groups - a list containing all parameter groups where each parameter group is a dict
- Return type:
state_dict
pypots.optim.lr_scheduler#
Learning rate schedulers available in PyPOTS. Their functionalities are the same with those in PyTorch,
the only difference that is also why we implement them is that you don’t have to pass according optimizers
into them immediately while initializing them. Instead, you can pass them into pypots.optim.base.Optimizer
after initialization and call their init_scheduler() method in pypots.optim.base.Optimizer.init_optimizer()
to initialize schedulers together with optimizers.
- class pypots.optim.lr_scheduler.LambdaLR(lr_lambda, last_epoch=-1, verbose=False)[source]#
Bases:
LRSchedulerSets the learning rate of each parameter group to the initial lr times a given function. When last_epoch=-1, sets initial lr as lr.
- Parameters:
lr_lambda (Callable or list,) – A function which computes a multiplicative factor given an integer parameter epoch, or a list of such functions, one for each group in optimizer.param_groups.
last_epoch (int,) – The index of last epoch. Default: -1.
verbose (bool,) – If
True, prints a message to stdout for each update. Default:False.
Notes
This class works the same with
torch.optim.lr_scheduler.LambdaLR. The only difference that is also why we implement them is that you don’t have to pass according optimizers into them immediately while initializing them.Example
>>> lambda1 = lambda epoch: epoch // 30 >>> scheduler = LambdaLR(lr_lambda=lambda1) >>> adam = pypots.optim.Adam(lr=1e-3, lr_scheduler=scheduler)
- init_scheduler(optimizer)[source]#
Initialize the scheduler. This method should be called in
pypots.optim.base.Optimizer.init_optimizer()to initialize the scheduler together with the optimizer.- Parameters:
optimizer (torch.optim.Optimizer) – The optimizer to be scheduled.
- get_last_lr()#
Return last computed learning rate by current scheduler.
- static print_lr(is_verbose, group, lr)#
Display the current learning rate.
- step()#
Step could be called after every batch update. This should be called in
pypots.optim.base.Optimizer.step()afterpypots.optim.base.Optimizer.torch_optimizer.step().
- class pypots.optim.lr_scheduler.MultiplicativeLR(lr_lambda, last_epoch=-1, verbose=False)[source]#
Bases:
LRSchedulerMultiply the learning rate of each parameter group by the factor given in the specified function. When last_epoch=-1, sets initial lr as lr.
- Parameters:
lr_lambda (Callable or list,) – A function which computes a multiplicative factor given an integer parameter epoch, or a list of such functions, one for each group in optimizer.param_groups.
last_epoch (int,) – The index of last epoch. Default: -1.
verbose (bool,) – If
True, prints a message to stdout for each update. Default:False.
Notes
This class works the same with
torch.optim.lr_scheduler.MultiplicativeLR. The only difference that is also why we implement them is that you don’t have to pass according optimizers into them immediately while initializing them.Example
>>> lmbda = lambda epoch: 0.95 >>> # xdoctest: +SKIP >>> scheduler = MultiplicativeLR(lr_lambda=lmbda) >>> adam = pypots.optim.Adam(lr=1e-3, lr_scheduler=scheduler)
- init_scheduler(optimizer)[source]#
Initialize the scheduler. This method should be called in
pypots.optim.base.Optimizer.init_optimizer()to initialize the scheduler together with the optimizer.- Parameters:
optimizer (torch.optim.Optimizer) – The optimizer to be scheduled.
- get_last_lr()#
Return last computed learning rate by current scheduler.
- static print_lr(is_verbose, group, lr)#
Display the current learning rate.
- step()#
Step could be called after every batch update. This should be called in
pypots.optim.base.Optimizer.step()afterpypots.optim.base.Optimizer.torch_optimizer.step().
- class pypots.optim.lr_scheduler.StepLR(step_size, gamma=0.1, last_epoch=-1, verbose=False)[source]#
Bases:
LRSchedulerDecays the learning rate of each parameter group by gamma every step_size epochs. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.
- Parameters:
Notes
This class works the same with
torch.optim.lr_scheduler.StepLR. The only difference that is also why we implement them is that you don’t have to pass according optimizers into them immediately while initializing them.Example
>>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.05 if epoch < 30 >>> # lr = 0.005 if 30 <= epoch < 60 >>> # lr = 0.0005 if 60 <= epoch < 90 >>> # ... >>> # xdoctest: +SKIP >>> scheduler = StepLR(step_size=30, gamma=0.1) >>> adam = pypots.optim.Adam(lr=1e-3, lr_scheduler=scheduler)
- get_last_lr()#
Return last computed learning rate by current scheduler.
- init_scheduler(optimizer)#
Initialize the scheduler. This method should be called in
pypots.optim.base.Optimizer.init_optimizer()to initialize the scheduler together with the optimizer.- Parameters:
optimizer (torch.optim.Optimizer) – The optimizer to be scheduled.
- static print_lr(is_verbose, group, lr)#
Display the current learning rate.
- step()#
Step could be called after every batch update. This should be called in
pypots.optim.base.Optimizer.step()afterpypots.optim.base.Optimizer.torch_optimizer.step().
- class pypots.optim.lr_scheduler.MultiStepLR(milestones, gamma=0.1, last_epoch=-1, verbose=False)[source]#
Bases:
LRSchedulerDecays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.
- Parameters:
Notes
This class works the same with
torch.optim.lr_scheduler.MultiStepLR. The only difference that is also why we implement them is that you don’t have to pass according optimizers into them immediately while initializing them.Example
>>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.05 if epoch < 30 >>> # lr = 0.005 if 30 <= epoch < 80 >>> # lr = 0.0005 if epoch >= 80 >>> # xdoctest: +SKIP >>> scheduler = MultiStepLR(milestones=[30,80], gamma=0.1) >>> adam = pypots.optim.Adam(lr=1e-3, lr_scheduler=scheduler)
- get_last_lr()#
Return last computed learning rate by current scheduler.
- init_scheduler(optimizer)#
Initialize the scheduler. This method should be called in
pypots.optim.base.Optimizer.init_optimizer()to initialize the scheduler together with the optimizer.- Parameters:
optimizer (torch.optim.Optimizer) – The optimizer to be scheduled.
- static print_lr(is_verbose, group, lr)#
Display the current learning rate.
- step()#
Step could be called after every batch update. This should be called in
pypots.optim.base.Optimizer.step()afterpypots.optim.base.Optimizer.torch_optimizer.step().
- class pypots.optim.lr_scheduler.ConstantLR(factor=0.3333333333333333, total_iters=5, last_epoch=-1, verbose=False)[source]#
Bases:
LRSchedulerDecays the learning rate of each parameter group by a small constant factor until the number of epoch reaches a pre-defined milestone: total_iters. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.
- Parameters:
factor (float, default=1./3.) – The number we multiply learning rate until the milestone.
total_iters (int, default=5,) – The number of steps that the scheduler decays the learning rate.
last_epoch (int) – The index of last epoch. Default: -1.
verbose (bool) – If
True, prints a message to stdout for each update. Default:False.
Notes
This class works the same with
torch.optim.lr_scheduler.ConstantLR. The only difference that is also why we implement them is that you don’t have to pass according optimizers into them immediately while initializing them.Example
>>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.025 if epoch == 0 >>> # lr = 0.025 if epoch == 1 >>> # lr = 0.025 if epoch == 2 >>> # lr = 0.025 if epoch == 3 >>> # lr = 0.05 if epoch >= 4 >>> # xdoctest: +SKIP >>> scheduler = ConstantLR(factor=0.5, total_iters=4) >>> adam = pypots.optim.Adam(lr=1e-3, lr_scheduler=scheduler)
- get_last_lr()#
Return last computed learning rate by current scheduler.
- init_scheduler(optimizer)#
Initialize the scheduler. This method should be called in
pypots.optim.base.Optimizer.init_optimizer()to initialize the scheduler together with the optimizer.- Parameters:
optimizer (torch.optim.Optimizer) – The optimizer to be scheduled.
- static print_lr(is_verbose, group, lr)#
Display the current learning rate.
- step()#
Step could be called after every batch update. This should be called in
pypots.optim.base.Optimizer.step()afterpypots.optim.base.Optimizer.torch_optimizer.step().
- class pypots.optim.lr_scheduler.ExponentialLR(gamma, last_epoch=-1, verbose=False)[source]#
Bases:
LRSchedulerDecays the learning rate of each parameter group by gamma every epoch. When last_epoch=-1, sets initial lr as lr.
- Parameters:
Notes
This class works the same with
torch.optim.lr_scheduler.ExponentialLR. The only difference that is also why we implement them is that you don’t have to pass according optimizers into them immediately while initializing them.Example
>>> scheduler = ExponentialLR(gamma=0.1) >>> adam = pypots.optim.Adam(lr=1e-3, lr_scheduler=scheduler)
- get_last_lr()#
Return last computed learning rate by current scheduler.
- init_scheduler(optimizer)#
Initialize the scheduler. This method should be called in
pypots.optim.base.Optimizer.init_optimizer()to initialize the scheduler together with the optimizer.- Parameters:
optimizer (torch.optim.Optimizer) – The optimizer to be scheduled.
- static print_lr(is_verbose, group, lr)#
Display the current learning rate.
- step()#
Step could be called after every batch update. This should be called in
pypots.optim.base.Optimizer.step()afterpypots.optim.base.Optimizer.torch_optimizer.step().
- class pypots.optim.lr_scheduler.LinearLR(start_factor=0.3333333333333333, end_factor=1.0, total_iters=5, last_epoch=-1, verbose=False)[source]#
Bases:
LRSchedulerDecays the learning rate of each parameter group by linearly changing small multiplicative factor until the number of epoch reaches a pre-defined milestone: total_iters. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.
- Parameters:
start_factor (float, default=1.0 / 3,) – The number we multiply learning rate in the first epoch. The multiplication factor changes towards end_factor in the following epochs.
end_factor (float, default=1.0,) – The number we multiply learning rate at the end of linear changing process.
total_iters (int, default=5,) – The number of iterations that multiplicative factor reaches to 1.
last_epoch (int) – The index of last epoch. Default: -1.
verbose (bool) – If
True, prints a message to stdout for each update. Default:False.
Notes
This class works the same with
torch.optim.lr_scheduler.LinearLR. The only difference that is also why we implement them is that you don’t have to pass according optimizers into them immediately while initializing them.Example
>>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.025 if epoch == 0 >>> # lr = 0.03125 if epoch == 1 >>> # lr = 0.0375 if epoch == 2 >>> # lr = 0.04375 if epoch == 3 >>> # lr = 0.05 if epoch >= 4 >>> # xdoctest: +SKIP >>> scheduler = LinearLR(start_factor=0.5, total_iters=4) >>> adam = pypots.optim.Adam(lr=1e-3, lr_scheduler=scheduler)
- get_last_lr()#
Return last computed learning rate by current scheduler.
- init_scheduler(optimizer)#
Initialize the scheduler. This method should be called in
pypots.optim.base.Optimizer.init_optimizer()to initialize the scheduler together with the optimizer.- Parameters:
optimizer (torch.optim.Optimizer) – The optimizer to be scheduled.
- static print_lr(is_verbose, group, lr)#
Display the current learning rate.
- step()#
Step could be called after every batch update. This should be called in
pypots.optim.base.Optimizer.step()afterpypots.optim.base.Optimizer.torch_optimizer.step().