pypots.nn package#

pypots.nn.functional#

pypots.nn.functional.nonstationary_norm(X, missing_mask=None)[source]#

Normalization from Non-stationary Transformer. Please refer to [22] for more details.

Parameters:
  • X (torch.Tensor) – Input data to be normalized. Shape: (n_samples, n_steps (seq_len), n_features).

  • missing_mask (torch.Tensor, optional) – Missing mask has the same shape as X. 1 indicates observed and 0 indicates missing.

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

  • X_enc (torch.Tensor) – Normalized data. Shape: (n_samples, n_steps (seq_len), n_features).

  • means (torch.Tensor) – Means values for de-normalization. Shape: (n_samples, n_features) or (n_samples, 1, n_features).

  • stdev (torch.Tensor) – Standard deviation values for de-normalization. Shape: (n_samples, n_features) or (n_samples, 1, n_features).

pypots.nn.functional.nonstationary_denorm(X, means, stdev)[source]#

De-Normalization from Non-stationary Transformer. Please refer to [22] for more details.

Parameters:
  • X (torch.Tensor) – Input data to be de-normalized. Shape: (n_samples, n_steps (seq_len), n_features).

  • means (torch.Tensor) – Means values for de-normalization . Shape: (n_samples, n_features) or (n_samples, 1, n_features).

  • stdev (torch.Tensor) – Standard deviation values for de-normalization. Shape: (n_samples, n_features) or (n_samples, 1, n_features).

Returns:

X_denorm – De-normalized data. Shape: (n_samples, n_steps (seq_len), n_features).

Return type:

torch.Tensor

pypots.nn.modules.autoformer#

The package including the modules of Autoformer.

Refer to the paper Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with autocorrelation for long-term series forecasting. In Advances in Neural Information Processing Systems, volume 34, pages 22419–22430. Curran Associates, Inc., 2021.

class pypots.nn.modules.autoformer.AutoCorrelation(factor=1, attention_dropout=0.1)[source]#
AutoCorrelation Mechanism with the following two phases:
  1. period-based dependencies discovery

  2. time delay aggregation

This block can replace the self-attention family mechanism seamlessly.

time_delay_agg_training(values, corr)[source]#

SpeedUp version of Autocorrelation (a batch-normalization style design) This is for the training phase.

time_delay_agg_inference(values, corr)[source]#

SpeedUp version of Autocorrelation (a batch-normalization style design) This is for the inference phase.

time_delay_agg_full(values, corr)[source]#

Standard version of Autocorrelation

forward(queries, keys, values, attn_mask)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.autoformer.AutoCorrelationLayer(correlation, d_model, n_heads, d_keys=None, d_values=None)[source]#
forward(queries, keys, values, attn_mask)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.autoformer.SeasonalLayerNorm(n_channels)[source]#

A special designed layer normalization for the seasonal part.

forward(x)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.autoformer.MovingAvgBlock(kernel_size, stride)[source]#

The moving average block to highlight the trend of time series.

forward(x)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.autoformer.SeriesDecompositionBlock(kernel_size)[source]#

Series decomposition block

forward(x)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.autoformer.AutoformerEncoderLayer(attention, d_model, d_ff=None, moving_avg=25, dropout=0.1, activation='relu')[source]#

Autoformer encoder layer with the progressive decomposition architecture.

forward(x, attn_mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.autoformer.AutoformerDecoderLayer(self_attention, cross_attention, d_model, c_out, d_ff=None, moving_avg=25, dropout=0.1, activation='relu')[source]#

Autoformer decoder layer with the progressive decomposition architecture

forward(x, cross, x_mask=None, cross_mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.autoformer.AutoformerEncoder(n_layers, n_heads, d_model, d_ffn, factor, moving_avg_window_size, dropout, activation='relu')[source]#
forward(x, attn_mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.brits#

The package including the modules of BRITS.

Refer to the paper Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. BRITS: Bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.

class pypots.nn.modules.brits.BackboneRITS(n_steps, n_features, rnn_hidden_size)[source]#

model RITS: Recurrent Imputation for Time Series

Attributes:
  • n_steps – sequence length (number of time steps)

  • n_features – number of features (input dimensions)

  • rnn_hidden_size – the hidden size of the RNN cell

  • rnn_cell – the LSTM cell to model temporal data

  • temp_decay_h – the temporal decay module to decay RNN hidden state

  • temp_decay_x – the temporal decay module to decay data in the raw feature space

  • hist_reg – the temporal-regression module to project RNN hidden state into the raw feature space

  • feat_reg – the feature-regression module

  • combining_weight – the module used to generate the weight to combine history regression and feature regression

Parameters:
  • n_steps (int) – sequence length (number of time steps)

  • n_features (int) – number of features (input dimensions)

  • rnn_hidden_size (int) – the hidden size of the RNN cell

forward(inputs, direction)[source]#
Parameters:
  • inputs (dict) – Input data, a dictionary includes feature values, missing masks, and time-gap values.

  • direction (str) – A keyword to extract data from inputs.

Return type:

Tuple[Tensor, Tensor, Tensor, Tensor]

Returns:

  • imputed_data – Input data with missing parts imputed. Shape of [batch size, sequence length, feature number].

  • estimations – Reconstructed data. Shape of [batch size, sequence length, feature number].

  • hidden_states (tensor,) – [batch size, RNN hidden size]

  • reconstruction_loss – reconstruction loss

class pypots.nn.modules.brits.BackboneBRITS(n_steps, n_features, rnn_hidden_size)[source]#

model BRITS: Bidirectional RITS BRITS consists of two RITS, which take time-series data from two directions (forward/backward) respectively.

Attributes:
  • n_steps – sequence length (number of time steps)

  • n_features – number of features (input dimensions)

  • rnn_hidden_size – the hidden size of the RNN cell

  • rits_f (RITS object) – the forward RITS model

  • rits_b (RITS object) – the backward RITS model

forward(inputs)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, ...]

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.brits.FeatureRegression(input_size)[source]#

The module used to capture the correlation between features for imputation in BRITS.

Attributes:
  • W (tensor) – The weights (parameters) of the module.

  • b (tensor) – The bias of the module.

  • m (buffer) (tensor) – The mask matrix, a squire matrix with diagonal entries all zeroes while left parts all ones. It is applied to the weight matrix to mask out the estimation contributions from features themselves. It is used to help enhance the imputation performance of the network.

Parameters:

input_size (the feature dimension of the input) –

forward(x)[source]#

Forward processing of the NN module.

Parameters:

x (tensor,) – the input for processing

Returns:

output – the processed result containing imputation from feature regression

Return type:

tensor,

pypots.nn.modules.crli#

The package including the modules of CRLI.

Refer to the paper Qianli Ma, Chuxin Chen, Sen Li, and Garrison W. Cottrell. Learning Representations for Incomplete Time Series Clustering. In AAAI, 35(10):8837–8846, May 2021.

class pypots.nn.modules.crli.BackboneCRLI(n_steps, n_features, n_generator_layers, rnn_hidden_size, decoder_fcn_output_dims, rnn_cell_type='GRU')[source]#
forward(X, missing_mask)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, ...]

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.crli.CrliGenerator(n_layers, n_features, d_hidden, cell_type)[source]#
forward(X, missing_mask)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor]

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.crli.CrliDecoder(n_steps, d_input, d_output, fcn_output_dims=None)[source]#
forward(generator_fb_hidden_states)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor]

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.crli.CrliDiscriminator(cell_type, d_input)[source]#
forward(X, missing_mask, imputation_latent)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tensor

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.crossformer#

The package including the modules of Crossformer.

Refer to the paper Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The 11th ICLR, 2023.

class pypots.nn.modules.crossformer.CrossformerEncoder(attn_layers)[source]#
forward(x, src_mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.crossformer.CrossformerDecoder(layers)[source]#
forward(x, cross)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.crossformer.TwoStageAttentionLayer(seg_num, factor, d_model, n_heads, d_k, d_v, d_ff=None, dropout=0.1, attn_dropout=0.1)[source]#

The Two Stage Attention (TSA) Layer input/output shape: [batch_size, Data_dim(D), Seg_num(L), d_model]

forward(x)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.crossformer.ScaleBlock(win_size, d_model, n_heads, d_ff, depth, dropout, seg_num, factor)[source]#
forward(x, attn_mask=None, tau=None, delta=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.crossformer.CrossformerDecoderLayer(self_attention, cross_attention, seg_len, d_model, d_ff=None, dropout=0.1)[source]#
forward(x, cross)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.csdi#

The package including the modules of CSDI.

Refer to the paper Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation. In NeurIPS, 2021.

class pypots.nn.modules.csdi.BackboneCSDI(n_layers, n_heads, n_channels, d_target, d_time_embedding, d_feature_embedding, d_diffusion_embedding, is_unconditional, n_diffusion_steps, schedule, beta_start, beta_end)[source]#
forward(observed_data, cond_mask, side_info, n_sampling_times)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.csdi.CsdiDiffusionEmbedding(n_diffusion_steps, d_embedding=128, d_projection=None)[source]#
forward(diffusion_step)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.csdi.CsdiDiffusionModel(n_diffusion_steps, d_diffusion_embedding, d_input, d_side, n_channels, n_heads, n_layers)[source]#
forward(x, cond_info, diffusion_step)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.csdi.CsdiResidualBlock(d_side, n_channels, diffusion_embedding_dim, nheads)[source]#
forward(x, cond_info, diffusion_emb)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.dlinear#

The package including the modules of DLinear.

Refer to the paper Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI, volume 37, pages 11121–11128, Jun. 2023.

class pypots.nn.modules.dlinear.BackboneDLinear(n_steps, n_features, individual=False, d_model=None)[source]#
forward(seasonal_init, trend_init)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.etsformer#

The package including the modules of ETSformer.

Refer to the paper Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. ETSformer: Exponential smoothing transformers for time-series forecasting. In ICLR, 2023.

class pypots.nn.modules.etsformer.ETSformerEncoder(layers)[source]#
forward(res, level, attn_mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.etsformer.ETSformerEncoderLayer(d_model, n_heads, c_out, seq_len, pred_len, k, dim_feedforward=None, dropout=0.1, activation='sigmoid', layer_norm_eps=1e-05)[source]#
forward(res, level, attn_mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.etsformer.ETSformerDecoder(layers)[source]#
forward(growths, seasons)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.etsformer.ETSformerDecoderLayer(d_model, nhead, c_out, pred_len, dropout=0.1)[source]#
forward(growth, season)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.fedformer#

The package including the modules of FEDformer.

Refer to the paper Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 27268–27286. PMLR, 17–23 Jul 2022.

class pypots.nn.modules.fedformer.FEDformerEncoder(n_steps, n_layers, n_heads, d_model, d_ffn, moving_avg_window_size, dropout, version='Fourier', modes=32, mode_select='random', activation='relu')[source]#
forward(X, attn_mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.fedformer.FEDformerDecoder(n_steps, n_pred_steps, n_layers, n_heads, d_model, d_ffn, d_output, moving_avg_window_size, dropout, version='Fourier', modes=32, mode_select='random', activation='relu')[source]#
forward(X, attn_mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.fedformer.MultiWaveletTransform(ich=1, k=8, alpha=16, c=128, nCZ=1, L=0, base='legendre', attention_dropout=0.1)[source]#

1D multiwavelet block.

forward(queries, keys, values, attn_mask)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.fedformer.MultiWaveletCross(in_channels, out_channels, seq_len_q, seq_len_kv, modes, c=64, k=8, ich=512, L=0, base='legendre', mode_select_method='random', initializer=None, activation='tanh', **kwargs)[source]#

1D Multiwavelet Cross Attention layer.

forward(q, k, v, mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.fedformer.FourierBlock(in_channels, out_channels, seq_len, modes=0, mode_select_method='random')[source]#
forward(q, k, v, mask)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.fedformer.FourierCrossAttention(in_channels, out_channels, seq_len_q, seq_len_kv, modes=64, mode_select_method='random', activation='tanh', policy=0, num_heads=8)[source]#
forward(q, k, v, mask)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.gpvae#

The package including the modules of GP-VAE.

Refer to the paper Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. GP-VAE: Deep probabilistic time series imputation. In International conference on artificial intelligence and statistics, pages 1651–1661. PMLR, 2020.

class pypots.nn.modules.gpvae.BackboneGPVAE(input_dim, time_length, latent_dim, encoder_sizes=(64, 64), decoder_sizes=(64, 64), beta=1, M=1, K=1, kernel='cauchy', sigma=1.0, length_scale=7.0, kernel_scales=1, window_size=24)[source]#

model GPVAE with Gaussian Process prior

Parameters:
  • input_dim (int,) – the feature dimension of the input

  • time_length (int,) – the length of each time series

  • latent_dim (int,) – the feature dimension of the latent embedding

  • encoder_sizes (tuple,) – the tuple of the network size in encoder

  • decoder_sizes (tuple,) – the tuple of the network size in decoder

  • beta (float,) – the weight of the KL divergence

  • M (int,) – the number of Monte Carlo samples for ELBO estimation

  • K (int,) – the number of importance weights for IWAE model

  • kernel (str,) – the Gaussian Process kernel [“cauchy”, “diffusion”, “rbf”, “matern”]

  • sigma (float,) – the scale parameter for a kernel function

  • length_scale (float,) – the length scale parameter for a kernel function

  • kernel_scales (int,) – the number of different length scales over latent space dimensions

forward(X, missing_mask)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.grud#

The package including the modules of GRU-D.

Refer to the paper “Che, Z., Purushotham, S., Cho, K., Sontag, D.A., & Liu, Y. (2018). Recurrent Neural Networks for Multivariate Time Series with Missing Values. Scientific Reports.”

class pypots.nn.modules.grud.BackboneGRUD(n_steps, n_features, rnn_hidden_size)[source]#
forward(X, missing_mask, deltas, empirical_mean, X_filledLOCF)[source]#

Forward processing of GRU-D.

Parameters:
  • X

  • missing_mask

  • deltas

  • empirical_mean

  • X_filledLOCF

Return type:

Tuple[list, Tensor]

Returns:

  • classification_pred

  • logits

class pypots.nn.modules.grud.TemporalDecay(input_size, output_size, diag=False)[source]#

The module used to generate the temporal decay factor gamma in the GRU-D model. Please refer to the original paper [16] for more details.

Attributes:
  • W (tensor,) – The weights (parameters) of the module.

  • b (tensor,) – The bias of the module.

Parameters:
  • input_size (int,) – the feature dimension of the input

  • output_size (int,) – the feature dimension of the output

  • diag (bool,) – whether to product the weight with an identity matrix before forward processing

References

forward(delta)[source]#

Forward processing of this NN module.

Parameters:

delta (tensor, shape [n_samples, n_steps, n_features]) – The time gaps.

Returns:

gamma – The temporal decay factor.

Return type:

tensor, of the same shape with parameter delta, values in (0,1]

pypots.nn.modules.informer#

The package including the modules of Informer.

Refer to the paper Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106–11115, 2021.

class pypots.nn.modules.informer.ProbAttention(mask_flag=True, factor=5, attention_dropout=0.1, scale=None)[source]#
forward(q, k, v, attn_mask=None, **kwargs)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.informer.ConvLayer(c_in)[source]#
forward(x)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.informer.InformerEncoderLayer(attention, d_model, d_ff=None, dropout=0.1, activation='relu')[source]#
forward(x, attn_mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.informer.InformerDecoderLayer(self_attention, cross_attention, d_model, d_ff=None, dropout=0.1, activation='relu')[source]#
forward(x, cross, x_mask=None, cross_mask=None, tau=None, delta=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.informer.InformerEncoder(attn_layers, conv_layers=None, norm_layer=None)[source]#
forward(x, attn_mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.informer.InformerDecoder(layers, norm_layer=None, projection=None)[source]#
forward(x, cross, x_mask=None, cross_mask=None, trend=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.mrnn#

The package including the modules of M-RNN.

Refer to the paper Jinsung Yoon, William R. Zame, and Mihaela van der Schaar. Estimating missing data in temporal data streams using multi-directional recurrent neural networks. IEEE Transactions on Biomedical Engineering, 66(5):14771490, 2019.

class pypots.nn.modules.mrnn.BackboneMRNN(n_steps, n_features, rnn_hidden_size)[source]#
forward(inputs)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor, Tensor]

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.mrnn.MrnnFcnRegression(feature_num)[source]#

M-RNN fully connection regression Layer

forward(x, missing_mask, target)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.patchtst#

The package including the modules of PatchTST.

Refer to the paper Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR, 2023.

class pypots.nn.modules.patchtst.PatchtstEncoder(n_layers, n_heads, d_model, d_ffn, d_k, d_v, dropout, attn_dropout)[source]#
forward(x, attn_mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.patchtst.PatchEmbedding(d_model, patch_len, stride, padding, dropout)[source]#
forward(x)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.patchtst.RegressionHead(n_features, d_model, d_output, head_dropout, y_range=None)[source]#
forward(x)[source]#

x: [bs x nvars x d_model x num_patch] output: [bs x output_dim]

class pypots.nn.modules.patchtst.ClassificationHead(n_features, d_model, n_classes, head_dropout)[source]#
forward(x)[source]#

x: [bs x nvars x d_model x num_patch] output: [bs x n_classes]

class pypots.nn.modules.patchtst.PredictionHead(d_model, n_patches, n_steps_forecast, head_dropout=0, individual=False, n_features=0)[source]#
forward(x)[source]#

x: [bs x nvars x d_model x num_patch] output: [bs x forecast_len x nvars]

pypots.nn.modules.raindrop#

The package including the modules of Raindrop.

Refer to the paper Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In ICLR, 2022.

class pypots.nn.modules.raindrop.BackboneRaindrop(n_features, n_layers, d_model, d_ffn, n_heads, n_classes, dropout=0.3, max_len=215, d_static=9, d_pe=16, aggregation='mean', sensor_wise_mask=False, static=False)[source]#
forward(X, timestamps, lengths)[source]#

Forward processing of BRITS.

Parameters:
  • X (Tensor) – The input tensor of shape (batch_size, n_features, max_len).

  • timestamps (Tensor) – The timestamps tensor of shape (batch_size, max_len).

  • lengths (Tensor) – The lengths tensor of shape (batch_size, 1).

Returns:

prediction

Return type:

torch.Tensor

class pypots.nn.modules.raindrop.ObservationPropagation(in_channels, out_channels, n_nodes, ob_dim, heads=1, concat=True, beta=False, dropout=0.0, edge_dim=None, bias=True, root_weight=True, **kwargs)[source]#
reset_parameters()[source]#

Resets all learnable parameters of the module.

Return type:

None

forward(x, p_t, edge_index, edge_weights=None, use_beta=False, edge_attr=None, return_attention_weights=None)[source]#
Parameters:

return_attention_weights (bool, optional) – If set to True, will additionally return the tuple (edge_index, attention_weights), holding the computed attention weights for each edge. (default: None)

Return type:

Tuple[Tensor, Any]

message(x_i, x_j, edge_weights, edge_attr, index, ptr, size_i)[source]#

Constructs messages from node j to node i in analogy to \phi_{\mathbf{\Theta}} for each edge in edge_index. This function can take any argument as input which was initially passed to propagate(). Furthermore, tensors passed to propagate() can be mapped to the respective nodes i and j by appending _i or _j to the variable name, .e.g. x_i and x_j.

Return type:

Tensor

aggregate(inputs, index, ptr=None, dim_size=None)[source]#

Aggregates messages from neighbors as \square_{j \in \mathcal{N}(i)}.

Takes in the output of message computation as first argument and any argument which was initially passed to propagate().

By default, this function will delegate its call to scatter functions that support “add”, “mean” and “max” operations as specified in __init__() by the aggr argument.

Return type:

Tensor

pypots.nn.modules.saits#

The package including the modules of SAITS.

Refer to the paper Wenjie Du, David Cote, and Yan Liu. SAITS: Self-Attention-based Imputation for Time Series. Expert Systems with Applications, 219:119619, 2023.

class pypots.nn.modules.saits.BackboneSAITS(n_layers, n_steps, n_features, d_model, d_ffn, n_heads, d_k, d_v, dropout, attn_dropout)[source]#
forward(X, missing_mask, attn_mask=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.saits.SaitsLoss(ORT_weight, MIT_weight, loss_calc_func=<function calc_mae>)[source]#
forward(reconstruction, X_ori, missing_mask, indicating_mask)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.timesnet#

The package including the modules of TimesNet.

Refer to the paper Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. In ICLR, 2023.

class pypots.nn.modules.timesnet.BackboneTimesNet(n_layers, n_steps, n_pred_steps, top_k, d_model, d_ffn, n_kernels)[source]#
forward(X)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tensor

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.timesnet.InceptionBlockV1(in_channels, out_channels, num_kernels=6, init_weight=True)[source]#
forward(x)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.timesnet.TimesBlock(seq_len, pred_len, top_k, d_model, d_ffn, num_kernels)[source]#
forward(x)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.transformer#

The package including the modules of Transformer.

Refer to the papers Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. and Wenjie Du, David Cote, and Yan Liu. SAITS: Self-Attention-based Imputation for Time Series. Expert Systems with Applications, 219:119619, 2023.

class pypots.nn.modules.transformer.ScaledDotProductAttention(temperature, attn_dropout=0.1)[source]#

Scaled dot-product attention.

Parameters:
  • temperature (float) – The temperature for scaling.

  • attn_dropout (float) – The dropout rate for the attention map.

forward(q, k, v, attn_mask=None, **kwargs)[source]#

Forward processing of the scaled dot-product attention.

Parameters:
  • q (Tensor) – Query tensor.

  • k (Tensor) – Key tensor.

  • v (Tensor) – Value tensor.

  • attn_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps]. 0 in attn_mask means values at the according position in the attention map will be masked out.

Return type:

Tuple[Tensor, Tensor]

Returns:

  • output – The result of Value multiplied with the scaled dot-product attention map.

  • attn – The scaled dot-product attention map.

class pypots.nn.modules.transformer.MultiHeadAttention(n_heads, d_model, d_k, d_v, attention_operator)[source]#

Transformer multi-head attention module.

Parameters:
  • n_heads (int) – The number of heads in multi-head attention.

  • d_model (int) – The dimension of the input tensor.

  • d_k (int) – The dimension of the key and query tensor.

  • d_v (int) – The dimension of the value tensor.

  • attention_operator (AttentionOperator) – The attention operator, e.g. the self-attention proposed in Transformer.

forward(q, k, v, attn_mask, **kwargs)[source]#

Forward processing of the multi-head attention module.

Parameters:
  • q (Tensor) – Query tensor.

  • k (Tensor) – Key tensor.

  • v (Tensor) – Value tensor.

  • attn_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps]. 0 in attn_mask means values at the according position in the attention map will be masked out.

Return type:

Tuple[Tensor, Tensor]

Returns:

  • v – The output of the multi-head attention layer.

  • attn_weights – The attention map.

class pypots.nn.modules.transformer.PositionalEncoding(d_hid, n_positions=1000)[source]#

The original positional-encoding module for Transformer.

Parameters:
  • d_hid (int) – The dimension of the hidden layer.

  • n_positions (int) – The max number of positions.

forward(x, return_only_pos=False)[source]#

Forward processing of the positional encoding module.

Parameters:
  • x (Tensor) – Input tensor.

  • return_only_pos (bool) – Whether to return only the positional encoding.

Return type:

Tensor

Returns:

  • If return_only_pos is True

    pos_enc:

    The positional encoding.

  • else

    x_with_pos:

    Output tensor, the input tensor with the positional encoding added.

class pypots.nn.modules.transformer.TransformerEncoderLayer(d_model, d_ffn, n_heads, d_k, d_v, slf_attn_opt, dropout=0.1)[source]#

Transformer encoder layer.

Parameters:
  • d_model (int) – The dimension of the input tensor.

  • d_ffn (int) – The dimension of the hidden layer.

  • n_heads (int) – The number of heads in multi-head attention.

  • d_k (int) – The dimension of the key and query tensor.

  • d_v (int) – The dimension of the value tensor.

  • slf_attn_opt (AttentionOperator) – The attention operator for the self multi-head attention module in the encoder layer.

  • dropout (float) – The dropout rate.

forward(enc_input, src_mask=None, **kwargs)[source]#

Forward processing of the encoder layer.

Parameters:
  • enc_input (Tensor) – Input tensor.

  • src_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps].

Return type:

Tuple[Tensor, Tensor]

Returns:

  • enc_output – Output tensor.

  • attn_weights – The attention map.

class pypots.nn.modules.transformer.TransformerDecoderLayer(d_model, d_ffn, n_heads, d_k, d_v, slf_attn_opt, enc_attn_opt, dropout=0.1)[source]#

Transformer decoder layer.

Parameters:
  • d_model (int) – The dimension of the input tensor.

  • d_ffn (int) – The dimension of the hidden layer.

  • n_heads (int) – The number of heads in multi-head attention.

  • d_k (int) – The dimension of the key and query tensor.

  • d_v (int) – The dimension of the value tensor.

  • slf_attn_opt (AttentionOperator) – The attention operator for the self multi-head attention module in the decoder layer.

  • enc_attn_opt (AttentionOperator) – The attention operator for the encoding multi-head attention module in the decoder layer.

  • dropout (float) – The dropout rate.

forward(dec_input, enc_output, slf_attn_mask=None, dec_enc_attn_mask=None, **kwargs)[source]#

Forward processing of the decoder layer.

Parameters:
  • dec_input (Tensor) – Input tensor.

  • enc_output (Tensor) – Output tensor from the encoder.

  • slf_attn_mask (Optional[Tensor]) – Masking tensor for the self-attention module. The shape should be [batch_size, n_heads, n_steps, n_steps].

  • dec_enc_attn_mask (Optional[Tensor]) – Masking tensor for the encoding attention module. The shape should be [batch_size, n_heads, n_steps, n_steps].

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

  • dec_output – Output tensor.

  • dec_slf_attn – The self-attention map.

  • dec_enc_attn – The encoding attention map.

class pypots.nn.modules.transformer.PositionWiseFeedForward(d_in, d_hid, dropout=0.1)[source]#

Position-wise feed forward network (FFN) in Transformer.

Parameters:
  • d_in (int) – The dimension of the input tensor.

  • d_hid (int) – The dimension of the hidden layer.

  • dropout (float) – The dropout rate.

forward(x)[source]#

Forward processing of the position-wise feed forward network.

Parameters:

x (Tensor) – Input tensor.

Returns:

Output tensor.

Return type:

x

class pypots.nn.modules.transformer.TransformerEncoder(n_layers, d_model, d_ffn, n_heads, d_k, d_v, dropout, attn_dropout)[source]#

Transformer encoder.

Parameters:
  • n_layers (int) – The number of layers in the encoder.

  • d_model (int) – The dimension of the module manipulation space. The input tensor will be projected to a space with d_model dimensions.

  • d_ffn (int) – The dimension of the hidden layer in the feed-forward network.

  • n_heads (int) – The number of heads in multi-head attention.

  • d_k (int) – The dimension of the key and query tensor.

  • d_v (int) – The dimension of the value tensor.

  • dropout (float) – The dropout rate.

  • attn_dropout (float) – The dropout rate for the attention map.

forward(x, src_mask=None)[source]#

Forward processing of the encoder.

Parameters:
  • x (Tensor) – Input tensor.

  • src_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps].

Return type:

Union[Tensor, Tuple[Tensor, list]]

Returns:

  • enc_output – Output tensor.

  • attn_weights_collector – A list containing the attention map from each encoder layer.

class pypots.nn.modules.transformer.TransformerDecoder(n_layers, n_steps, n_features, d_model, d_ffn, n_heads, d_k, d_v, dropout, attn_dropout)[source]#

Transformer decoder.

Parameters:
  • n_layers (int) – The number of layers in the decoder.

  • n_steps (int) – The number of time steps in the input tensor.

  • n_features (int) – The number of features in the input tensor.

  • d_model (int) – The dimension of the module manipulation space. The input tensor will be projected to a space with d_model dimensions.

  • d_ffn (int) – The dimension of the hidden layer in the feed-forward network.

  • n_heads (int) – The number of heads in multi-head attention.

  • d_k (int) – The dimension of the key and query tensor.

  • d_v (int) – The dimension of the value tensor.

  • dropout (float) – The dropout rate.

  • attn_dropout (float) – The dropout rate for the attention map.

forward(trg_seq, enc_output, trg_mask=None, src_mask=None, return_attn_weights=False)[source]#

Forward processing of the decoder.

Parameters:
  • trg_seq (Tensor) – Input tensor.

  • enc_output (Tensor) – Output tensor from the encoder.

  • trg_mask (Optional[Tensor]) – Masking tensor for the self-attention module.

  • src_mask (Optional[Tensor]) – Masking tensor for the encoding attention module.

  • return_attn_weights (bool) – Whether to return the attention map.

Return type:

Union[Tensor, Tuple[Tensor, list, list]]

Returns:

  • dec_output – Output tensor.

  • dec_slf_attn_collector – A list containing the self-attention map from each decoder layer.

  • dec_enc_attn_collector – A list containing the encoding attention map from each decoder layer.

pypots.nn.modules.usgan#

The package including the modules of USGAN.

Refer to the paper Xiaoye Miao, Yangyang Wu, Jun Wang, Yunjun Gao, Xudong Mao, and Jianwei Yin. Generative Semi-supervised Learning for Multivariate Time Series Imputation. In AAAI, 35(10):8983–8991, May 2021.

class pypots.nn.modules.usgan.BackboneUSGAN(n_steps, n_features, rnn_hidden_size, lambda_mse, hint_rate=0.7, dropout_rate=0.0)[source]#

USGAN model

forward(inputs, training_object='generator', training=True)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, ...]

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.vader#

The package including the modules of VaDER.

Refer to the paper Johann de Jong, Mohammad Asif Emon, Ping Wu, Reagon Karki, Meemansa Sood, Patrice Godard, Ashar Ahmad, Henri Vrooman, Martin Hofmann-Apitius, and Holger Fröhlich. Deep learning for clustering of multivariate clinical patient trajectories with missing values. GigaScience, 8(11):giz134, November 2019.

class pypots.nn.modules.vader.BackboneVaDER(n_steps, d_input, n_clusters, d_rnn_hidden, d_mu_stddev, eps=1e-09, alpha=1.0)[source]#
Parameters:
  • n_steps (int) –

  • d_input (int) –

  • n_clusters (int) –

  • d_rnn_hidden (int) –

  • d_mu_stddev (int) –

  • eps (float) –

  • alpha (float) – Weight of the latent loss. The final loss = `alpha`*latent loss + reconstruction loss

forward(X, missing_mask)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor]

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.vader.PeepholeLSTMCell(input_size, hidden_size, bias=True)[source]#

Notes

This implementation is adapted from https://gist.github.com/Kaixhin/57901e91e5c5a8bac3eb0cbbdd3aba81

forward(X, hx=None)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor]

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.vader.ImplicitImputation(d_input)[source]#
forward(X, missing_mask)[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tensor

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.vader.GMMLayer(d_hidden, n_clusters)[source]#
forward()[source]#

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor, Tensor]

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.